I figured out that I had to authenticate using kinit. But hadoop UGI object which does the authentication had abug in it as a result of which the authentication wasn't happening. We are working on a fix in the hadoop code.
Cheers, Suhas. On Sun, Aug 11, 2013 at 1:10 PM, Jarek Jarcec Cecho <[email protected]>wrote: > Hi Suhas, > you should not be specifying anything in the sqoop-site.xml regarding > kerberos. You should authenticate yourself (using kinit) and Sqoop will > simply use those credentials to communicate with Hadoop and HBase. > > Would you mind sharing with us entire Sqoop command line and entire log > generated with parameter --verbose? > > Jarcec > > On Tue, Aug 06, 2013 at 01:30:35PM -0700, Suhas Satish wrote: > > Does this mean that sqoop tries to read hbase-site.xml and then expectes > > hbase to pass the delegation token to it thru hbase.security.user class > ? > > I am using hbase 94.9 > > Hbase complains with the following msg - > > 2013-08-05 11:59:33,121 ERROR > > org.apache.hadoop.hbase.regionserver.HRegionServer: > > org.apache.hadoop.hbase.security.AccessDeniedException: Token generation > > only allowed for Kerberos authenticated clients > > at > > > org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:87) > > > > What am I missing here? Should I specify anything in sqoop-site.xml > > related to kerberos? > > > > Cheers, > > Suhas. > > > > > > On Tue, Aug 6, 2013 at 11:23 AM, Abraham Elmahrek <[email protected]> > wrote: > > > > > Sorry, apparently this is an HBase specific token. See here > > > http://wiki.apache.org/hadoop/Hbase/HBaseTokenAuthentication. > > > > > > > > > On Tue, Aug 6, 2013 at 11:13 AM, Abraham Elmahrek <[email protected] > >wrote: > > > > > >> Suhas, > > >> > > >> Sqoop 1.4.3 simply fetches the authenticated user from credentials > cache > > >> and fetches a delegation token for HBase. See > > >> https://issues.apache.org/jira/browse/SQOOP-599 for more information. > > >> > > >> -Abe > > >> > > >> > > >> On Tue, Aug 6, 2013 at 11:09 AM, Suhas Satish <[email protected] > >wrote: > > >> > > >>> I was able to isolate this problem to the Sqoop side not picking up > > >>> correct kerberos credentials. Hbase is picking up the correct > kerberos > > >>> credentials when Hbase put and scan are done in isolation without > using > > >>> Sqoop. > > >>> > > >>> A direct map-reduce put into HBase uses the following 2 methods - > > >>> HBaseConfiguration.merge(conf, HBaseConfiguration.create(conf)); > > >>> TableMapReduceUtil.initCredentials(job); > > >>> > > >>> I was looking at how sqoop 1.4.3 does HBase puts to see if it > converts > > >>> sqoop import arguments into map-reduce jobs and uses the above > methods > > >>> somewhere. This is what I found - > > >>> HBasePutProcessor.java - SqoopRecordProcessor that performs a HBase > > >>> "put" operation - has a method to get hadoop configuration, but none > to > > >>> merge any kerberos specific configurations specified in > sqoop-site.xml- > > >>> > > >>> public Configuration getConf() { > > >>> return this.conf; > > >>> > > >>> > > >>> > > >>> HBaseUtil.java - makes sure hbase jars are present on class path > > >>> PutTransformer.java - converts jdbc statements in the form of K-V > map > > >>> into hbase put commands and returns a list > > >>> ToStringPutTransformer.java - extends the above class > > >>> > > >>> Does anyone know sqoop internals of how to specify kerberos > > >>> configurations and get sqoop to read them? > > >>> > > >>> Cheers, > > >>> Suhas. > > >>> > > >>> > > >>> On Tue, Aug 6, 2013 at 10:31 AM, Suhas Satish < > [email protected]>wrote: > > >>> > > >>>> Ataching the logs here at the time of authentication, I do not see > any > > >>>> error msges here. > > >>>> > > >>>> /var/log/kadmind.log > > >>>> /var/log/krb5kdc.log > > >>>> > > >>>> Please let me know if there is any other places I can find other log > > >>>> files > > >>>> > > >>>> Cheers, > > >>>> Suhas. > > >>>> > > >>>> > > >>>> On Mon, Aug 5, 2013 at 4:48 PM, Abraham Elmahrek <[email protected] > >wrote: > > >>>> > > >>>>> User, > > >>>>> > > >>>>> Could you please provide your KDC logs around the time you tried to > > >>>>> authenticate? > > >>>>> > > >>>>> Note: A kerberos client will negotiate the encryption algorithm it > > >>>>> can/will use with the KDC. It may choose AES-256. > > >>>>> > > >>>>> -Abe > > >>>>> > > >>>>> > > >>>>> On Mon, Aug 5, 2013 at 3:55 PM, Suhas Satish < > [email protected]>wrote: > > >>>>> > > >>>>>> I generated a keytab with the following cmd and it supports > multiple > > >>>>>> encryption types other than aes256 as listed below. > > >>>>>> But I still get the same error from sqoop import tool because the > > >>>>>> sqoop.keytab is not being read (sqoop being the hbase client in > this case). > > >>>>>> > > >>>>>> kadmin: ktadd -k sqoop.keytab kuser1 > > >>>>>> Entry for principal kuser1 with kvno 2, encryption type > > >>>>>> aes256-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab. > > >>>>>> Entry for principal kuser1 with kvno 2, encryption type > > >>>>>> aes128-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab. > > >>>>>> Entry for principal kuser1 with kvno 2, encryption type > des3-cbc-sha1 > > >>>>>> added to keytab WRFILE:sqoop.keytab. > > >>>>>> Entry for principal kuser1 with kvno 2, encryption type > arcfour-hmac > > >>>>>> added to keytab WRFILE:sqoop.keytab. > > >>>>>> Entry for principal kuser1 with kvno 2, encryption type > des-hmac-sha1 > > >>>>>> added to keytab WRFILE:sqoop.keytab. > > >>>>>> Entry for principal kuser1 with kvno 2, encryption type > des-cbc-md5 > > >>>>>> added to keytab WRFILE:sqoop.keytab. > > >>>>>> > > >>>>>> Here are some more debug logs I obtained from kerberos - > > >>>>>> > > >>>>>> *kadmin: getprinc kuser1* > > >>>>>> Principal: [email protected] > > >>>>>> Expiration date: [never] > > >>>>>> Last password change: Mon Aug 05 15:40:30 PDT 2013 > > >>>>>> Password expiration date: [none] > > >>>>>> Maximum ticket life: 1 day 00:00:00 > > >>>>>> Maximum renewable life: 0 days 00:00:00 > > >>>>>> Last modified: Mon Aug 05 15:40:30 PDT 2013 (mapr/[email protected]) > > >>>>>> Last successful authentication: [never] > > >>>>>> Last failed authentication: [never] > > >>>>>> Failed password attempts: 0 > > >>>>>> Number of keys: 6 > > >>>>>> Key: vno 2, aes256-cts-hmac-sha1-96, no salt > > >>>>>> Key: vno 2, aes128-cts-hmac-sha1-96, no salt > > >>>>>> Key: vno 2, des3-cbc-sha1, no salt > > >>>>>> Key: vno 2, arcfour-hmac, no salt > > >>>>>> Key: vno 2, des-hmac-sha1, no salt > > >>>>>> Key: vno 2, des-cbc-md5, no salt > > >>>>>> MKey: vno 1 > > >>>>>> Attributes: > > >>>>>> Policy: [none] > > >>>>>> > > >>>>>> *getprinc hbase/qa-node133.qa.lab* > > >>>>>> Principal: hbase/[email protected] > > >>>>>> Expiration date: [never] > > >>>>>> Last password change: Mon Jul 29 19:17:46 PDT 2013 > > >>>>>> Password expiration date: [none] > > >>>>>> Maximum ticket life: 0 days 10:00:00 > > >>>>>> Maximum renewable life: 7 days 00:00:00 > > >>>>>> Last modified: Mon Jul 29 19:17:46 PDT 2013 (kuser1/[email protected]) > > >>>>>> Last successful authentication: [never] > > >>>>>> Last failed authentication: [never] > > >>>>>> Failed password attempts: 0 > > >>>>>> Number of keys: 6 > > >>>>>> Key: vno 2, aes256-cts-hmac-sha1-96, no salt > > >>>>>> Key: vno 2, aes128-cts-hmac-sha1-96, no salt > > >>>>>> Key: vno 2, des3-cbc-sha1, no salt > > >>>>>> Key: vno 2, arcfour-hmac, no salt > > >>>>>> Key: vno 2, des-hmac-sha1, no salt > > >>>>>> Key: vno 2, des-cbc-md5, no salt > > >>>>>> MKey: vno 1 > > >>>>>> Attributes: > > >>>>>> Policy: [none] > > >>>>>> > > >>>>>> > > >>>>>> Thanks, > > >>>>>> Suhas. > > >>>>>> > > >>>>>> > > >>>>>> On Mon, Aug 5, 2013 at 2:29 PM, Abraham Elmahrek < > [email protected]>wrote: > > >>>>>> > > >>>>>>> There should be a password. You should have a keytab associated > with > > >>>>>>> that principal, which would allow you to authenticate as that > principal. > > >>>>>>> See > > >>>>>>> > http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Security-Guide/CDH4-Security-Guide.htmlformore > details on how that works. > > >>>>>>> > > >>>>>>> A couple of things... > > >>>>>>> 1. You need to make your kerberos credentials renewable. Right > now > > >>>>>>> it seems like you cannot renew. See > > >>>>>>> > http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Security-Guide/cdh4sg_topic_17.html > > >>>>>>> . > > >>>>>>> 2. AES256 encryption is not inherently supported. Did you install > > >>>>>>> support for AES256? > > >>>>>>> > > >>>>>>> -Abe > > >>>>>>> > > >>>>>>> > > >>>>>>> On Mon, Aug 5, 2013 at 1:53 PM, Suhas Satish < > [email protected] > > >>>>>>> > wrote: > > >>>>>>> > > >>>>>>>> klist -e -v > > >>>>>>>> > > >>>>>>>> Ticket cache: FILE:/tmp/krb5cc_0 > > >>>>>>>> Default principal: [email protected] > > >>>>>>>> > > >>>>>>>> Valid starting Expires Service principal > > >>>>>>>> 08/05/13 12:34:42 08/05/13 22:34:42 krbtgt/[email protected] > > >>>>>>>> renew until 08/05/13 12:34:42, Etype (skey, tkt): > > >>>>>>>> aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 > > >>>>>>>> > > >>>>>>>> Kerberos 5 version 1.10.3 > > >>>>>>>> > > >>>>>>>> The principal in hbase-site.xml is > > >>>>>>>> hbase/[email protected] > > >>>>>>>> > > >>>>>>>> How do I create a credential using kinit matching that in > > >>>>>>>> hbase-site.xml? kinit hbase/qa-node133.qa.lab throws an > error msg > > >>>>>>>> *kinit: Password incorrect while getting initial credentials* > > >>>>>>>> *although I know that there is no password for that principal. * > > >>>>>>>> * > > >>>>>>>> * > > >>>>>>>> * > > >>>>>>>> * > > >>>>>>>> > > >>>>>>>> Cheers, > > >>>>>>>> Suhas. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Mon, Aug 5, 2013 at 12:52 PM, Abraham Elmahrek < > [email protected] > > >>>>>>>> > wrote: > > >>>>>>>> > > >>>>>>>>> Hi there, > > >>>>>>>>> > > >>>>>>>>> It seems like your client isn't authenticated in both cases. > You > > >>>>>>>>> seem to be receiving errors from HBase and Sqoop. Sqoop 1.4.3 > should simply > > >>>>>>>>> work if your user is already authenticated. Internally, Sqoop > is generating > > >>>>>>>>> delegation tokens to communicate with HBase. It cannot do that > without > > >>>>>>>>> being properly authenticated first though. > > >>>>>>>>> > > >>>>>>>>> Could you provide the output of the following command: > > >>>>>>>>> "klist -e -v" > > >>>>>>>>> > > >>>>>>>>> -Abe > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> On Mon, Aug 5, 2013 at 12:15 PM, Suhas Satish < > > >>>>>>>>> [email protected]> wrote: > > >>>>>>>>> > > >>>>>>>>>> I have configured hbase 94.9 with kerberos successfully for > > >>>>>>>>>> authentication and authorization as mentioned in the CDH > security docs. I > > >>>>>>>>>> am using sqoop 1.4.3. Is there any configuration required > from the sqoop > > >>>>>>>>>> client side for kerberos? > > >>>>>>>>>> > > >>>>>>>>>> I have the following permissions on hbase tables - > > >>>>>>>>>> hbase(main):003:0> grant 'kuser1', 'RWXCA', 'demo' > > >>>>>>>>>> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: > > >>>>>>>>>> org.apache.hadoop.hbase.security.AccessDeniedException: > Insufficient > > >>>>>>>>>> permissions (user=kuser1, scope=demo, family=, qualifer=, > action=ADMIN) > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> bin/sqoop import --connect jdbc:mysql://10.10.1.10/TestDB > > >>>>>>>>>> --table t1 --hbase-table t1 --column-family world > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> When I try to import into it using sqoop with the above cmd, I > > >>>>>>>>>> get the following error - > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> 2013-08-05 11:59:33,121 ERROR > > >>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: > > >>>>>>>>>> org.apache.hadoop.hbase.security.AccessDeniedException: Token > > >>>>>>>>>> generation only allowed for Kerberos authenticated clients > > >>>>>>>>>> at > > >>>>>>>>>> > org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:87) > > >>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > >>>>>>>>>> at > > >>>>>>>>>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > >>>>>>>>>> at > > >>>>>>>>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > >>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597) > > >>>>>>>>>> at > > >>>>>>>>>> > org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.java:5576) > > >>>>>>>>>> at > > >>>>>>>>>> > org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(HRegionServer.java:3868) > > >>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > > >>>>>>>>>> at > > >>>>>>>>>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > >>>>>>>>>> at > > >>>>>>>>>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > >>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597) > > >>>>>>>>>> at > > >>>>>>>>>> > org.apache.hadoop.hbase.ipc.SecureRpcEngine$Server.call(SecureRpcEngine.java:308) > > >>>>>>>>>> at > > >>>>>>>>>> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Cheers, > > >>>>>>>>>> Suhas. > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > > >
