Does this mean that sqoop tries to read hbase-site.xml and then expectes hbase to pass the delegation token to it thru hbase.security.user class ? I am using hbase 94.9 Hbase complains with the following msg - 2013-08-05 11:59:33,121 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: org.apache.hadoop.hbase.security.AccessDeniedException: Token generation only allowed for Kerberos authenticated clients at org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:87)
What am I missing here? Should I specify anything in sqoop-site.xml related to kerberos? Cheers, Suhas. On Tue, Aug 6, 2013 at 11:23 AM, Abraham Elmahrek <[email protected]> wrote: > Sorry, apparently this is an HBase specific token. See here > http://wiki.apache.org/hadoop/Hbase/HBaseTokenAuthentication. > > > On Tue, Aug 6, 2013 at 11:13 AM, Abraham Elmahrek <[email protected]>wrote: > >> Suhas, >> >> Sqoop 1.4.3 simply fetches the authenticated user from credentials cache >> and fetches a delegation token for HBase. See >> https://issues.apache.org/jira/browse/SQOOP-599 for more information. >> >> -Abe >> >> >> On Tue, Aug 6, 2013 at 11:09 AM, Suhas Satish <[email protected]>wrote: >> >>> I was able to isolate this problem to the Sqoop side not picking up >>> correct kerberos credentials. Hbase is picking up the correct kerberos >>> credentials when Hbase put and scan are done in isolation without using >>> Sqoop. >>> >>> A direct map-reduce put into HBase uses the following 2 methods - >>> HBaseConfiguration.merge(conf, HBaseConfiguration.create(conf)); >>> TableMapReduceUtil.initCredentials(job); >>> >>> I was looking at how sqoop 1.4.3 does HBase puts to see if it converts >>> sqoop import arguments into map-reduce jobs and uses the above methods >>> somewhere. This is what I found - >>> HBasePutProcessor.java - SqoopRecordProcessor that performs a HBase >>> "put" operation - has a method to get hadoop configuration, but none to >>> merge any kerberos specific configurations specified in sqoop-site.xml- >>> >>> public Configuration getConf() { >>> return this.conf; >>> >>> >>> >>> HBaseUtil.java - makes sure hbase jars are present on class path >>> PutTransformer.java - converts jdbc statements in the form of K-V map >>> into hbase put commands and returns a list >>> ToStringPutTransformer.java - extends the above class >>> >>> Does anyone know sqoop internals of how to specify kerberos >>> configurations and get sqoop to read them? >>> >>> Cheers, >>> Suhas. >>> >>> >>> On Tue, Aug 6, 2013 at 10:31 AM, Suhas Satish <[email protected]>wrote: >>> >>>> Ataching the logs here at the time of authentication, I do not see any >>>> error msges here. >>>> >>>> /var/log/kadmind.log >>>> /var/log/krb5kdc.log >>>> >>>> Please let me know if there is any other places I can find other log >>>> files >>>> >>>> Cheers, >>>> Suhas. >>>> >>>> >>>> On Mon, Aug 5, 2013 at 4:48 PM, Abraham Elmahrek <[email protected]>wrote: >>>> >>>>> User, >>>>> >>>>> Could you please provide your KDC logs around the time you tried to >>>>> authenticate? >>>>> >>>>> Note: A kerberos client will negotiate the encryption algorithm it >>>>> can/will use with the KDC. It may choose AES-256. >>>>> >>>>> -Abe >>>>> >>>>> >>>>> On Mon, Aug 5, 2013 at 3:55 PM, Suhas Satish >>>>> <[email protected]>wrote: >>>>> >>>>>> I generated a keytab with the following cmd and it supports multiple >>>>>> encryption types other than aes256 as listed below. >>>>>> But I still get the same error from sqoop import tool because the >>>>>> sqoop.keytab is not being read (sqoop being the hbase client in this >>>>>> case). >>>>>> >>>>>> kadmin: ktadd -k sqoop.keytab kuser1 >>>>>> Entry for principal kuser1 with kvno 2, encryption type >>>>>> aes256-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab. >>>>>> Entry for principal kuser1 with kvno 2, encryption type >>>>>> aes128-cts-hmac-sha1-96 added to keytab WRFILE:sqoop.keytab. >>>>>> Entry for principal kuser1 with kvno 2, encryption type des3-cbc-sha1 >>>>>> added to keytab WRFILE:sqoop.keytab. >>>>>> Entry for principal kuser1 with kvno 2, encryption type arcfour-hmac >>>>>> added to keytab WRFILE:sqoop.keytab. >>>>>> Entry for principal kuser1 with kvno 2, encryption type des-hmac-sha1 >>>>>> added to keytab WRFILE:sqoop.keytab. >>>>>> Entry for principal kuser1 with kvno 2, encryption type des-cbc-md5 >>>>>> added to keytab WRFILE:sqoop.keytab. >>>>>> >>>>>> Here are some more debug logs I obtained from kerberos - >>>>>> >>>>>> *kadmin: getprinc kuser1* >>>>>> Principal: [email protected] >>>>>> Expiration date: [never] >>>>>> Last password change: Mon Aug 05 15:40:30 PDT 2013 >>>>>> Password expiration date: [none] >>>>>> Maximum ticket life: 1 day 00:00:00 >>>>>> Maximum renewable life: 0 days 00:00:00 >>>>>> Last modified: Mon Aug 05 15:40:30 PDT 2013 (mapr/[email protected]) >>>>>> Last successful authentication: [never] >>>>>> Last failed authentication: [never] >>>>>> Failed password attempts: 0 >>>>>> Number of keys: 6 >>>>>> Key: vno 2, aes256-cts-hmac-sha1-96, no salt >>>>>> Key: vno 2, aes128-cts-hmac-sha1-96, no salt >>>>>> Key: vno 2, des3-cbc-sha1, no salt >>>>>> Key: vno 2, arcfour-hmac, no salt >>>>>> Key: vno 2, des-hmac-sha1, no salt >>>>>> Key: vno 2, des-cbc-md5, no salt >>>>>> MKey: vno 1 >>>>>> Attributes: >>>>>> Policy: [none] >>>>>> >>>>>> *getprinc hbase/qa-node133.qa.lab* >>>>>> Principal: hbase/[email protected] >>>>>> Expiration date: [never] >>>>>> Last password change: Mon Jul 29 19:17:46 PDT 2013 >>>>>> Password expiration date: [none] >>>>>> Maximum ticket life: 0 days 10:00:00 >>>>>> Maximum renewable life: 7 days 00:00:00 >>>>>> Last modified: Mon Jul 29 19:17:46 PDT 2013 (kuser1/[email protected]) >>>>>> Last successful authentication: [never] >>>>>> Last failed authentication: [never] >>>>>> Failed password attempts: 0 >>>>>> Number of keys: 6 >>>>>> Key: vno 2, aes256-cts-hmac-sha1-96, no salt >>>>>> Key: vno 2, aes128-cts-hmac-sha1-96, no salt >>>>>> Key: vno 2, des3-cbc-sha1, no salt >>>>>> Key: vno 2, arcfour-hmac, no salt >>>>>> Key: vno 2, des-hmac-sha1, no salt >>>>>> Key: vno 2, des-cbc-md5, no salt >>>>>> MKey: vno 1 >>>>>> Attributes: >>>>>> Policy: [none] >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Suhas. >>>>>> >>>>>> >>>>>> On Mon, Aug 5, 2013 at 2:29 PM, Abraham Elmahrek >>>>>> <[email protected]>wrote: >>>>>> >>>>>>> There should be a password. You should have a keytab associated with >>>>>>> that principal, which would allow you to authenticate as that principal. >>>>>>> See >>>>>>> http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Security-Guide/CDH4-Security-Guide.htmlfor >>>>>>> more details on how that works. >>>>>>> >>>>>>> A couple of things... >>>>>>> 1. You need to make your kerberos credentials renewable. Right now >>>>>>> it seems like you cannot renew. See >>>>>>> http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Security-Guide/cdh4sg_topic_17.html >>>>>>> . >>>>>>> 2. AES256 encryption is not inherently supported. Did you install >>>>>>> support for AES256? >>>>>>> >>>>>>> -Abe >>>>>>> >>>>>>> >>>>>>> On Mon, Aug 5, 2013 at 1:53 PM, Suhas Satish <[email protected] >>>>>>> > wrote: >>>>>>> >>>>>>>> klist -e -v >>>>>>>> >>>>>>>> Ticket cache: FILE:/tmp/krb5cc_0 >>>>>>>> Default principal: [email protected] >>>>>>>> >>>>>>>> Valid starting Expires Service principal >>>>>>>> 08/05/13 12:34:42 08/05/13 22:34:42 krbtgt/[email protected] >>>>>>>> renew until 08/05/13 12:34:42, Etype (skey, tkt): >>>>>>>> aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 >>>>>>>> >>>>>>>> Kerberos 5 version 1.10.3 >>>>>>>> >>>>>>>> The principal in hbase-site.xml is >>>>>>>> hbase/[email protected] >>>>>>>> >>>>>>>> How do I create a credential using kinit matching that in >>>>>>>> hbase-site.xml? kinit hbase/qa-node133.qa.lab throws an error msg >>>>>>>> *kinit: Password incorrect while getting initial credentials* >>>>>>>> *although I know that there is no password for that principal. * >>>>>>>> * >>>>>>>> * >>>>>>>> * >>>>>>>> * >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Suhas. >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Aug 5, 2013 at 12:52 PM, Abraham Elmahrek <[email protected] >>>>>>>> > wrote: >>>>>>>> >>>>>>>>> Hi there, >>>>>>>>> >>>>>>>>> It seems like your client isn't authenticated in both cases. You >>>>>>>>> seem to be receiving errors from HBase and Sqoop. Sqoop 1.4.3 should >>>>>>>>> simply >>>>>>>>> work if your user is already authenticated. Internally, Sqoop is >>>>>>>>> generating >>>>>>>>> delegation tokens to communicate with HBase. It cannot do that without >>>>>>>>> being properly authenticated first though. >>>>>>>>> >>>>>>>>> Could you provide the output of the following command: >>>>>>>>> "klist -e -v" >>>>>>>>> >>>>>>>>> -Abe >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Aug 5, 2013 at 12:15 PM, Suhas Satish < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> I have configured hbase 94.9 with kerberos successfully for >>>>>>>>>> authentication and authorization as mentioned in the CDH security >>>>>>>>>> docs. I >>>>>>>>>> am using sqoop 1.4.3. Is there any configuration required from the >>>>>>>>>> sqoop >>>>>>>>>> client side for kerberos? >>>>>>>>>> >>>>>>>>>> I have the following permissions on hbase tables - >>>>>>>>>> hbase(main):003:0> grant 'kuser1', 'RWXCA', 'demo' >>>>>>>>>> ERROR: org.apache.hadoop.hbase.security.AccessDeniedException: >>>>>>>>>> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient >>>>>>>>>> permissions (user=kuser1, scope=demo, family=, qualifer=, >>>>>>>>>> action=ADMIN) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> bin/sqoop import --connect jdbc:mysql://10.10.1.10/TestDB >>>>>>>>>> --table t1 --hbase-table t1 --column-family world >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> When I try to import into it using sqoop with the above cmd, I >>>>>>>>>> get the following error - >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> 2013-08-05 11:59:33,121 ERROR >>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: >>>>>>>>>> org.apache.hadoop.hbase.security.AccessDeniedException: Token >>>>>>>>>> generation only allowed for Kerberos authenticated clients >>>>>>>>>> at >>>>>>>>>> org.apache.hadoop.hbase.security.token.TokenProvider.getAuthenticationToken(TokenProvider.java:87) >>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>>>> at >>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>>>>>>>>> at >>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>>>>>> at >>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.java:5576) >>>>>>>>>> at >>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(HRegionServer.java:3868) >>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>>>> at >>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >>>>>>>>>> at >>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597) >>>>>>>>>> at >>>>>>>>>> org.apache.hadoop.hbase.ipc.SecureRpcEngine$Server.call(SecureRpcEngine.java:308) >>>>>>>>>> at >>>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> Suhas. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
