[
https://issues.apache.org/jira/browse/SQOOP-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15211465#comment-15211465
]
Ping Wang commented on SQOOP-2894:
----------------------------------
I found Parquet support via SQOOP-1390 used Kite SDK Api. By the stack trace,
kitesdk.data.spi.hive.MetaStoreUtil called HiveMetaStoreClient. There's a Kite
bug KITE-1014 for "Fix support for Hive datasets on Kerberos enabled clusters."
on version 1.1.0. Sqoop 1.4.6 is using Kite 1.0.0 without this fix.
I made the changes below:
1) Upgrade Kite to latest version for Sqoop dependency
2) From Sqoop side, add the hive configuration and send it to Kite
After the two fix, the error above in this bug is gone. But a new problem
occurred. The Kerberos authentication
sqoop import --connect jdbc:db2://xxx:50000/testdb --username xxx--password
xxx--table users --hive-import -hive-table hhh555 --as-parquetfile -m 1
--verbose
... ...
2016-03-24 21:21:12,647 DEBUG [main]
org.apache.hadoop.security.UserGroupInformation: hadoop login
2016-03-24 21:21:12,649 DEBUG [main]
org.apache.hadoop.security.UserGroupInformation: hadoop login commit
2016-03-24 21:21:12,650 DEBUG [main]
org.apache.hadoop.security.UserGroupInformation: using kerberos
user:[email protected]
2016-03-24 21:21:12,650 DEBUG [main]
org.apache.hadoop.security.UserGroupInformation: Using user:
"[email protected]" with name [email protected]
2016-03-24 21:21:12,650 DEBUG [main]
org.apache.hadoop.security.UserGroupInformation: User entry: "[email protected]"
2016-03-24 21:21:12,657 DEBUG [main]
org.apache.hadoop.security.UserGroupInformation: UGI
loginUser:[email protected] (auth:KERBEROS)
2016-03-24 21:21:12,657 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2016-03-24 21:21:12,657 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN,
Service: , Ident: (appAttemptId { application_id { id: 6 cluster_timestamp:
1458832712880 } attemptId: 1 } keyId: 1898169565)
2016-03-24 21:21:12,752 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: HDFS_DELEGATION_TOKEN,
Service: 9.30.151.107:8020, Ident: (HDFS_DELEGATION_TOKEN token 38 for
ambari-qa)
2016-03-24 21:21:12,753 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind:
TIMELINE_DELEGATION_TOKEN, Service: 9.30.151.107:8188, Ident: (owner=ambari-qa,
renewer=yarn, realUser=, issueDate=1458879665842, maxDate=1459484465842,
sequenceNumber=32, masterKeyId=37)
2016-03-24 21:21:12,672 DEBUG [TGT Renewer for [email protected]]
org.apache.hadoop.security.UserGroupInformation: Found tgt Ticket (hex) =
... ...
2016-03-24 21:21:13,728 DEBUG [main]
org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository: Loading dataset:
hhh555
2016-03-24 21:21:13,823 INFO [main] hive.metastore: Trying to connect to
metastore with URI thrift://xxx:9083
2016-03-24 21:21:13,830 DEBUG [main]
org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:ambari-qa
(auth:SIMPLE)
from:org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:403)
2016-03-24 21:21:13,858 DEBUG [main]
org.apache.hadoop.security.UserGroupInformation: PrivilegedAction as:ambari-qa
(auth:SIMPLE)
from:org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
2016-03-24 21:21:13,858 DEBUG [main]
org.apache.thrift.transport.TSaslTransport: opening transport
org.apache.thrift.transport.TSaslClientTransport@25f7391e
2016-03-24 21:21:13,864 ERROR [main]
org.apache.thrift.transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at
org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:432)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:237)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:182)
at org.kitesdk.data.spi.hive.MetaStoreUtil.<init>(MetaStoreUtil.java:89)
at
org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.getMetaStoreUtil(HiveAbstractMetadataProvider.java:63)
at
org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.resolveNamespace(HiveAbstractMetadataProvider.java:270)
at
org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.resolveNamespace(HiveAbstractMetadataProvider.java:255)
at
org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.load(HiveAbstractMetadataProvider.java:102)
at
org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.load(FileSystemDatasetRepository.java:197)
at org.kitesdk.data.Datasets.load(Datasets.java:108)
at org.kitesdk.data.Datasets.load(Datasets.java:165)
at
org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.load(DatasetKeyOutputFormat.java:542)
at
org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.getOutputCommitter(DatasetKeyOutputFormat.java:505)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:476)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:458)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1560)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:458)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:377)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1518)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)
at
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 34 more
> Hive import with Parquet failed in Kerberos enabled cluster
> -----------------------------------------------------------
>
> Key: SQOOP-2894
> URL: https://issues.apache.org/jira/browse/SQOOP-2894
> Project: Sqoop
> Issue Type: Bug
> Components: tools
> Affects Versions: 1.4.6
> Environment: Redhat 6.6, Sqoop 1.4.6+Hadoop 2.7.2+Hive 1.2.1
> Reporter: Ping Wang
> Labels: security
>
> Importing data from external database to hive with Parquet option failed in
> the kerberos environment. (It can success without kerberos).
> The sqoop command I used:
> sqoop import --connect jdbc:db2://xxx:50000/testdb --username xxx --password
> xxx --table users --hive-import -hive-table users3 --as-parquetfile -m 1
> The import job failed:
> ......
> 2016-02-26 04:20:07,020 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
> 2016-02-26 04:20:08,088 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config
> null
> 2016-02-26 04:20:08,918 INFO [main] hive.metastore: Trying to connect to
> metastore with URI thrift://xxx:9083
> 2016-02-26 04:30:09,207 WARN [main] hive.metastore: set_ugi() not successful,
> Likely cause: new client talking to old server. Continuing without it.
> org.apache.thrift.transport.TTransportException:
> java.net.SocketTimeoutException: Read timed out
> at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:129)
> at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:380)
> at
> org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:3688)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:3674)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:448)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:237)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:182)
> at org.kitesdk.data.spi.hive.MetaStoreUtil.<init>(MetaStoreUtil.java:82)
> at
> org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.getMetaStoreUtil(HiveAbstractMetadataProvider.java:63)
> at
> org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.resolveNamespace(HiveAbstractMetadataProvider.java:270)
> at
> org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.resolveNamespace(HiveAbstractMetadataProvider.java:255)
> at
> org.kitesdk.data.spi.hive.HiveAbstractMetadataProvider.load(HiveAbstractMetadataProvider.java:102)
> at
> org.kitesdk.data.spi.filesystem.FileSystemDatasetRepository.load(FileSystemDatasetRepository.java:192)
> at org.kitesdk.data.Datasets.load(Datasets.java:108)
> at org.kitesdk.data.Datasets.load(Datasets.java:165)
> at
> org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.load(DatasetKeyOutputFormat.java:510)
> at
> org.kitesdk.data.mapreduce.DatasetKeyOutputFormat.getOutputCommitter(DatasetKeyOutputFormat.java:473)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:476)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:458)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1560)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:458)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:377)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1518)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1515)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1448)
> .......
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
