[ https://issues.apache.org/jira/browse/SQOOP-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15582749#comment-15582749 ]
Markus Kemper commented on SQOOP-3025: -------------------------------------- Hey Ruslan, Are you able to provide the following: 1. On the host executing the Sqoop CLI command what are your security settings for [hive conf] and [hadoop conf] 2. Our [sqoop test] appears to be working without error [hive conf] $ cat /etc/hive/conf/*.xml | egrep -B 1 -A 1 -i "security.authentication|hadoop.security.authorization<" <property> <name>hadoop.security.authentication</name> <value>kerberos</value> <property> <name>hadoop.security.authorization</name> <value>true</value> [hadoop conf] $ cat /etc/hadoop/conf/*.xml | egrep -B 1 -A 1 -i "security.authentication|hadoop.security.authorization<" <property> <name>hadoop.security.authentication</name> <value>kerberos</value> <property> <name>hadoop.security.authorization</name> <value>true</value> [sqoop test] $ sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "select * from t1" Output: ----------------------------------------------- | C1 | C2 | ----------------------------------------------- | 1 | one | ----------------------------------------------- $ sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table T1 --target-dir /user/xxx/t1_text --delete-target-dir --num-mappers 1 --as-textfile --hive-import --hive-database default --hive-table t1_text --verbose $ hive -e "use default; create table t1_parquet stored as parquet as select * from t1_text;" $ sqoop export --connect $MYCONN --username $MYUSER --password $MYPSWD --table T1 --num-mappers 2 --hcatalog-database default --hcatalog-table t1_parquet --direct Output: 16/10/17 09:01:16 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.8.4-SNAPSHOT 16/10/17 09:01:16 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 16/10/17 09:01:16 INFO manager.SqlManager: Using default fetchSize of 1000 16/10/17 09:01:18 INFO oracle.OraOopOracleQueries: Current schema is: SQOOP 16/10/17 09:01:18 INFO oracle.OraOopManagerFactory: ************************************************** *** Using Data Connector for Oracle and Hadoop *** ************************************************** <SNIP> 16/10/17 09:01:24 INFO client.RMProxy: Connecting to ResourceManager at xxx.cloudera.com/xxx.xx.xx.xxx:8032 16/10/17 09:01:25 INFO hdfs.DFSClient: Created token for admin: HDFS_DELEGATION_TOKEN owner=x...@xxx.cloudera.com, renewer=yarn, realUser=, issueDate=1476720085263, maxDate=1477324885263, sequenceNumber=8, masterKeyId=6 on ha-hdfs:ns1 16/10/17 09:01:25 INFO security.TokenCache: Got dt for hdfs://ns1; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:ns1, Ident: (token for admin: HDFS_DELEGATION_TOKEN owner=x...@xxx.cloudera.com, renewer=yarn, realUser=, issueDate=1476720085263, maxDate=1477324885263, sequenceNumber=8, masterKeyId=6) <SNIP> 16/10/17 09:02:04 INFO mapreduce.Job: map 0% reduce 0% 16/10/17 09:02:22 INFO mapreduce.Job: map 100% reduce 0% 16/10/17 09:02:22 INFO mapreduce.Job: Job job_1476708184468_0005 completed successfully <SNIP> 16/10/17 09:02:22 INFO mapreduce.ExportJobBase: Transferred 8.1641 KB in 57.6968 seconds (144.8954 bytes/sec) 16/10/17 09:02:22 INFO mapreduce.ExportJobBase: Exported 1 records. $ sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "select * from t1" Output: ----------------------------------------------- | C1 | C2 | ----------------------------------------------- | 1 | one | | 1 | one | ----------------------------------------------- Thanks, Markus > Oracle Direct export from HCatalog parquet table gives NPE in kerberized > cluster > -------------------------------------------------------------------------------- > > Key: SQOOP-3025 > URL: https://issues.apache.org/jira/browse/SQOOP-3025 > Project: Sqoop > Issue Type: Bug > Components: build, codegen, connectors/oracle, hive-integration, > metastore > Affects Versions: 1.4.6 > Environment: Hadoop 2.6; Sqoop 1.4.6; Oracle 12c > Reporter: Ruslan Dautkhanov > Attachments: scoop_output.log, > sqoop_OracleDirect_mapper_nullPointer_exception.txt > > > Getting following exception in one of the mappers stack when try to export to > Oracle from Hive Parquet > {noformat} > 2016-10-14 23:14:01,027 ERROR [main] > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: The following error > occurred during configurePreparedStatement() > java.lang.NullPointerException > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.setBindValueAtName(OraOopOutputFormatBase.java:432) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.configurePreparedStatementColumns(OraOopOutputFormatBase.java:547) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:246) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303) > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153) > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) > at > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > 2016-10-14 23:14:01,074 INFO [main] > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Time spent > performing an "exchange subpartition with table": 0.046365 sec. > 2016-10-14 23:14:01,074 DEBUG [main] > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Dropping temporary > mapper table "ORAOOP_20161014_231320_3" > 2016-10-14 23:14:01,131 INFO [main] org.apache.hadoop.mapred.MapTask: > Ignoring exception during close for > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@54848551 > java.io.IOException: java.sql.SQLException: java.lang.NullPointerException > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:197) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) > at > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.sql.SQLException: java.lang.NullPointerException > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:255) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303) > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153) > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194) > {noformat} > Will attach full log file as an attachment. > A minimal sqoop export parameters that reproduce this problem: > {quote} > sqoop export -Dmapred.map.max.attempts=1 --connect > "jdbc:oracle:thin:@//somehost:1355/adash_prod_batch" --username > DISCOVER_STG_OWNER --password somePassword --table amf_trans_stg > --hcatalog-database disc_dv --hcatalog-table amf_trans_dv_09142016 > --num-mappers 2 --columns "individ,part_code" --direct --verbose > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)