[ 
https://issues.apache.org/jira/browse/SQOOP-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ruslan Dautkhanov updated SQOOP-3025:
-------------------------------------
    Description: 
Getting following exception in one of the mappers stack when try to export to 
Oracle from Hive Parquet 

{noformat}
2016-10-14 23:14:01,027 ERROR [main] 
org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: The following error 
occurred during configurePreparedStatement()
java.lang.NullPointerException
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.setBindValueAtName(OraOopOutputFormatBase.java:432)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.configurePreparedStatementColumns(OraOopOutputFormatBase.java:547)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:246)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303)
        at 
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153)
        at 
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107)
        at 
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
        at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
2016-10-14 23:14:01,074 INFO [main] 
org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Time spent performing 
an "exchange subpartition with table": 0.046365 sec.
2016-10-14 23:14:01,074 DEBUG [main] 
org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Dropping temporary 
mapper table "ORAOOP_20161014_231320_3"
2016-10-14 23:14:01,131 INFO [main] org.apache.hadoop.mapred.MapTask: Ignoring 
exception during close for 
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@54848551
java.io.IOException: java.sql.SQLException: java.lang.NullPointerException
        at 
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:197)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107)
        at 
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
        at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.sql.SQLException: java.lang.NullPointerException
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:255)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303)
        at 
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153)
        at 
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194)
{noformat}

Will attach full log file as an attachment.

A minimal sqoop export parameters that reproduce this problem:

{noformat}
sqoop export -Dmapred.map.max.attempts=1 --connect 
"jdbc:oracle:thin:@//somehost:1355/adash_prod_batch" --username 
DISCOVER_STG_OWNER --password somePassword --table amf_trans_stg 
--hcatalog-database disc_dv --hcatalog-table amf_trans_dv_09142016 
--num-mappers 2 --columns "individ,part_code" --direct --verbose 
{noformat}

  was:
Getting following exception in one of the mappers stack when try to export to 
Oracle from Hive Parquet 

{noformat}
2016-10-14 23:14:01,027 ERROR [main] 
org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: The following error 
occurred during configurePreparedStatement()
java.lang.NullPointerException
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.setBindValueAtName(OraOopOutputFormatBase.java:432)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.configurePreparedStatementColumns(OraOopOutputFormatBase.java:547)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:246)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303)
        at 
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153)
        at 
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107)
        at 
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
        at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
2016-10-14 23:14:01,074 INFO [main] 
org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Time spent performing 
an "exchange subpartition with table": 0.046365 sec.
2016-10-14 23:14:01,074 DEBUG [main] 
org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Dropping temporary 
mapper table "ORAOOP_20161014_231320_3"
2016-10-14 23:14:01,131 INFO [main] org.apache.hadoop.mapred.MapTask: Ignoring 
exception during close for 
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@54848551
java.io.IOException: java.sql.SQLException: java.lang.NullPointerException
        at 
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:197)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107)
        at 
org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
        at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.sql.SQLException: java.lang.NullPointerException
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:255)
        at 
org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303)
        at 
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153)
        at 
org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194)
{noformat}

Will attach full log file as an attachment.

Sqoop export options used:

{noformat}
    sqoop_params = ['sqoop', 'export'
            # marreduce options:
            , '-Dmapreduce.map.memory.mb=5000'
            , '-Dmapreduce.map.java.opts=-Xmx4000m'
            , '-Dmapreduce.job.name=sqooping %s@hadoop -> %s@oracle' % 
(inp_table, out_table)
            , '-Dmapreduce.map.maxattempts=1' 
            # Oracle (OraOOP) specific Sqoop options:
            , 
'-Dsqoop.connection.factories=org.apache.sqoop.manager.oracle.OraOopManagerFactory'
            , '-Doraoop.oracle.append.values.hint.usage=ON'
            , '-Doraoop.partitioned=true'
            , '-Doraoop.nologging=true'
            , '-Doraoop.drop.table=true'
            , '-Doraoop.template.table=%s_template' % out_table
            , '-Doraoop.temporary.table.storage.clause=' + storage_clause
            , '-Doraoop.table.storage.clause=' + storage_clause
            , '-Doraoop.oracle.rac.service.name=adash_prod_batch'
            # generic sqoop options:
            , '-Dsqoop.export.records.per.statement=5000' 
            , '-Dsqoop.export.statements.per.transaction=2000'
            , '--direct'    #activate Oracle direct mode = 'Data Connector for 
Oracle and Hadoop'
            , '--connect', "jdbc:oracle:thin:@" + jdbc_tns
            , '--username', out_schema, '--password', 
jdbc_password.decode('base64')
            , '--table', out_table
            , '--hcatalog-database', inp_schema
            , '--hcatalog-table', inp_table
            , '--batch'
            , '--num-mappers', str(num_mappers)
            ]
{noformat}


> Oracle Direct export from HCatalog parquet table gives NPE in kerberized 
> cluster
> --------------------------------------------------------------------------------
>
>                 Key: SQOOP-3025
>                 URL: https://issues.apache.org/jira/browse/SQOOP-3025
>             Project: Sqoop
>          Issue Type: Bug
>          Components: build, codegen, connectors/oracle, hive-integration, 
> metastore
>    Affects Versions: 1.4.6
>         Environment: Hadoop 2.6; Sqoop 1.4.6; Oracle 12c
>            Reporter: Ruslan Dautkhanov
>         Attachments: scoop_output.log, 
> sqoop_OracleDirect_mapper_nullPointer_exception.txt
>
>
> Getting following exception in one of the mappers stack when try to export to 
> Oracle from Hive Parquet 
> {noformat}
> 2016-10-14 23:14:01,027 ERROR [main] 
> org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: The following error 
> occurred during configurePreparedStatement()
> java.lang.NullPointerException
>       at 
> org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.setBindValueAtName(OraOopOutputFormatBase.java:432)
>       at 
> org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.configurePreparedStatementColumns(OraOopOutputFormatBase.java:547)
>       at 
> org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:246)
>       at 
> org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303)
>       at 
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153)
>       at 
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194)
>       at 
> org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578)
>       at 
> org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107)
>       at 
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
>       at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
>       at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
>       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>       at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>       at java.security.AccessController.doPrivileged(Native Method)
>       at javax.security.auth.Subject.doAs(Subject.java:415)
>       at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>       at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> 2016-10-14 23:14:01,074 INFO [main] 
> org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Time spent 
> performing an "exchange subpartition with table": 0.046365 sec.
> 2016-10-14 23:14:01,074 DEBUG [main] 
> org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Dropping temporary 
> mapper table "ORAOOP_20161014_231320_3"
> 2016-10-14 23:14:01,131 INFO [main] org.apache.hadoop.mapred.MapTask: 
> Ignoring exception during close for 
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@54848551
> java.io.IOException: java.sql.SQLException: java.lang.NullPointerException
>       at 
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:197)
>       at 
> org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578)
>       at 
> org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107)
>       at 
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670)
>       at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016)
>       at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797)
>       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>       at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>       at java.security.AccessController.doPrivileged(Native Method)
>       at javax.security.auth.Subject.doAs(Subject.java:415)
>       at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>       at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.sql.SQLException: java.lang.NullPointerException
>       at 
> org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:255)
>       at 
> org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303)
>       at 
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153)
>       at 
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194)
> {noformat}
> Will attach full log file as an attachment.
> A minimal sqoop export parameters that reproduce this problem:
> {noformat}
> sqoop export -Dmapred.map.max.attempts=1 --connect 
> "jdbc:oracle:thin:@//somehost:1355/adash_prod_batch" --username 
> DISCOVER_STG_OWNER --password somePassword --table amf_trans_stg 
> --hcatalog-database disc_dv --hcatalog-table amf_trans_dv_09142016 
> --num-mappers 2 --columns "individ,part_code" --direct --verbose 
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to