[jira] [Commented] (SQOOP-3022) sqoop export for Oracle generates tremendous amounts of redo logs
[ https://issues.apache.org/jira/browse/SQOOP-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578492#comment-15578492 ] Ruslan Dautkhanov commented on SQOOP-3022: -- Hi [~maugli], I still wasn't able to run tests yet- running into NullPointerException in Oracle direct mode - created https://issues.apache.org/jira/browse/SQOOP-3025 Thank you, Ruslan > sqoop export for Oracle generates tremendous amounts of redo logs > - > > Key: SQOOP-3022 > URL: https://issues.apache.org/jira/browse/SQOOP-3022 > Project: Sqoop > Issue Type: Bug > Components: codegen, connectors, connectors/oracle >Affects Versions: 1.4.3, 1.4.4, 1.4.5, 1.4.6 >Reporter: Ruslan Dautkhanov > Labels: export, oracle > > Sqoop export for Oracle generates tremendous amounts of redo logs (comparable > to export size or more). > We have put target tables in nologgin mode, but Oracle will still generate > redo logs unless +APPEND Oracle insert hint is used. > See https://oracle-base.com/articles/misc/append-hint for examples. > Please add an option for sqoop to generate insert statements in Oracle with > APPEND statement. Our databases are swamped with redo/archived logs whenever > we sqoop data to them. This is easily avoidable. And from business > prospective sqooping to staging tables in nologgin mode is totally fine. > Thank you. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-3022) sqoop export for Oracle generates tremendous amounts of redo logs
[ https://issues.apache.org/jira/browse/SQOOP-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578484#comment-15578484 ] Attila Szabo commented on SQOOP-3022: - Hi [~Tagar], Did you have chance to validate the solution? Are the results satisfying for you? Can we resolve this ticket, or do you have any additional requests here? Thanks, [~maugli] > sqoop export for Oracle generates tremendous amounts of redo logs > - > > Key: SQOOP-3022 > URL: https://issues.apache.org/jira/browse/SQOOP-3022 > Project: Sqoop > Issue Type: Bug > Components: codegen, connectors, connectors/oracle >Affects Versions: 1.4.3, 1.4.4, 1.4.5, 1.4.6 >Reporter: Ruslan Dautkhanov > Labels: export, oracle > > Sqoop export for Oracle generates tremendous amounts of redo logs (comparable > to export size or more). > We have put target tables in nologgin mode, but Oracle will still generate > redo logs unless +APPEND Oracle insert hint is used. > See https://oracle-base.com/articles/misc/append-hint for examples. > Please add an option for sqoop to generate insert statements in Oracle with > APPEND statement. Our databases are swamped with redo/archived logs whenever > we sqoop data to them. This is easily avoidable. And from business > prospective sqooping to staging tables in nologgin mode is totally fine. > Thank you. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SQOOP-3025) Oracle Direct export from HCatalog parquet table gives NPE in kerberized cluster
[ https://issues.apache.org/jira/browse/SQOOP-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578439#comment-15578439 ] Ruslan Dautkhanov commented on SQOOP-3025: -- Notice in the full log {quote} org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:svc_odiprd (auth:SIMPLE) cause:java.io.IOException: java.sql.SQLException: java.lang.NullPointerException {quote} Does OraOOP support Kerberos authentication? Not sure why it is showing *auth:SIMPLE* otherwise? auth:SIMPLE should not be used in a kerberized cluster? > Oracle Direct export from HCatalog parquet table gives NPE in kerberized > cluster > > > Key: SQOOP-3025 > URL: https://issues.apache.org/jira/browse/SQOOP-3025 > Project: Sqoop > Issue Type: Bug > Components: build, codegen, connectors/oracle, hive-integration, > metastore >Affects Versions: 1.4.6 > Environment: Hadoop 2.6; Sqoop 1.4.6; Oracle 12c >Reporter: Ruslan Dautkhanov > Attachments: scoop_output.log, > sqoop_OracleDirect_mapper_nullPointer_exception.txt > > > Getting following exception in one of the mappers stack when try to export to > Oracle from Hive Parquet > {noformat} > 2016-10-14 23:14:01,027 ERROR [main] > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: The following error > occurred during configurePreparedStatement() > java.lang.NullPointerException > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.setBindValueAtName(OraOopOutputFormatBase.java:432) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.configurePreparedStatementColumns(OraOopOutputFormatBase.java:547) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:246) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303) > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153) > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) > at > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > 2016-10-14 23:14:01,074 INFO [main] > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Time spent > performing an "exchange subpartition with table": 0.046365 sec. > 2016-10-14 23:14:01,074 DEBUG [main] > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Dropping temporary > mapper table "ORAOOP_20161014_231320_3" > 2016-10-14 23:14:01,131 INFO [main] org.apache.hadoop.mapred.MapTask: > Ignoring exception during close for > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@54848551 > java.io.IOException: java.sql.SQLException: java.lang.NullPointerException > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:197) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) > at > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at
[jira] [Updated] (SQOOP-3025) Oracle Direct export from HCatalog parquet table gives NPE in kerberized cluster
[ https://issues.apache.org/jira/browse/SQOOP-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruslan Dautkhanov updated SQOOP-3025: - Description: Getting following exception in one of the mappers stack when try to export to Oracle from Hive Parquet {noformat} 2016-10-14 23:14:01,027 ERROR [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: The following error occurred during configurePreparedStatement() java.lang.NullPointerException at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.setBindValueAtName(OraOopOutputFormatBase.java:432) at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.configurePreparedStatementColumns(OraOopOutputFormatBase.java:547) at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:246) at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303) at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153) at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194) at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2016-10-14 23:14:01,074 INFO [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Time spent performing an "exchange subpartition with table": 0.046365 sec. 2016-10-14 23:14:01,074 DEBUG [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Dropping temporary mapper table "ORAOOP_20161014_231320_3" 2016-10-14 23:14:01,131 INFO [main] org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@54848551 java.io.IOException: java.sql.SQLException: java.lang.NullPointerException at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:197) at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.sql.SQLException: java.lang.NullPointerException at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:255) at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303) at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153) at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194) {noformat} Will attach full log file as an attachment. A minimal sqoop export parameters that reproduce this problem: {quote} sqoop export -Dmapred.map.max.attempts=1 --connect "jdbc:oracle:thin:@//somehost:1355/adash_prod_batch" --username DISCOVER_STG_OWNER --password somePassword --table amf_trans_stg --hcatalog-database disc_dv --hcatalog-table amf_trans_dv_09142016 --num-mappers 2 --columns "individ,part_code" --direct --verbose {quote} was: Getting following exception in one of the mappers
[jira] [Updated] (SQOOP-3025) Oracle Direct export from HCatalog parquet table gives NPE in kerberized cluster
[ https://issues.apache.org/jira/browse/SQOOP-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruslan Dautkhanov updated SQOOP-3025: - Description: Getting following exception in one of the mappers stack when try to export to Oracle from Hive Parquet {noformat} 2016-10-14 23:14:01,027 ERROR [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: The following error occurred during configurePreparedStatement() java.lang.NullPointerException at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.setBindValueAtName(OraOopOutputFormatBase.java:432) at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.configurePreparedStatementColumns(OraOopOutputFormatBase.java:547) at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:246) at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303) at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153) at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194) at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2016-10-14 23:14:01,074 INFO [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Time spent performing an "exchange subpartition with table": 0.046365 sec. 2016-10-14 23:14:01,074 DEBUG [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Dropping temporary mapper table "ORAOOP_20161014_231320_3" 2016-10-14 23:14:01,131 INFO [main] org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@54848551 java.io.IOException: java.sql.SQLException: java.lang.NullPointerException at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:197) at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.sql.SQLException: java.lang.NullPointerException at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:255) at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303) at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153) at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194) {noformat} Will attach full log file as an attachment. A minimal sqoop export parameters that reproduce this problem: {noformat} sqoop export -Dmapred.map.max.attempts=1 --connect "jdbc:oracle:thin:@//somehost:1355/adash_prod_batch" --username DISCOVER_STG_OWNER --password somePassword --table amf_trans_stg --hcatalog-database disc_dv --hcatalog-table amf_trans_dv_09142016 --num-mappers 2 --columns "individ,part_code" --direct --verbose {noformat} was: Getting following exception in one of the
[jira] [Updated] (SQOOP-3025) Oracle Direct export from HCatalog parquet table gives NPE in kerberized cluster
[ https://issues.apache.org/jira/browse/SQOOP-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruslan Dautkhanov updated SQOOP-3025: - Attachment: scoop_output.log > Oracle Direct export from HCatalog parquet table gives NPE in kerberized > cluster > > > Key: SQOOP-3025 > URL: https://issues.apache.org/jira/browse/SQOOP-3025 > Project: Sqoop > Issue Type: Bug > Components: build, codegen, connectors/oracle, hive-integration, > metastore >Affects Versions: 1.4.6 > Environment: Hadoop 2.6; Sqoop 1.4.6; Oracle 12c >Reporter: Ruslan Dautkhanov > Attachments: scoop_output.log, > sqoop_OracleDirect_mapper_nullPointer_exception.txt > > > Getting following exception in one of the mappers stack when try to export to > Oracle from Hive Parquet > {noformat} > 2016-10-14 23:14:01,027 ERROR [main] > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: The following error > occurred during configurePreparedStatement() > java.lang.NullPointerException > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.setBindValueAtName(OraOopOutputFormatBase.java:432) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.configurePreparedStatementColumns(OraOopOutputFormatBase.java:547) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:246) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303) > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153) > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) > at > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > 2016-10-14 23:14:01,074 INFO [main] > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Time spent > performing an "exchange subpartition with table": 0.046365 sec. > 2016-10-14 23:14:01,074 DEBUG [main] > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Dropping temporary > mapper table "ORAOOP_20161014_231320_3" > 2016-10-14 23:14:01,131 INFO [main] org.apache.hadoop.mapred.MapTask: > Ignoring exception during close for > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@54848551 > java.io.IOException: java.sql.SQLException: java.lang.NullPointerException > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:197) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) > at > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.sql.SQLException: java.lang.NullPointerException > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:255) > at >
[jira] [Comment Edited] (SQOOP-3025) Oracle Direct export from HCatalog parquet table gives NPE in kerberized cluster
[ https://issues.apache.org/jira/browse/SQOOP-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578380#comment-15578380 ] Ruslan Dautkhanov edited comment on SQOOP-3025 at 10/15/16 4:25 PM: attached full log from failed mapper was (Author: tagar): full log from failed mapper > Oracle Direct export from HCatalog parquet table gives NPE in kerberized > cluster > > > Key: SQOOP-3025 > URL: https://issues.apache.org/jira/browse/SQOOP-3025 > Project: Sqoop > Issue Type: Bug > Components: build, codegen, connectors/oracle, hive-integration, > metastore >Affects Versions: 1.4.6 > Environment: Hadoop 2.6; Sqoop 1.4.6; Oracle 12c >Reporter: Ruslan Dautkhanov > Attachments: sqoop_OracleDirect_mapper_nullPointer_exception.txt > > > Getting following exception in one of the mappers stack when try to export to > Oracle from Hive Parquet > {noformat} > 2016-10-14 23:14:01,027 ERROR [main] > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: The following error > occurred during configurePreparedStatement() > java.lang.NullPointerException > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.setBindValueAtName(OraOopOutputFormatBase.java:432) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.configurePreparedStatementColumns(OraOopOutputFormatBase.java:547) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:246) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303) > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153) > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) > at > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > 2016-10-14 23:14:01,074 INFO [main] > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Time spent > performing an "exchange subpartition with table": 0.046365 sec. > 2016-10-14 23:14:01,074 DEBUG [main] > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Dropping temporary > mapper table "ORAOOP_20161014_231320_3" > 2016-10-14 23:14:01,131 INFO [main] org.apache.hadoop.mapred.MapTask: > Ignoring exception during close for > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@54848551 > java.io.IOException: java.sql.SQLException: java.lang.NullPointerException > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:197) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) > at > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.sql.SQLException: java.lang.NullPointerException > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:255) >
[jira] [Updated] (SQOOP-3025) Oracle Direct export from HCatalog parquet table gives NPE in kerberized cluster
[ https://issues.apache.org/jira/browse/SQOOP-3025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruslan Dautkhanov updated SQOOP-3025: - Attachment: sqoop_OracleDirect_mapper_nullPointer_exception.txt full log from failed mapper > Oracle Direct export from HCatalog parquet table gives NPE in kerberized > cluster > > > Key: SQOOP-3025 > URL: https://issues.apache.org/jira/browse/SQOOP-3025 > Project: Sqoop > Issue Type: Bug > Components: build, codegen, connectors/oracle, hive-integration, > metastore >Affects Versions: 1.4.6 > Environment: Hadoop 2.6; Sqoop 1.4.6; Oracle 12c >Reporter: Ruslan Dautkhanov > Attachments: sqoop_OracleDirect_mapper_nullPointer_exception.txt > > > Getting following exception in one of the mappers stack when try to export to > Oracle from Hive Parquet > {noformat} > 2016-10-14 23:14:01,027 ERROR [main] > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: The following error > occurred during configurePreparedStatement() > java.lang.NullPointerException > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.setBindValueAtName(OraOopOutputFormatBase.java:432) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.configurePreparedStatementColumns(OraOopOutputFormatBase.java:547) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:246) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303) > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153) > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) > at > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > 2016-10-14 23:14:01,074 INFO [main] > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Time spent > performing an "exchange subpartition with table": 0.046365 sec. > 2016-10-14 23:14:01,074 DEBUG [main] > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Dropping temporary > mapper table "ORAOOP_20161014_231320_3" > 2016-10-14 23:14:01,131 INFO [main] org.apache.hadoop.mapred.MapTask: > Ignoring exception during close for > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@54848551 > java.io.IOException: java.sql.SQLException: java.lang.NullPointerException > at > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:197) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) > at > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) > at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > Caused by: java.sql.SQLException: java.lang.NullPointerException > at > org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:255) > at >
[jira] [Created] (SQOOP-3025) Oracle Direct export from HCatalog parquet table gives NPE in kerberized cluster
Ruslan Dautkhanov created SQOOP-3025: Summary: Oracle Direct export from HCatalog parquet table gives NPE in kerberized cluster Key: SQOOP-3025 URL: https://issues.apache.org/jira/browse/SQOOP-3025 Project: Sqoop Issue Type: Bug Components: build, codegen, connectors/oracle, hive-integration, metastore Affects Versions: 1.4.6 Environment: Hadoop 2.6; Sqoop 1.4.6; Oracle 12c Reporter: Ruslan Dautkhanov Attachments: sqoop_OracleDirect_mapper_nullPointer_exception.txt Getting following exception in one of the mappers stack when try to export to Oracle from Hive Parquet {noformat} 2016-10-14 23:14:01,027 ERROR [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: The following error occurred during configurePreparedStatement() java.lang.NullPointerException at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.setBindValueAtName(OraOopOutputFormatBase.java:432) at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.configurePreparedStatementColumns(OraOopOutputFormatBase.java:547) at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:246) at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303) at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153) at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194) at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 2016-10-14 23:14:01,074 INFO [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Time spent performing an "exchange subpartition with table": 0.046365 sec. 2016-10-14 23:14:01,074 DEBUG [main] org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert: Dropping temporary mapper table "ORAOOP_20161014_231320_3" 2016-10-14 23:14:01,131 INFO [main] org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector@54848551 java.io.IOException: java.sql.SQLException: java.lang.NullPointerException at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:197) at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.close(OraOopOutputFormatBase.java:578) at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.close(OraOopOutputFormatInsert.java:107) at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:670) at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:2016) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:797) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.sql.SQLException: java.lang.NullPointerException at org.apache.sqoop.manager.oracle.OraOopOutputFormatInsert$OraOopDBRecordWriterInsert.configurePreparedStatement(OraOopOutputFormatInsert.java:255) at org.apache.sqoop.manager.oracle.OraOopOutputFormatBase$OraOopDBRecordWriterBase.getPreparedStatement(OraOopOutputFormatBase.java:303) at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:153) at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:194) {noformat} Will attach full log file as an attachment. Sqoop export options used: {noformat}