I was able to successfully import data from Netezza to HDFS with & without the
"--direct" option. I'm also able to export data to Netezza from HDFS
successfully without the "-direct' option.
But SQOOP fails when I try to export data to Netezza from HDFS with the
"-direct" option. It surprisingly does not recognize the "owner" of the table
properly. Can you please help check the issue?
NETEZZA DBMS Version: Release 7.0.4.3-P2 [Build 35925]
NETEZZA driver: nzjdbc3.jar
Here is the SQOOP statement & the error;
/sqoop_hdp2/bin/sqoop export
--direct
--connect "jdbc:netezza://netezza-server:5480/NZ_DB;schema=ADMIN;"
--username nzuser --password nzpass
--export-dir /user/nzuser/NZ_DB/NZ_TBL_GEOG --table "NZ_TBL_GEOG"
--num-mappers 8
--input-fields-terminated-by ","
--input-escaped-by '\\'
--verbose
Error:
14/07/25 00:56:45 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4.1
14/07/25 00:56:45 DEBUG tool.BaseSqoopTool: Enabled debug logging.
14/07/25 00:56:45 WARN tool.BaseSqoopTool: Setting your password on the
command-line is insecure. Consider using -P instead.
14/07/25 00:56:45 DEBUG sqoop.ConnFactory: Loaded manager factory:
com.cloudera.sqoop.manager.DefaultManagerFactory
14/07/25 00:56:45 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
com.cloudera.sqoop.manager.DefaultManagerFactory
14/07/25 00:56:45 DEBUG manager.DefaultManagerFactory: Trying with scheme:
jdbc:netezza:
14/07/25 00:56:45 INFO manager.SqlManager: Using default fetchSize of 1000
14/07/25 00:56:45 DEBUG sqoop.ConnFactory: Instantiated ConnManager
org.apache.sqoop.manager.DirectNetezzaManager@643c0007
14/07/25 00:56:45 INFO tool.CodeGenTool: The connection manager declares that
it self manages mapping between records & fields and rows & columns. No class
will will be generated.
14/07/25 00:56:46 DEBUG manager.SqlManager: No connection paramenters
specified. Using regular API for making connection.
14/07/25 00:56:47 ERROR manager.DirectNetezzaManager: NZ_TBL_GEOG is not a
valid Netezza table. Please make sure that you have connected to the Netezza
DB and the table name is right. The current values are
connection string : jdbc:netezza://
netezza-server:5480/NZ_DB;schema=admin;
table owner : shyam (windows network login ID)
table name : NZ_TBL_GEOG
Now, I removed the "schema" from the JDBC connection string and added "owner"
name to the table name in the "-table" switch. The owner name of the table
repeats itself 2 times in the error log below. "NZ_DB.ADMIN.ADMIN.NZ_TBL_GEOG".
Here is the SQOOP statement;
/sqoop_hdp2/bin/sqoop export
--direct
--connect "jdbc:netezza://netezza-server:5480/NZ_DB"
--username nzuser --password nzpass
--export-dir /user/nzuser/NZ_DB/NZ_TBL_GEOG
--table "ADMIN.NZ_TBL_GEOG"
--num-mappers 8
--input-fields-terminated-by ","
--input-escaped-by '\\'
--verbose
Error:
14/07/25 01:09:07 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4.1
14/07/25 01:09:07 DEBUG tool.BaseSqoopTool: Enabled debug logging.
14/07/25 01:09:07 WARN tool.BaseSqoopTool: Setting your password on the
command-line is insecure. Consider using -P instead.
14/07/25 01:09:07 DEBUG sqoop.ConnFactory: Loaded manager factory:
com.cloudera.sqoop.manager.DefaultManagerFactory
14/07/25 01:09:07 DEBUG sqoop.ConnFactory: Trying ManagerFactory:
com.cloudera.sqoop.manager.DefaultManagerFactory
14/07/25 01:09:07 DEBUG manager.DefaultManagerFactory: Trying with scheme:
jdbc:netezza:
14/07/25 01:09:07 INFO manager.SqlManager: Using default fetchSize of
1000
14/07/25 01:09:07 DEBUG sqoop.ConnFactory: Instantiated ConnManager
org.apache.sqoop.manager.DirectNetezzaManager@4cc39a20
14/07/25 01:09:07 INFO tool.CodeGenTool: The connection manager declares that
it self manages mapping between records & fields and rows & columns. No class
will will be generated.
14/07/25 01:09:07 DEBUG manager.SqlManager: No connection paramenters
specified. Using regular API for making connection.
14/07/25 01:09:09 INFO mapreduce.ExportJobBase: Beginning export of
ADMIN.NZ_TBL_GEOG
14/07/25 01:09:09 DEBUG util.ClassLoaderStack: Attempting to load jar through
URL:
jar:file:/usr/lib/hadoop/hadoop-common-2.2.0.2.0.6.0-102.jar!/
14/07/25 01:09:09 DEBUG util.ClassLoaderStack: Previous classloader is
sun.misc.Launcher$AppClassLoader@546b97fd
14/07/25 01:09:09 DEBUG util.ClassLoaderStack: Loaded jar into current
JVM: jar:file:/usr/lib/hadoop/hadoop-common-2.2.0.2.0.6.0-102.jar!/
14/07/25 01:09:09 DEBUG util.ClassLoaderStack: Added classloader for
jar /usr/lib/hadoop/hadoop-common-2.2.0.2.0.6.0-102.jar:
java.net.FactoryURLClassLoader@6655bb93
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/
StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/hive/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/St
aticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/07/25 01:09:09 INFO Configuration.deprecation: mapred.jar is
deprecated. Instead, use mapreduce.job.jar
14/07/25 01:09:09 DEBUG mapreduce.JobBase: Using InputFormat: class
org.apache.sqoop.mapreduce.ExportInputFormat
14/07/25 01:09:10 DEBUG db.DBConfiguration: Securing password into job
credentials store
14/07/25 01:09:10 DEBUG manager.SqlManager: No connection paramenters
specified. Using regular API for making connection.
14/07/25 01:09:11 ERROR manager.SqlManager: Error executing statement:
org.netezza.error.NzSQLException: ERROR: relation does not exist:
NZ_DB.ADMIN.ADMIN.NZ_TBL_GEOG
at
org.netezza.internal.QueryExecutor.getNextResult(QueryExecutor.java:280)
at org.netezza.internal.QueryExecutor.execute(QueryExecutor.java:76)
at org.netezza.sql.NzConnection.execute(NzConnection.java:2819)
at
org.netezza.sql.NzPreparedStatament._execute(NzPreparedStatament.java:1126)
at
org.netezza.sql.NzPreparedStatament.prepare(NzPreparedStatament.java:1143)
at
org.netezza.sql.NzPreparedStatament.<init(NzPreparedStatament.java:89)
at
org.netezza.sql.NzConnection.prepareStatement(NzConnection.java:1588)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:689)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:712)
at
org.apache.sqoop.manager.SqlManager.getColumnNamesForRawQuery(SqlManager.java:132)
at
org.apache.sqoop.manager.SqlManager.getColumnNames(SqlManager.java:116)
at
org.apache.sqoop.mapreduce.netezza.NetezzaExternalTableExportJob.configureOutputFormat(NetezzaExternalTableExportJob.java:128)
at
org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:398)
at
org.apache.sqoop.manager.DirectNetezzaManager.exportTable(DirectNetezzaManager.java:179)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:81)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
14/07/25 01:09:11 DEBUG util.ClassLoaderStack: Restoring classloader:
sun.misc.Launcher$AppClassLoader@546b97fd
14/07/25 01:09:11 ERROR sqoop.Sqoop: Got exception running Sqoop:
java.lang.NullPointerException java.lang.NullPointerException
at
org.apache.sqoop.mapreduce.db.DBOutputFormat.setOutput(DBOutputFormat.java:139)
at
org.apache.sqoop.mapreduce.netezza.NetezzaExternalTableExportJob.configureOutputFormat(NetezzaExternalTableExportJob.java:134)
at
org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:398)
at
org.apache.sqoop.manager.DirectNetezzaManager.exportTable(DirectNetezzaManager.java:179)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:81)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
Thanks,
Shyam Munjuluri
Integrated Systems Engineering
Office: (M, W, F) 860-273-0595
WAH: (Tu, Th) 860-404-5219
This e-mail may contain confidential or privileged information. If
you think you have received this e-mail in error, please advise the
sender by reply e-mail and then delete this e-mail immediately.
Thank you. Aetna
