Hi Muthu,
the given message is a “Warning” and not an “Error”. Sqoop is warning you that
you are using —driver option that will force Sqoop to use Generic JDBC
connector. That is fine in your case as Sqoop don’t have specialized connector
for Vertica. You can always use --connect-manager parameter to get rid of the
message as is the warning suggesting:
sqoop export --connect-manager
org.apache.sqoop.manager.GenericJdbcManager ...
You are however hitting a different problem, my best guess that Vertica is not
supporting multi value insert statements that Sqoop is generating by default.
Try to disable this feature using property sqoop.export.records.per.statement:
sqoop export -Dsqoop.export.records.per.statement=1 —connect ...
Jarcec
On Aug 25, 2014, at 6:55 AM, Muthu Pandi <[email protected]> wrote:
> Hi every one,
> Is it possible to export data from HDFS to vetica.I
> successfully imported data from vertica to HDFS using sqoop.When i Export i
> get the following error.Please give me solution as soon as possible.this is
> the error which am getting,
>
> WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver
> however appropriate connection manager is not being set (via
> --connection-manager). Sqoop is going to fall back to
> org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which
> connection manager should be used next time.
>
> will above will have impact.How do i create connection manager for vertica.
>
> My sqoop query is
>
> sqoop export --driver com.vertica.jdbc.Driver --connect
> "jdbc:vertica://X.X.X.X:5433/dbname" --username username -P --table tablename
> -m 1 --export-dir /ver.txt --input-fields-terminated-by ','
> --lines-terminated-by '\n'
>
>
> and am using hadoop2.4.1 and sqoop1.4.4 version
>
>
>
> 14/08/25 10:12:57 WARN tool.BaseSqoopTool: Setting your password on the
> command-line is insecure. Consider using -P instead.
> 14/08/25 10:12:57 WARN sqoop.ConnFactory: Parameter --driver is set to an
> explicit driver however appropriate connection manager is not being set (via
> --connection-manager). Sqoop is going to fall back to
> org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which
> connection manager should be used next time.
> 14/08/25 10:12:57 INFO manager.SqlManager: Using default fetchSize of 1000
> 14/08/25 10:12:57 INFO tool.CodeGenTool: Beginning code generation
> 14/08/25 10:12:58 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM AGE AS t WHERE 1=0
> 14/08/25 10:12:58 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM AGE AS t WHERE 1=0
> 14/08/25 10:12:58 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is
> /opt/hadoop-2.4.1
> Note: /tmp/sqoop-hadoop2/compile/f0a438b0e6007fea6da95808e5f9468e/AGE.java
> uses or overrides a deprecated API.
> Note: Recompile with -Xlint:deprecation for details.
> 14/08/25 10:13:02 INFO orm.CompilationManager: Writing jar file:
> /tmp/sqoop-hadoop2/compile/f0a438b0e6007fea6da95808e5f9468e/AGE.jar
> 14/08/25 10:13:03 INFO mapreduce.ExportJobBase: Beginning export of AGE
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/opt/hadoop-2.4.1/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> 14/08/25 10:13:03 INFO Configuration.deprecation: mapred.jar is deprecated.
> Instead, use mapreduce.job.jar
> 14/08/25 10:13:06 INFO manager.SqlManager: Executing SQL statement: SELECT
> t.* FROM AGE AS t WHERE 1=0
> 14/08/25 10:13:06 INFO Configuration.deprecation:
> mapred.reduce.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.reduce.speculative
> 14/08/25 10:13:06 INFO Configuration.deprecation:
> mapred.map.tasks.speculative.execution is deprecated. Instead, use
> mapreduce.map.speculative
> 14/08/25 10:13:06 INFO Configuration.deprecation: mapred.map.tasks is
> deprecated. Instead, use mapreduce.job.maps
> 14/08/25 10:13:06 INFO client.RMProxy: Connecting to ResourceManager at
> nn01/10.10.200.74:8032
> 14/08/25 10:13:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/08/25 10:13:27 INFO input.FileInputFormat: Total input paths to process : 1
> 14/08/25 10:13:27 INFO mapreduce.JobSubmitter: number of splits:1
> 14/08/25 10:13:28 INFO mapreduce.JobSubmitter: Submitting tokens for job:
> job_1408941947869_0001
> 14/08/25 10:13:30 INFO impl.YarnClientImpl: Submitted application
> application_1408941947869_0001
> 14/08/25 10:13:30 INFO mapreduce.Job: The url to track the job:
> http://nn01:8088/proxy/application_1408941947869_0001/
> 14/08/25 10:13:30 INFO mapreduce.Job: Running job: job_1408941947869_0001
> 14/08/25 10:13:44 INFO mapreduce.Job: Job job_1408941947869_0001 running in
> uber mode : false
> 14/08/25 10:13:44 INFO mapreduce.Job: map 0% reduce 0%
> 14/08/25 10:14:11 INFO mapreduce.Job: map 100% reduce 0%
> 14/08/25 10:14:11 INFO mapreduce.Job: Task Id :
> attempt_1408941947869_0001_m_000000_0, Status : FAILED
> Error: java.io.IOException: java.sql.SQLException: [Vertica][VJDBC](4856)
> ERROR: Syntax error at or near ","
> at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.sql.SQLException: [Vertica][VJDBC](4856) ERROR: Syntax error
> at or near ","
> at com.vertica.util.ServerErrorData.buildException(Unknown Source)
> at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown Source)
> at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown Source)
> at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> at com.vertica.jdbc.SPreparedStatement.<init>(Unknown Source)
> at com.vertica.jdbc.VerticaPreparedStatementImpl.<init>(Unknown Source)
> at com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown
> Source)
> at com.vertica.jdbc.SConnection.prepareStatement(Unknown Source)
> at
> org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.getPreparedStatement(ExportOutputFormat.java:141)
> at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149)
> at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181)
> ... 8 more
> Caused by: com.vertica.util.ServerException: [Vertica][VJDBC](4856) ERROR:
> Syntax error at or near ","
> ... 20 more
>
> 14/08/25 10:14:12 INFO mapreduce.Job: map 0% reduce 0%
> 14/08/25 10:14:25 INFO mapreduce.Job: Task Id :
> attempt_1408941947869_0001_m_000000_1, Status : FAILED
> Error: java.io.IOException: java.sql.SQLException: [Vertica][VJDBC](4856)
> ERROR: Syntax error at or near ","
> at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.sql.SQLException: [Vertica][VJDBC](4856) ERROR: Syntax error
> at or near ","
> at com.vertica.util.ServerErrorData.buildException(Unknown Source)
> at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown Source)
> at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown Source)
> at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> at com.vertica.jdbc.SPreparedStatement.<init>(Unknown Source)
> at com.vertica.jdbc.VerticaPreparedStatementImpl.<init>(Unknown Source)
> at com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown
> Source)
> at com.vertica.jdbc.SConnection.prepareStatement(Unknown Source)
> at
> org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.getPreparedStatement(ExportOutputFormat.java:141)
> at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149)
> at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181)
> ... 8 more
> Caused by: com.vertica.util.ServerException: [Vertica][VJDBC](4856) ERROR:
> Syntax error at or near ","
> ... 20 more
>
> 14/08/25 10:14:34 INFO mapreduce.Job: Task Id :
> attempt_1408941947869_0001_m_000000_2, Status : FAILED
> Error: java.io.IOException: java.sql.SQLSyntaxErrorException:
> [Vertica][VJDBC](4856) ERROR: Syntax error at or near ","
> at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:647)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> Caused by: java.sql.SQLSyntaxErrorException: [Vertica][VJDBC](4856) ERROR:
> Syntax error at or near ","
> at com.vertica.util.ServerErrorData.buildException(Unknown Source)
> at com.vertica.io.ProtocolStream.readExpectedMessage(Unknown Source)
> at com.vertica.dataengine.VDataEngine.prepareImpl(Unknown Source)
> at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> at com.vertica.dataengine.VDataEngine.prepare(Unknown Source)
> at com.vertica.jdbc.common.SPreparedStatement.<init>(Unknown Source)
> at com.vertica.jdbc.jdbc4.S4PreparedStatement.<init>(Unknown Source)
> at com.vertica.jdbc.VerticaJdbc4PreparedStatementImpl.<init>(Unknown
> Source)
> at com.vertica.jdbc.VJDBCObjectFactory.createPreparedStatement(Unknown
> Source)
> at com.vertica.jdbc.common.SConnection.prepareStatement(Unknown Source)
> at
> org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.getPreparedStatement(ExportOutputFormat.java:141)
> at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149)
> at
> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181)
> ... 8 more
> Caused by: com.vertica.support.exceptions.SyntaxErrorException:
> [Vertica][VJDBC](4856) ERROR: Syntax error at or near ","
> ... 21 more
>
> 14/08/25 10:14:43 INFO mapreduce.Job: map 100% reduce 0%
> 14/08/25 10:14:44 INFO mapreduce.Job: Job job_1408941947869_0001 failed with
> state FAILED due to: Task failed task_1408941947869_0001_m_000000
> Job failed as tasks failed. failedMaps:1 failedReduces:0
>
> 14/08/25 10:14:45 INFO mapreduce.Job: Counters: 9
> Job Counters
> Failed map tasks=4
> Launched map tasks=4
> Other local map tasks=3
> Rack-local map tasks=1
> Total time spent by all maps in occupied slots (ms)=53177
> Total time spent by all reduces in occupied slots (ms)=0
> Total time spent by all map tasks (ms)=53177
> Total vcore-seconds taken by all map tasks=53177
> Total megabyte-seconds taken by all map tasks=54453248
> 14/08/25 10:14:45 WARN mapreduce.Counters: Group FileSystemCounters is
> deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
> 14/08/25 10:14:45 INFO mapreduce.ExportJobBase: Transferred 0 bytes in
> 99.0548 seconds (0 bytes/sec)
> 14/08/25 10:14:45 WARN mapreduce.Counters: Group
> org.apache.hadoop.mapred.Task$Counter is deprecated. Use
> org.apache.hadoop.mapreduce.TaskCounter instead
> 14/08/25 10:14:45 INFO mapreduce.ExportJobBase: Exported 0 records.
> 14/08/25 10:14:45 ERROR tool.ExportTool: Error during export: Export job
> failed!
> Regards
> Muthupandi.K
>
>
>
> <image002.png>
>
