Here's what I use: sqoop export -Doraoop.export.merge=true -Doraoop.update.key.extra.columns="period_start_time" -D mapred.map.max.attempts=1 -D sqoop.export.records.per.statement=1000 --connect ... --username user --password pass --table shapira2 --export-dir /user/hive/warehouse/tables/shapira --verbose -m 16 --update-key co_gid
On Tue, Jul 29, 2014 at 11:37 AM, Leonardo Brambilla <[email protected]> wrote: > Hello there, does anybody here use Sqoop to export and update to Oracle DB? > can you please share the command you use? > thanks > > > On Mon, Jul 21, 2014 at 10:40 AM, Leonardo Brambilla > <[email protected]> wrote: >> >> Hello, any thoughts on this? >> Can someone tell me if possible to run sqoop export and specify a jar file >> and the main class? I tried "--jar-file" and "--class-name" without luck. >> Does anyone have an example of a running command using jar and class? >> Thanks >> >> >> On Fri, Jul 18, 2014 at 1:59 PM, Leonardo Brambilla >> <[email protected]> wrote: >>> >>> Yes, I'm using all upper case, and you can see in the log extract that >>> the query built is: >>> 14/07/18 12:45:04 INFO manager.SqlManager: Executing SQL statement: >>> SELECT t.* FROM SEARCH_KEYWORDS_AGGREGATION t WHERE 1=0 >>> >>> which is fine, and to be sure I copy pasted it into sqldeveloper and it >>> runs ok. >>> Maybe Sqoop is having problems parsing the result from oracle in my >>> environment, not sure. >>> >>> thanks >>> >>> >>> On Fri, Jul 18, 2014 at 1:53 PM, Gwen Shapira <[email protected]> >>> wrote: >>>> >>>> Looks like Sqoop is failing to fetch column list from the table. Can >>>> you check that $table is ALL UPPER CASE? >>>> We are doing "select ... from user_tab_columns where >>>> table_name='$table'; >>>> >>>> Oracle stores tables as upper case, so $table has to match. >>>> >>>> Gwen >>>> >>>> On Fri, Jul 18, 2014 at 9:49 AM, Leonardo Brambilla >>>> <[email protected]> wrote: >>>> > Well, if I try to run the command without --columns I get an error: >>>> > Attempted to generate class with no columns! >>>> > Part of the log: >>>> > 14/07/18 12:45:04 INFO tool.CodeGenTool: Beginning code generation >>>> > 14/07/18 12:45:04 DEBUG manager.OracleManager: Using column names >>>> > query: >>>> > SELECT t.* FROM SEARCH_KEYWORDS_AGGREGATION t WHERE 1=0 >>>> > 14/07/18 12:45:04 DEBUG manager.SqlManager: Execute >>>> > getColumnTypesRawQuery : >>>> > SELECT t.* FROM SEARCH_KEYWORDS_AGGREGATION t WHERE 1=0 >>>> > 14/07/18 12:45:04 DEBUG manager.OracleManager$ConnCache: Got cached >>>> > connection for jdbc:oracle:thin:@devbox.com:1541/devdb/uDev >>>> > 14/07/18 12:45:04 INFO manager.OracleManager: Time zone has been set >>>> > to GMT >>>> > 14/07/18 12:45:04 DEBUG manager.SqlManager: Using fetchSize for next >>>> > query: >>>> > 1000 >>>> > 14/07/18 12:45:04 INFO manager.SqlManager: Executing SQL statement: >>>> > SELECT >>>> > t.* FROM SEARCH_KEYWORDS_AGGREGATION t WHERE 1=0 >>>> > 14/07/18 12:45:04 DEBUG manager.OracleManager$ConnCache: Caching >>>> > released >>>> > connection for jdbc:oracle:thin:@devbox.com:1541/devdb/uDev >>>> > 14/07/18 12:45:04 DEBUG orm.ClassWriter: selected columns: >>>> > 14/07/18 12:45:04 DEBUG orm.ClassWriter: db write column order: >>>> > 14/07/18 12:45:04 DEBUG orm.ClassWriter: SEARCH_DATE [from >>>> > --update-key >>>> > parameter] >>>> > 14/07/18 12:45:04 ERROR sqoop.Sqoop: Got exception running Sqoop: >>>> > java.lang.IllegalArgumentException: Attempted to generate class with >>>> > no >>>> > columns! >>>> > java.lang.IllegalArgumentException: Attempted to generate class with >>>> > no >>>> > columns! >>>> > at >>>> > >>>> > org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1295) >>>> > at >>>> > org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1176) >>>> > at >>>> > org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96) >>>> > at >>>> > org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:64) >>>> > at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:100) >>>> > at org.apache.sqoop.Sqoop.run(Sqoop.java:145) >>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) >>>> > at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181) >>>> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220) >>>> > at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229) >>>> > at org.apache.sqoop.Sqoop.main(Sqoop.java:238) >>>> > >>>> > look in the log there is a "selected columns: " (empty) >>>> > >>>> > >>>> > On Fri, Jul 18, 2014 at 1:13 PM, Gwen Shapira <[email protected]> >>>> > wrote: >>>> >> >>>> >> If we omit --columns the source file has to match the DB in number of >>>> >> columns, types and order. >>>> >> >>>> >> On Fri, Jul 18, 2014 at 8:23 AM, Leonardo Brambilla >>>> >> <[email protected]> wrote: >>>> >> > Hello some more findings, could this bug be related to my problem? >>>> >> > https://issues.apache.org/jira/browse/SQOOP-824 >>>> >> > I know it says it's fixed since 1.4.3 but maybe that brought some >>>> >> > other >>>> >> > case. >>>> >> > I still don't understand how to run export command without >>>> >> > specifying >>>> >> > the >>>> >> > --columns parameter, can you tell me what is the default behavior >>>> >> > when >>>> >> > you >>>> >> > omit --columns? Does the source file need to have the same column >>>> >> > order >>>> >> > than >>>> >> > the target table? >>>> >> > >>>> >> > Thanks >>>> >> > >>>> >> > >>>> >> > On Fri, Jul 18, 2014 at 12:46 AM, Leonardo Brambilla >>>> >> > <[email protected]> wrote: >>>> >> >> >>>> >> >> I think I found something. The java class generated when using >>>> >> >> update-key >>>> >> >> differs from the one without update-key. The one that throws >>>> >> >> exception >>>> >> >> is >>>> >> >> missing to write the fields that are not specified in the >>>> >> >> update-key. I >>>> >> >> also >>>> >> >> see (with --verbose) that when using --update-key there is an >>>> >> >> extra >>>> >> >> debug >>>> >> >> line that says >>>> >> >> 14/07/17 22:53:27 DEBUG orm.ClassWriter: db write column order: >>>> >> >> 14/07/17 22:53:27 DEBUG orm.ClassWriter: SEARCH_DATE >>>> >> >> >>>> >> >> Below is the method generated for the command without --update-key >>>> >> >> >>>> >> >> public int write(PreparedStatement __dbStmt, int __off) throws >>>> >> >> SQLException { >>>> >> >> JdbcWritableBridge.writeTimestamp(SEARCH_DATE, 1 + __off, 93, >>>> >> >> __dbStmt); >>>> >> >> JdbcWritableBridge.writeString(SEARCH_TYPE, 2 + __off, 12, >>>> >> >> __dbStmt); >>>> >> >> JdbcWritableBridge.writeString(USER_AGENT, 3 + __off, 12, >>>> >> >> __dbStmt); >>>> >> >> JdbcWritableBridge.writeString(SRCH_KEYWORD, 4 + __off, 12, >>>> >> >> __dbStmt); >>>> >> >> JdbcWritableBridge.writeBigDecimal(SRCH_COUNT, 5 + __off, 2, >>>> >> >> __dbStmt); >>>> >> >> return 5; >>>> >> >> } >>>> >> >> >>>> >> >> Below is the one generated for the command with --update-key >>>> >> >> public int write(PreparedStatement __dbStmt, int __off) throws >>>> >> >> SQLException { >>>> >> >> JdbcWritableBridge.writeTimestamp(SEARCH_DATE, 1 + __off, 93, >>>> >> >> __dbStmt); >>>> >> >> return 1; >>>> >> >> } >>>> >> >> >>>> >> >> I tried to force export to use the properly generated class with >>>> >> >> parameters "jar-file" and "class-name" but that didn't work, if >>>> >> >> like >>>> >> >> those >>>> >> >> params are not allowed in the export command. This is what I tried >>>> >> >> to >>>> >> >> force >>>> >> >> using the properly generated source >>>> >> >> sqoop export \ >>>> >> >> --connect jdbc:oracle:thin:@ddb04.local.com:1541/test04 \ >>>> >> >> --update-key "SEARCH_DATE" \ >>>> >> >> --columns $columns \ >>>> >> >> --table $table --username $user --password $passwd \ >>>> >> >> --fields-terminated-by "=" --export-dir $exportDir >>>> >> >> --jar-file SEARCH_TABLE.jar --class-name SEARCH_TABLE >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> On Thu, Jul 17, 2014 at 5:04 PM, Leonardo Brambilla >>>> >> >> <[email protected]> wrote: >>>> >> >>> >>>> >> >>> Yes, the update-key is a subset of columns. >>>> >> >>> >>>> >> >>> >>>> >> >>> On Thu, Jul 17, 2014 at 4:16 PM, Gwen Shapira >>>> >> >>> <[email protected]> >>>> >> >>> wrote: >>>> >> >>>> >>>> >> >>>> Does the update column appear in $columns? It should be in >>>> >> >>>> there. >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> On Thu, Jul 17, 2014 at 10:48 AM, Leonardo Brambilla >>>> >> >>>> <[email protected]> wrote: >>>> >> >>>>> >>>> >> >>>>> Hi Gwen, thank you for replying. >>>> >> >>>>> >>>> >> >>>>> I went to the data node, the userlogs and all I found in syslog >>>> >> >>>>> file >>>> >> >>>>> is >>>> >> >>>>> what I already posted: >>>> >> >>>>> 2014-07-17 10:19:09,280 INFO >>>> >> >>>>> org.apache.hadoop.util.NativeCodeLoader: >>>> >> >>>>> Loaded the native-hadoop library >>>> >> >>>>> 2014-07-17 10:19:09,700 INFO >>>> >> >>>>> org.apache.hadoop.util.ProcessTree: >>>> >> >>>>> setsid >>>> >> >>>>> exited with exit code 0 >>>> >> >>>>> 2014-07-17 10:19:09,706 INFO org.apache.hadoop.mapred.Task: >>>> >> >>>>> Using >>>> >> >>>>> ResourceCalculatorPlugin : >>>> >> >>>>> org.apache.hadoop.util.LinuxResourceCalculatorPlugin@34c3a7c0 >>>> >> >>>>> 2014-07-17 10:19:10,266 INFO >>>> >> >>>>> org.apache.sqoop.mapreduce.AutoProgressMapper: Auto-progress >>>> >> >>>>> thread >>>> >> >>>>> is >>>> >> >>>>> finished. keepGoing=false >>>> >> >>>>> 2014-07-17 10:19:10,476 INFO >>>> >> >>>>> org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' >>>> >> >>>>> truncater >>>> >> >>>>> with mapRetainSize=-1 and reduceRetainSize=-1 >>>> >> >>>>> 2014-07-17 10:19:10,537 INFO >>>> >> >>>>> org.apache.hadoop.io.nativeio.NativeIO: >>>> >> >>>>> Initialized cache for UID to User mapping with a cache timeout >>>> >> >>>>> of >>>> >> >>>>> 14400 >>>> >> >>>>> seconds. >>>> >> >>>>> 2014-07-17 10:19:10,537 INFO >>>> >> >>>>> org.apache.hadoop.io.nativeio.NativeIO: >>>> >> >>>>> Got UserName elance for UID 666 from the native implementation >>>> >> >>>>> 2014-07-17 10:19:10,539 ERROR >>>> >> >>>>> org.apache.hadoop.security.UserGroupInformation: >>>> >> >>>>> PriviledgedActionException >>>> >> >>>>> as:elance cause:java.io.IOException: java.sql.SQLException: >>>> >> >>>>> Missing >>>> >> >>>>> IN or >>>> >> >>>>> OUT parameter at index:: 2 >>>> >> >>>>> 2014-07-17 10:19:10,540 WARN org.apache.hadoop.mapred.Child: >>>> >> >>>>> Error >>>> >> >>>>> running child >>>> >> >>>>> java.io.IOException: java.sql.SQLException: Missing IN or OUT >>>> >> >>>>> parameter >>>> >> >>>>> at index:: 2 >>>> >> >>>>> at >>>> >> >>>>> >>>> >> >>>>> >>>> >> >>>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184) >>>> >> >>>>> at >>>> >> >>>>> >>>> >> >>>>> >>>> >> >>>>> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:651) >>>> >> >>>>> at >>>> >> >>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766) >>>> >> >>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) >>>> >> >>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >>>> >> >>>>> at java.security.AccessController.doPrivileged(Native Method) >>>> >> >>>>> at javax.security.auth.Subject.doAs(Subject.java:415) >>>> >> >>>>> at >>>> >> >>>>> >>>> >> >>>>> >>>> >> >>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) >>>> >> >>>>> at org.apache.hadoop.mapred.Child.main(Child.java:249) >>>> >> >>>>> Caused by: java.sql.SQLException: Missing IN or OUT parameter >>>> >> >>>>> at >>>> >> >>>>> index:: 2 >>>> >> >>>>> at >>>> >> >>>>> >>>> >> >>>>> >>>> >> >>>>> oracle.jdbc.driver.OraclePreparedStatement.processCompletedBindRow(OraclePreparedStatement.java:1844) >>>> >> >>>>> at >>>> >> >>>>> >>>> >> >>>>> >>>> >> >>>>> oracle.jdbc.driver.OraclePreparedStatement.addBatch(OraclePreparedStatement.java:10213) >>>> >> >>>>> at >>>> >> >>>>> >>>> >> >>>>> >>>> >> >>>>> oracle.jdbc.driver.OraclePreparedStatementWrapper.addBatch(OraclePreparedStatementWrapper.java:1362) >>>> >> >>>>> at >>>> >> >>>>> >>>> >> >>>>> >>>> >> >>>>> org.apache.sqoop.mapreduce.UpdateOutputFormat$UpdateRecordWriter.getPreparedStatement(UpdateOutputFormat.java:174) >>>> >> >>>>> at >>>> >> >>>>> >>>> >> >>>>> >>>> >> >>>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149) >>>> >> >>>>> at >>>> >> >>>>> >>>> >> >>>>> >>>> >> >>>>> org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181) >>>> >> >>>>> ... 8 more >>>> >> >>>>> 2014-07-17 10:19:10,543 INFO org.apache.hadoop.mapred.Task: >>>> >> >>>>> Runnning >>>> >> >>>>> cleanup for the task >>>> >> >>>>> >>>> >> >>>>> There isn't more data than that. >>>> >> >>>>> Can you please check my sqoop command and validate that I'm >>>> >> >>>>> using >>>> >> >>>>> the >>>> >> >>>>> proper arguments? The argument "--columns" is used in export to >>>> >> >>>>> tell >>>> >> >>>>> sqoop >>>> >> >>>>> the order in which it should read the columns from the file >>>> >> >>>>> right? >>>> >> >>>>> Does the last column need to have delimiter too? >>>> >> >>>>> The source file should be ok, have in mind that it works for >>>> >> >>>>> insert >>>> >> >>>>> but >>>> >> >>>>> fails when I add the parameter --update-key >>>> >> >>>>> >>>> >> >>>>> Thanks >>>> >> >>>>> Leo >>>> >> >>>>> >>>> >> >>>>> >>>> >> >>>>> On Thu, Jul 17, 2014 at 1:52 PM, Gwen Shapira >>>> >> >>>>> <[email protected]> >>>> >> >>>>> wrote: >>>> >> >>>>>> >>>> >> >>>>>> I can confirm that Sqoop export update works on Oracle, both >>>> >> >>>>>> with >>>> >> >>>>>> and >>>> >> >>>>>> without Oraoop. >>>> >> >>>>>> >>>> >> >>>>>> The specific exception you are getting indicates that Oracle >>>> >> >>>>>> expects >>>> >> >>>>>> at least 4 columns of data and the HDFS file may have less >>>> >> >>>>>> than >>>> >> >>>>>> that. >>>> >> >>>>>> >>>> >> >>>>>> Can you double check that the columns in Oracle and your data >>>> >> >>>>>> file >>>> >> >>>>>> match? And that you are using a correct delimiter? >>>> >> >>>>>> >>>> >> >>>>>> And as Jarcec said, if you have access to the Task Tracker >>>> >> >>>>>> user >>>> >> >>>>>> logs >>>> >> >>>>>> for one of the mappers, you'll have much more details to work >>>> >> >>>>>> with >>>> >> >>>>>> - for >>>> >> >>>>>> example the specific line that failed. >>>> >> >>>>>> >>>> >> >>>>>> Gwen >>>> >> >>>>>> >>>> >> >>>>>> >>>> >> >>>>>> >>>> >> >>>>>> >>>> >> >>>>>> On Thu, Jul 17, 2014 at 7:44 AM, Leonardo Brambilla >>>> >> >>>>>> <[email protected]> wrote: >>>> >> >>>>>>> >>>> >> >>>>>>> Hello Jarek, >>>> >> >>>>>>> >>>> >> >>>>>>> I'm getting back to this issue, I'm trying to fix it by using >>>> >> >>>>>>> Oraoop >>>> >> >>>>>>> but that doesn't avoid the exception: >>>> >> >>>>>>> java.io.IOException: java.sql.SQLException: Missing IN or OUT >>>> >> >>>>>>> parameter at index:: 4 >>>> >> >>>>>>> >>>> >> >>>>>>> I ran a couple of tests and I can tell that the following >>>> >> >>>>>>> command >>>> >> >>>>>>> works to insert new rows: >>>> >> >>>>>>> sqoop export \ >>>> >> >>>>>>> --connect jdbc:oracle:thin:@ddb04.local.com:1541/test04 \ >>>> >> >>>>>>> --columns $columns \ >>>> >> >>>>>>> --table $table --username $user --password $passwd \ >>>> >> >>>>>>> --fields-terminated-by "=" --export-dir $exportDir >>>> >> >>>>>>> >>>> >> >>>>>>> But the following command (just added --update-key) throws an >>>> >> >>>>>>> exception: >>>> >> >>>>>>> sqoop export \ >>>> >> >>>>>>> --connect jdbc:oracle:thin:@ddb04.local.com:1541/test04 \ >>>> >> >>>>>>> --update-key "SEARCH_DATE" \ >>>> >> >>>>>>> --columns $columns \ >>>> >> >>>>>>> --table $table --username $user --password $passwd \ >>>> >> >>>>>>> --fields-terminated-by "=" --export-dir $exportDir >>>> >> >>>>>>> >>>> >> >>>>>>> DB is oracle 11.2.0.2.0 >>>> >> >>>>>>> Sqoop is 1.4.4 >>>> >> >>>>>>> Java 1.7 >>>> >> >>>>>>> Oraoop 1.6 >>>> >> >>>>>>> Oracle jdbc driver "ojdb6c.jar" implementation version >>>> >> >>>>>>> 11.2.0.3.0 >>>> >> >>>>>>> >>>> >> >>>>>>> Like I said before, all the log I can get from the failed >>>> >> >>>>>>> task I >>>> >> >>>>>>> already posted here. >>>> >> >>>>>>> >>>> >> >>>>>>> Can you confirm that Sqoop export update works on Oracle DBs? >>>> >> >>>>>>> Thanks in advance >>>> >> >>>>>>> Leo >>>> >> >>>>>>> >>>> >> >>>>>>> >>>> >> >>>>>>> >>>> >> >>>>>>> On Fri, May 16, 2014 at 4:51 PM, Jarek Jarcec Cecho >>>> >> >>>>>>> <[email protected]> wrote: >>>> >> >>>>>>>> >>>> >> >>>>>>>> Hi Leonardo, >>>> >> >>>>>>>> sadly the Sqoop output might not be that much helpful in >>>> >> >>>>>>>> this >>>> >> >>>>>>>> case, >>>> >> >>>>>>>> could you please share with us the failed map task log? >>>> >> >>>>>>>> >>>> >> >>>>>>>> The easiest way how to get it on Hadoop 1.x is to open the >>>> >> >>>>>>>> job >>>> >> >>>>>>>> tracker webinterface, find the failed Sqoop job and navigate >>>> >> >>>>>>>> to >>>> >> >>>>>>>> the failed >>>> >> >>>>>>>> map tasks. >>>> >> >>>>>>>> >>>> >> >>>>>>>> Jarcec >>>> >> >>>>>>>> >>>> >> >>>>>>>> On Tue, May 13, 2014 at 11:36:34AM -0300, Leonardo Brambilla >>>> >> >>>>>>>> wrote: >>>> >> >>>>>>>> > Hi Jarek, find below the full sqoop generated log. I went >>>> >> >>>>>>>> > through >>>> >> >>>>>>>> > all the >>>> >> >>>>>>>> > Cluster's nodes for this task logs and there is nothing >>>> >> >>>>>>>> > more >>>> >> >>>>>>>> > than >>>> >> >>>>>>>> > this same >>>> >> >>>>>>>> > error. I really don't know what else to look for. >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > Thanks >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > Warning: /usr/lib/hbase does not exist! HBase imports will >>>> >> >>>>>>>> > fail. >>>> >> >>>>>>>> > Please set $HBASE_HOME to the root of your HBase >>>> >> >>>>>>>> > installation. >>>> >> >>>>>>>> > 14/05/13 10:26:41 WARN tool.BaseSqoopTool: Setting your >>>> >> >>>>>>>> > password >>>> >> >>>>>>>> > on the >>>> >> >>>>>>>> > command-line is insecure. Consider using -P instead. >>>> >> >>>>>>>> > 14/05/13 10:26:41 INFO manager.SqlManager: Using default >>>> >> >>>>>>>> > fetchSize >>>> >> >>>>>>>> > of 1000 >>>> >> >>>>>>>> > 14/05/13 10:26:41 INFO manager.OracleManager: Time zone >>>> >> >>>>>>>> > has >>>> >> >>>>>>>> > been >>>> >> >>>>>>>> > set to GMT >>>> >> >>>>>>>> > 14/05/13 10:26:41 INFO tool.CodeGenTool: Beginning code >>>> >> >>>>>>>> > generation >>>> >> >>>>>>>> > 14/05/13 10:26:41 INFO manager.OracleManager: Time zone >>>> >> >>>>>>>> > has >>>> >> >>>>>>>> > been >>>> >> >>>>>>>> > set to GMT >>>> >> >>>>>>>> > 14/05/13 10:26:41 INFO manager.SqlManager: Executing SQL >>>> >> >>>>>>>> > statement: SELECT >>>> >> >>>>>>>> > t.* FROM etl.EXPT_SPAM_RED_JOB t WHERE 1=0 >>>> >> >>>>>>>> > 14/05/13 10:26:41 INFO orm.CompilationManager: >>>> >> >>>>>>>> > HADOOP_MAPRED_HOME >>>> >> >>>>>>>> > is >>>> >> >>>>>>>> > /home/elance/hadoop >>>> >> >>>>>>>> > Note: >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > /tmp/sqoop-elance/compile/9f8f413ab105fbe67d985bdb29534d27/etl_EXPT_SPAM_RED_JOB.java >>>> >> >>>>>>>> > uses or overrides a deprecated API. >>>> >> >>>>>>>> > Note: Recompile with -Xlint:deprecation for details. >>>> >> >>>>>>>> > 14/05/13 10:26:42 INFO orm.CompilationManager: Writing jar >>>> >> >>>>>>>> > file: >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > /tmp/sqoop-elance/compile/9f8f413ab105fbe67d985bdb29534d27/etl.EXPT_SPAM_RED_JOB.jar >>>> >> >>>>>>>> > 14/05/13 10:26:42 INFO mapreduce.ExportJobBase: Beginning >>>> >> >>>>>>>> > export >>>> >> >>>>>>>> > of >>>> >> >>>>>>>> > etl.EXPT_SPAM_RED_JOB >>>> >> >>>>>>>> > 14/05/13 10:26:43 INFO input.FileInputFormat: Total input >>>> >> >>>>>>>> > paths >>>> >> >>>>>>>> > to >>>> >> >>>>>>>> > process >>>> >> >>>>>>>> > : 1 >>>> >> >>>>>>>> > 14/05/13 10:26:43 INFO input.FileInputFormat: Total input >>>> >> >>>>>>>> > paths >>>> >> >>>>>>>> > to >>>> >> >>>>>>>> > process >>>> >> >>>>>>>> > : 1 >>>> >> >>>>>>>> > 14/05/13 10:26:44 INFO mapred.JobClient: Running job: >>>> >> >>>>>>>> > job_201404190827_0998 >>>> >> >>>>>>>> > 14/05/13 10:26:45 INFO mapred.JobClient: map 0% reduce 0% >>>> >> >>>>>>>> > 14/05/13 10:26:53 INFO mapred.JobClient: map 25% reduce >>>> >> >>>>>>>> > 0% >>>> >> >>>>>>>> > 14/05/13 10:26:54 INFO mapred.JobClient: map 75% reduce >>>> >> >>>>>>>> > 0% >>>> >> >>>>>>>> > 14/05/13 10:26:55 INFO mapred.JobClient: Task Id : >>>> >> >>>>>>>> > attempt_201404190827_0998_m_000001_0, Status : FAILED >>>> >> >>>>>>>> > java.io.IOException: java.sql.SQLException: Missing IN or >>>> >> >>>>>>>> > OUT >>>> >> >>>>>>>> > parameter at >>>> >> >>>>>>>> > index:: 4 >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:651) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > org.apache.hadoop.mapred.Child$4.run(Child.java:255) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > java.security.AccessController.doPrivileged(Native >>>> >> >>>>>>>> > Method) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > javax.security.auth.Subject.doAs(Subject.java:415) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > org.apache.hadoop.mapred.Child.main(Child.java:249) >>>> >> >>>>>>>> > Caused by: java.sql.SQLException: Missing IN or OUT >>>> >> >>>>>>>> > parameter >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > index:: 4 >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > oracle.jdbc.driver.OraclePreparedStatement.processCompletedBindRow(OraclePreparedStatement.java:1844) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > oracle.jdbc.driver.OraclePreparedStatement.addBatch(OraclePreparedStatement.java:10213) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > oracle.jdbc.driver.OraclePreparedStatementWrapper.addBatch(OraclePreparedStatementWrapper.java:1362) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.sqoop.mapreduce.UpdateOutputFormat$UpdateRecordWriter.getPreparedStatement(UpdateOutputFormat.java:174) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181) >>>> >> >>>>>>>> > ... 8 more >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > 14/05/13 10:27:00 INFO mapred.JobClient: Task Id : >>>> >> >>>>>>>> > attempt_201404190827_0998_m_000001_1, Status : FAILED >>>> >> >>>>>>>> > java.io.IOException: java.sql.SQLException: Missing IN or >>>> >> >>>>>>>> > OUT >>>> >> >>>>>>>> > parameter at >>>> >> >>>>>>>> > index:: 4 >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:651) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > org.apache.hadoop.mapred.Child$4.run(Child.java:255) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > java.security.AccessController.doPrivileged(Native >>>> >> >>>>>>>> > Method) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > javax.security.auth.Subject.doAs(Subject.java:415) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > org.apache.hadoop.mapred.Child.main(Child.java:249) >>>> >> >>>>>>>> > Caused by: java.sql.SQLException: Missing IN or OUT >>>> >> >>>>>>>> > parameter >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > index:: 4 >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > oracle.jdbc.driver.OraclePreparedStatement.processCompletedBindRow(OraclePreparedStatement.java:1844) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > oracle.jdbc.driver.OraclePreparedStatement.addBatch(OraclePreparedStatement.java:10213) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > oracle.jdbc.driver.OraclePreparedStatementWrapper.addBatch(OraclePreparedStatementWrapper.java:1362) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.sqoop.mapreduce.UpdateOutputFormat$UpdateRecordWriter.getPreparedStatement(UpdateOutputFormat.java:174) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181) >>>> >> >>>>>>>> > ... 8 more >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > 14/05/13 10:27:05 INFO mapred.JobClient: Task Id : >>>> >> >>>>>>>> > attempt_201404190827_0998_m_000001_2, Status : FAILED >>>> >> >>>>>>>> > java.io.IOException: java.sql.SQLException: Missing IN or >>>> >> >>>>>>>> > OUT >>>> >> >>>>>>>> > parameter at >>>> >> >>>>>>>> > index:: 4 >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:184) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.close(MapTask.java:651) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > org.apache.hadoop.mapred.Child$4.run(Child.java:255) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > java.security.AccessController.doPrivileged(Native >>>> >> >>>>>>>> > Method) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > javax.security.auth.Subject.doAs(Subject.java:415) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > org.apache.hadoop.mapred.Child.main(Child.java:249) >>>> >> >>>>>>>> > Caused by: java.sql.SQLException: Missing IN or OUT >>>> >> >>>>>>>> > parameter >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > index:: 4 >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > oracle.jdbc.driver.OraclePreparedStatement.processCompletedBindRow(OraclePreparedStatement.java:1844) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > oracle.jdbc.driver.OraclePreparedStatement.addBatch(OraclePreparedStatement.java:10213) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > oracle.jdbc.driver.OraclePreparedStatementWrapper.addBatch(OraclePreparedStatementWrapper.java:1362) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.sqoop.mapreduce.UpdateOutputFormat$UpdateRecordWriter.getPreparedStatement(UpdateOutputFormat.java:174) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149) >>>> >> >>>>>>>> > at >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.close(AsyncSqlRecordWriter.java:181) >>>> >> >>>>>>>> > ... 8 more >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Job complete: >>>> >> >>>>>>>> > job_201404190827_0998 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Counters: 20 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Job Counters >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: >>>> >> >>>>>>>> > SLOTS_MILLIS_MAPS=30548 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Total time >>>> >> >>>>>>>> > spent >>>> >> >>>>>>>> > by >>>> >> >>>>>>>> > all >>>> >> >>>>>>>> > reduces waiting after reserving slots (ms)=0 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Total time >>>> >> >>>>>>>> > spent >>>> >> >>>>>>>> > by >>>> >> >>>>>>>> > all maps >>>> >> >>>>>>>> > waiting after reserving slots (ms)=0 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Rack-local >>>> >> >>>>>>>> > map >>>> >> >>>>>>>> > tasks=5 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Launched map >>>> >> >>>>>>>> > tasks=7 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Data-local >>>> >> >>>>>>>> > map >>>> >> >>>>>>>> > tasks=2 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: >>>> >> >>>>>>>> > SLOTS_MILLIS_REDUCES=0 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Failed map >>>> >> >>>>>>>> > tasks=1 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: File Output >>>> >> >>>>>>>> > Format >>>> >> >>>>>>>> > Counters >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Bytes >>>> >> >>>>>>>> > Written=0 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: >>>> >> >>>>>>>> > FileSystemCounters >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: >>>> >> >>>>>>>> > HDFS_BYTES_READ=459 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: >>>> >> >>>>>>>> > FILE_BYTES_WRITTEN=189077 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: File Input >>>> >> >>>>>>>> > Format >>>> >> >>>>>>>> > Counters >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Bytes Read=0 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Map-Reduce >>>> >> >>>>>>>> > Framework >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Map input >>>> >> >>>>>>>> > records=0 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Physical >>>> >> >>>>>>>> > memory >>>> >> >>>>>>>> > (bytes) >>>> >> >>>>>>>> > snapshot=363053056 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Spilled >>>> >> >>>>>>>> > Records=0 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: CPU time >>>> >> >>>>>>>> > spent >>>> >> >>>>>>>> > (ms)=2110 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Total >>>> >> >>>>>>>> > committed >>>> >> >>>>>>>> > heap >>>> >> >>>>>>>> > usage >>>> >> >>>>>>>> > (bytes)=553517056 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Virtual >>>> >> >>>>>>>> > memory >>>> >> >>>>>>>> > (bytes) >>>> >> >>>>>>>> > snapshot=2344087552 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: Map output >>>> >> >>>>>>>> > records=0 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapred.JobClient: >>>> >> >>>>>>>> > SPLIT_RAW_BYTES=404 >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapreduce.ExportJobBase: >>>> >> >>>>>>>> > Transferred 459 >>>> >> >>>>>>>> > bytes in >>>> >> >>>>>>>> > 30.0642 seconds (15.2673 bytes/sec) >>>> >> >>>>>>>> > 14/05/13 10:27:13 INFO mapreduce.ExportJobBase: Exported 0 >>>> >> >>>>>>>> > records. >>>> >> >>>>>>>> > 14/05/13 10:27:13 ERROR tool.ExportTool: Error during >>>> >> >>>>>>>> > export: >>>> >> >>>>>>>> > Export job >>>> >> >>>>>>>> > failed! >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > On Mon, May 12, 2014 at 10:44 PM, Jarek Jarcec Cecho >>>> >> >>>>>>>> > <[email protected]>wrote: >>>> >> >>>>>>>> > >>>> >> >>>>>>>> > > The map task log contain entire executed query and lot >>>> >> >>>>>>>> > > of >>>> >> >>>>>>>> > > additional >>>> >> >>>>>>>> > > information and hence it's supper useful in such cases. >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > Jarcec >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > On Mon, May 12, 2014 at 02:59:56PM -0300, Leonardo >>>> >> >>>>>>>> > > Brambilla >>>> >> >>>>>>>> > > wrote: >>>> >> >>>>>>>> > > > Hi Jarek, >>>> >> >>>>>>>> > > > >>>> >> >>>>>>>> > > > thanks for replying, I don't have the logs. I'll see >>>> >> >>>>>>>> > > > if I >>>> >> >>>>>>>> > > > can >>>> >> >>>>>>>> > > > run the >>>> >> >>>>>>>> > > task >>>> >> >>>>>>>> > > > again and then keep the logs. >>>> >> >>>>>>>> > > > >>>> >> >>>>>>>> > > > Anyway, I don't remember seeing anything else than >>>> >> >>>>>>>> > > > this >>>> >> >>>>>>>> > > > SQLException >>>> >> >>>>>>>> > > about >>>> >> >>>>>>>> > > > missing parameter. >>>> >> >>>>>>>> > > > >>>> >> >>>>>>>> > > > Leo >>>> >> >>>>>>>> > > > >>>> >> >>>>>>>> > > > >>>> >> >>>>>>>> > > > On Sun, May 11, 2014 at 10:59 AM, Jarek Jarcec Cecho >>>> >> >>>>>>>> > > > <[email protected] >>>> >> >>>>>>>> > > >wrote: >>>> >> >>>>>>>> > > > >>>> >> >>>>>>>> > > > > Hi Leonardo, >>>> >> >>>>>>>> > > > > would you mind sharing with us task log from the >>>> >> >>>>>>>> > > > > failed >>>> >> >>>>>>>> > > > > map >>>> >> >>>>>>>> > > > > task? >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > > > Jarcec >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > > > On Sun, May 11, 2014 at 10:33:11AM -0300, Leonardo >>>> >> >>>>>>>> > > > > Brambilla >>>> >> >>>>>>>> > > > > wrote: >>>> >> >>>>>>>> > > > > > Hello, I am struggling to make it work, what is a >>>> >> >>>>>>>> > > > > > really >>>> >> >>>>>>>> > > > > > required >>>> >> >>>>>>>> > > > > feature. >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > > I have a process that daily generates new data, >>>> >> >>>>>>>> > > > > > this >>>> >> >>>>>>>> > > > > > data >>>> >> >>>>>>>> > > > > > needs to be >>>> >> >>>>>>>> > > > > > pushed to a table in Oracle, the table might >>>> >> >>>>>>>> > > > > > already >>>> >> >>>>>>>> > > > > > have >>>> >> >>>>>>>> > > > > > same data >>>> >> >>>>>>>> > > from >>>> >> >>>>>>>> > > > > > previous loads. I need to avoid duplicating data >>>> >> >>>>>>>> > > > > > on it. >>>> >> >>>>>>>> > > > > > Pretty common >>>> >> >>>>>>>> > > > > > scenario right? =) >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > > I am using sqoop export for this, no special >>>> >> >>>>>>>> > > > > > arguments, >>>> >> >>>>>>>> > > > > > just columns, >>>> >> >>>>>>>> > > > > > fields-terminated-by, table and db connection, >>>> >> >>>>>>>> > > > > > plus the >>>> >> >>>>>>>> > > > > > argument >>>> >> >>>>>>>> > > > > > "update-mode allowinsert". >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > > Now, when I also include the argument "update-key" >>>> >> >>>>>>>> > > > > > with >>>> >> >>>>>>>> > > > > > a >>>> >> >>>>>>>> > > > > > comma >>>> >> >>>>>>>> > > separated >>>> >> >>>>>>>> > > > > > list of fields (which is the same for arg columns) >>>> >> >>>>>>>> > > > > > I >>>> >> >>>>>>>> > > > > > get >>>> >> >>>>>>>> > > > > > the >>>> >> >>>>>>>> > > following >>>> >> >>>>>>>> > > > > > oracle driver error: >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > > 14/05/07 16:00:03 INFO mapred.JobClient: Task Id : >>>> >> >>>>>>>> > > > > > attempt_201404190827_0928_m_000003_2, Status : >>>> >> >>>>>>>> > > > > > FAILED >>>> >> >>>>>>>> > > > > > java.io.IOException: Can't export data, please >>>> >> >>>>>>>> > > > > > check >>>> >> >>>>>>>> > > > > > task >>>> >> >>>>>>>> > > > > > tracker >>>> >> >>>>>>>> > > logs >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > > org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > > > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > > org.apache.hadoop.mapred.Child$4.run(Child.java:255) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > java.security.AccessController.doPrivileged(Native >>>> >> >>>>>>>> > > > > > Method) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > javax.security.auth.Subject.doAs(Subject.java:415) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > > org.apache.hadoop.mapred.Child.main(Child.java:249) >>>> >> >>>>>>>> > > > > > Caused by: java.io.IOException: >>>> >> >>>>>>>> > > > > > java.sql.SQLException: >>>> >> >>>>>>>> > > > > > Missing IN or >>>> >> >>>>>>>> > > OUT >>>> >> >>>>>>>> > > > > > parameter at index:: 4 >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:220) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:46) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:639) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:84) >>>> >> >>>>>>>> > > > > > ... 10 more >>>> >> >>>>>>>> > > > > > *Caused by: java.sql.SQLException: Missing IN or >>>> >> >>>>>>>> > > > > > OUT >>>> >> >>>>>>>> > > > > > parameter at >>>> >> >>>>>>>> > > > > index:: 4* >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > oracle.jdbc.driver.OraclePreparedStatement.processCompletedBindRow(OraclePreparedStatement.java:1844) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > oracle.jdbc.driver.OraclePreparedStatement.addBatch(OraclePreparedStatement.java:10213) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > oracle.jdbc.driver.OraclePreparedStatementWrapper.addBatch(OraclePreparedStatementWrapper.java:1362) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > org.apache.sqoop.mapreduce.UpdateOutputFormat$UpdateRecordWriter.getPreparedStatement(UpdateOutputFormat.java:174) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.execUpdate(AsyncSqlRecordWriter.java:149) >>>> >> >>>>>>>> > > > > > at >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>>> > > org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.write(AsyncSqlRecordWriter.java:215) >>>> >> >>>>>>>> > > > > > ... 14 more >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > > I'm using Sqoop 1.4.3 with hadoop1, also tried >>>> >> >>>>>>>> > > > > > 1.4.4 >>>> >> >>>>>>>> > > > > > with >>>> >> >>>>>>>> > > > > > same >>>> >> >>>>>>>> > > result. I >>>> >> >>>>>>>> > > > > > have the standard Oracle JDBC driver 6 with Java >>>> >> >>>>>>>> > > > > > 7. >>>> >> >>>>>>>> > > > > > I went through all the documentation, Sqoop user >>>> >> >>>>>>>> > > > > > guide >>>> >> >>>>>>>> > > > > > says this is >>>> >> >>>>>>>> > > > > > supported for built-in connector which I >>>> >> >>>>>>>> > > > > > understand I >>>> >> >>>>>>>> > > > > > am >>>> >> >>>>>>>> > > > > > using. >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > > Here is the full command: >>>> >> >>>>>>>> > > > > > $sqoopExecutable export \ >>>> >> >>>>>>>> > > > > > --outdir $outdir \ >>>> >> >>>>>>>> > > > > > --connect $connectionString --table $table_client >>>> >> >>>>>>>> > > > > > --username $dbUser >>>> >> >>>>>>>> > > > > > --password $dbUserPasswd \ >>>> >> >>>>>>>> > > > > > --columns CLIENT_ID,EXP_ID,BUCKET_ID --update-key >>>> >> >>>>>>>> > > > > > CLIENT_ID,EXP_ID,BUCKET_ID \ >>>> >> >>>>>>>> > > > > > --fields-terminated-by '\t' --update-mode >>>> >> >>>>>>>> > > > > > allowinsert \ >>>> >> >>>>>>>> > > > > > --export-dir $dataSource_client > $sqoopLog 2>&1 >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > > Can someone please shed some light on this? >>>> >> >>>>>>>> > > > > > Thank you in advance. >>>> >> >>>>>>>> > > > > > >>>> >> >>>>>>>> > > > > > Leo >>>> >> >>>>>>>> > > > > >>>> >> >>>>>>>> > > >>>> >> >>>>>>> >>>> >> >>>>>>> >>>> >> >>>>>> >>>> >> >>>>> >>>> >> >>>> >>>> >> >>> >>>> >> >> >>>> >> > >>>> > >>>> > >>> >>> >> >
