[ 
https://issues.apache.org/jira/browse/SQOOP-3139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zoltán Tóth reassigned SQOOP-3139:
----------------------------------

    Assignee: Zoltán Tóth  (was: Anna Szonyi)

> sqoop tries to re execute select query during import in case of a connection 
> reset error and this is causing lots of duplicate records from source
> --------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SQOOP-3139
>                 URL: https://issues.apache.org/jira/browse/SQOOP-3139
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.4.6
>         Environment: IBM Hadoop distribution: 4.2.0
> version: 1.4.6_IBM_27
> Sqoop 1.4.6_IBM_27
>            Reporter: hemanth meka
>            Assignee: Zoltán Tóth
>         Attachments: SQOOP-3139.patch
>
>
> We cannot reproduce this issue as it depends on network. Here is a edited log 
> content for understanding the issue. 
> Log start
> .
> .
> 2017-02-22 07:35:37,638 INFO [main] 
> org.apache.sqoop.mapreduce.sqlserver.SqlServerRecordReader: Using query: 
> select sequence_number, analytical_bundle_masked where ( Sequence_Number >= 
> 8571429 ) AND ( Sequence_Number <= 10000000 )
> 2017-02-22 07:35:37,662 INFO [main] 
> org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select 
> sequence_number, analytical_bundle_masked where ( Sequence_Number >= 8571429 
> ) AND ( Sequence_Number <= 10000000 )
> 2017-02-22 07:39:00,533 ERROR [main] 
> org.apache.sqoop.mapreduce.db.DBRecordReader: Top level exception: 
> com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
>       at 
> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399)
>       at 
> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2383)
>       at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1884)
>       at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6685)
>       at com.microsoft.sqlserver.jdbc.TDSReader.nextPacket(IOBuffer.java:6595)
>       at 
> com.microsoft.sqlserver.jdbc.TDSReader.ensurePayload(IOBuffer.java:6571)
>       at com.microsoft.sqlserver.jdbc.TDSReader.readBytes(IOBuffer.java:6864)
>       at 
> com.microsoft.sqlserver.jdbc.TDSReader.readWrappedBytes(IOBuffer.java:6886)
>       at 
> com.microsoft.sqlserver.jdbc.TDSReader.readUnsignedShort(IOBuffer.java:6801)
>       at 
> com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValuePrep(dtv.java:3570)
>       at com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValue(dtv.java:3936)
>       at com.microsoft.sqlserver.jdbc.DTV.getValue(dtv.java:226)
>       at com.microsoft.sqlserver.jdbc.Column.getValue(Column.java:144)
>       at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2099)
>       at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2084)
>       at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getString(SQLServerResultSet.java:2427)
>       at 
> org.apache.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:71)
>       at 
> com.cloudera.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:61)
>       at QueryResult.readFields0(QueryResult.java:10706)
>       at QueryResult.readFields(QueryResult.java:10415)
>       at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)
>       at 
> org.apache.sqoop.mapreduce.db.SQLServerDBRecordReader.nextKeyValue(SQLServerDBRecordReader.java:148)
>       at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
>       at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
>       at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
>       at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>       at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>       at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
>       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>       at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>       at java.security.AccessController.doPrivileged(Native Method)
>       at javax.security.auth.Subject.doAs(Subject.java:422)
>       at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>       at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> 2017-02-22 07:39:00,552 WARN [main] 
> org.apache.sqoop.mapreduce.db.SQLServerDBRecordReader: Trying to recover from 
> DB read failure: 
> java.io.IOException: SQLException in nextKeyValue
>       at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277)
>       at 
> org.apache.sqoop.mapreduce.db.SQLServerDBRecordReader.nextKeyValue(SQLServerDBRecordReader.java:148)
>       at 
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556)
>       at 
> org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80)
>       at 
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91)
>       at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
>       at 
> org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
>       at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
>       at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>       at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>       at java.security.AccessController.doPrivileged(Native Method)
>       at javax.security.auth.Subject.doAs(Subject.java:422)
>       at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>       at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: Connection reset
>       at 
> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2399)
>       at 
> com.microsoft.sqlserver.jdbc.SQLServerConnection.terminate(SQLServerConnection.java:2383)
>       at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1884)
>       at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:6685)
>       at com.microsoft.sqlserver.jdbc.TDSReader.nextPacket(IOBuffer.java:6595)
>       at 
> com.microsoft.sqlserver.jdbc.TDSReader.ensurePayload(IOBuffer.java:6571)
>       at com.microsoft.sqlserver.jdbc.TDSReader.readBytes(IOBuffer.java:6864)
>       at 
> com.microsoft.sqlserver.jdbc.TDSReader.readWrappedBytes(IOBuffer.java:6886)
>       at 
> com.microsoft.sqlserver.jdbc.TDSReader.readUnsignedShort(IOBuffer.java:6801)
>       at 
> com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValuePrep(dtv.java:3570)
>       at com.microsoft.sqlserver.jdbc.ServerDTVImpl.getValue(dtv.java:3936)
>       at com.microsoft.sqlserver.jdbc.DTV.getValue(dtv.java:226)
>       at com.microsoft.sqlserver.jdbc.Column.getValue(Column.java:144)
>       at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2099)
>       at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getValue(SQLServerResultSet.java:2084)
>       at 
> com.microsoft.sqlserver.jdbc.SQLServerResultSet.getString(SQLServerResultSet.java:2427)
>       at 
> org.apache.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:71)
>       at 
> com.cloudera.sqoop.lib.JdbcWritableBridge.readString(JdbcWritableBridge.java:61)
>       at QueryResult.readFields0(QueryResult.java:10706)
>       at QueryResult.readFields(QueryResult.java:10415)
>       at 
> org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244)
>       ... 13 more
> .
> .
> 2017-02-22 07:39:00,928 INFO [main] 
> org.apache.sqoop.mapreduce.sqlserver.SqlServerRecordReader: Using query: 
> select sequence_number, analytical_bundle_masked where ( Sequence_Number >= 
> 8571429 ) AND ( Sequence_Number <= 10000000 )
> 2017-02-22 07:39:00,929 INFO [main] 
> org.apache.sqoop.mapreduce.db.DBRecordReader: Executing query: select 
> sequence_number, analytical_bundle_masked where ( Sequence_Number >= 8571429 
> ) AND ( Sequence_Number <= 10000000 )
> 2017-02-22 08:05:03,199 INFO [Thread-15] 
> org.apache.sqoop.mapreduce.AutoProgressMapper: Auto-progress thread is 
> finished. keepGoing=false
> 2017-02-22 08:05:03,295 INFO [main] org.apache.hadoop.mapred.Task: 
> Task:attempt_1486732304049_0615_m_000006_0 is done. And is in the process of 
> committing
> 2017-02-22 08:05:03,344 INFO [main] org.apache.hadoop.mapred.Task: Task 
> attempt_1486732304049_0615_m_000006_0 is allowed to commit now
> 2017-02-22 08:05:03,350 INFO [main] 
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: Saved output of 
> task 'attempt_1486732304049_0615_m_000006_0' to 
> hdfs://ehaascluster/jcrew/hdfs/qa/secure/work/customer/experian/experian_jb_analytical_bundle/_temporary/1/task_1486732304049_0615_m_000006
> 2017-02-22 08:05:03,390 INFO [main] org.apache.hadoop.mapred.Task: Task 
> 'attempt_1486732304049_0615_m_000006_1' done.
> End of LogType:syslog
> Container: container_e58_1486732304049_0615_01_000005 on 
> bi4c-11665-data-1.bi.services.bluemix.net_45454
> ==========================================================================================================
> LogType:stderr
> Log Upload Time:Wed Feb 22 08:07:25 +0000 2017
> LogLength:243
> Log Contents:
> log4j:WARN No appenders could be found for logger 
> (org.apache.hadoop.metrics2.impl.MetricsSystemImpl).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more 
> info.
> End of LogType:stderr
> LogType:stdout
> Log Upload Time:Wed Feb 22 08:07:25 +0000 2017
> LogLength:0
> Log Contents:
> End of LogType:stdout
> LogType:syslog
> Log Upload Time:Wed Feb 22 08:07:25 +0000 2017
> LogLength:104358



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to