----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11320/#review33294 -----------------------------------------------------------
Ship it! Thanks for uploading the revised patch. LGTM. - Venkat Ranganathan On Jan. 31, 2014, 1:24 a.m., Shuaishuai Nie wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/11320/ > ----------------------------------------------------------- > > (Updated Jan. 31, 2014, 1:24 a.m.) > > > Review request for Sqoop and Venkat Ranganathan. > > > Bugs: SQOOP-1056 > https://issues.apache.org/jira/browse/SQOOP-1056 > > > Repository: sqoop-trunk > > > Description > ------- > > Implement a pluggable way for handling connection failures, and/or > intermittent errors in Sqoop. This is especially crucial in environments > where the probability of connections getting reset or throttled is high. > In case of intermittent failures in Sqoop, due to connection losses or server > throttling, Sqoop does not recover from those failures. As a result, the > running Sqoop task would eventually fail, and a new task is started. In those > cases, Sqoop does not always guarantee that tasks can safely be restarted. > For example, if part of the records is already committed to the database, > then restarting the task would result in some failures like primary key > violations. Even for Sqoop jobs which commit the records only at the end of > the task, any failures towards the end of the task would involve reprocessing > the whole range of split owned by the task, and any progress is lost. (More > detail please refer to the document attached at SQOOP-1056) > > > Diffs > ----- > > build.xml 74150e6 > ivy.xml c00d30c > ivy/libraries.properties 6b9b2d7 > src/java/org/apache/sqoop/manager/SQLServerManager.java eea5a1d > src/java/org/apache/sqoop/manager/SqlManager.java 8248263 > src/java/org/apache/sqoop/mapreduce/SQLServerAsyncDBExecThread.java > PRE-CREATION > src/java/org/apache/sqoop/mapreduce/SQLServerExportDBExecThread.java > PRE-CREATION > > src/java/org/apache/sqoop/mapreduce/SQLServerResilientExportOutputFormat.java > PRE-CREATION > > src/java/org/apache/sqoop/mapreduce/SQLServerResilientUpdateOutputFormat.java > PRE-CREATION > src/java/org/apache/sqoop/mapreduce/SQLServerUpdateDBExecThread.java > PRE-CREATION > src/java/org/apache/sqoop/mapreduce/db/BasicRetrySQLFailureHandler.java > PRE-CREATION > src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 7b9e4d2 > src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java > 4b44244 > src/java/org/apache/sqoop/mapreduce/db/SQLFailureHandler.java PRE-CREATION > > src/java/org/apache/sqoop/mapreduce/db/SQLServerConnectionFailureHandler.java > PRE-CREATION > src/java/org/apache/sqoop/mapreduce/db/SQLServerDBInputFormat.java > PRE-CREATION > src/java/org/apache/sqoop/mapreduce/db/SQLServerDBRecordReader.java > PRE-CREATION > src/java/org/apache/sqoop/mapreduce/sqlserver/SqlServerRecordReader.java > 2c08f12 > src/test/aop/build/aop.xml PRE-CREATION > src/test/aop/org/apache/sqoop/fi/FiConfig.java PRE-CREATION > src/test/aop/org/apache/sqoop/fi/ProbabilityModel.java PRE-CREATION > src/test/aop/org/apache/sqoop/mapreduce/SqlServerExportAspect.aj > PRE-CREATION > src/test/aop/org/apache/sqoop/mapreduce/db/SqlServerImportAspect.aj > PRE-CREATION > src/test/fi-site.xml PRE-CREATION > > Diff: https://reviews.apache.org/r/11320/diff/ > > > Testing > ------- > > Tested with fault injection on SQL server integration tests from SQOOP-1035 > > > Thanks, > > Shuaishuai Nie > >
