-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11320/
-----------------------------------------------------------
(Updated Jan. 31, 2014, 1:24 a.m.)
Review request for Sqoop and Venkat Ranganathan.
Changes
-------
Combine patch from SQOOP-1056 and SQOOP-1057
Bugs: SQOOP-1056
https://issues.apache.org/jira/browse/SQOOP-1056
Repository: sqoop-trunk
Description
-------
Implement a pluggable way for handling connection failures, and/or intermittent
errors in Sqoop. This is especially crucial in environments where the
probability of connections getting reset or throttled is high.
In case of intermittent failures in Sqoop, due to connection losses or server
throttling, Sqoop does not recover from those failures. As a result, the
running Sqoop task would eventually fail, and a new task is started. In those
cases, Sqoop does not always guarantee that tasks can safely be restarted. For
example, if part of the records is already committed to the database, then
restarting the task would result in some failures like primary key violations.
Even for Sqoop jobs which commit the records only at the end of the task, any
failures towards the end of the task would involve reprocessing the whole range
of split owned by the task, and any progress is lost. (More detail please refer
to the document attached at SQOOP-1056)
Diffs (updated)
-----
build.xml 74150e6
ivy.xml c00d30c
ivy/libraries.properties 6b9b2d7
src/java/org/apache/sqoop/manager/SQLServerManager.java eea5a1d
src/java/org/apache/sqoop/manager/SqlManager.java 8248263
src/java/org/apache/sqoop/mapreduce/SQLServerAsyncDBExecThread.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/SQLServerExportDBExecThread.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/SQLServerResilientExportOutputFormat.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/SQLServerResilientUpdateOutputFormat.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/SQLServerUpdateDBExecThread.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/db/BasicRetrySQLFailureHandler.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 7b9e4d2
src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 4b44244
src/java/org/apache/sqoop/mapreduce/db/SQLFailureHandler.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/db/SQLServerConnectionFailureHandler.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/db/SQLServerDBInputFormat.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/db/SQLServerDBRecordReader.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/sqlserver/SqlServerRecordReader.java
2c08f12
src/test/aop/build/aop.xml PRE-CREATION
src/test/aop/org/apache/sqoop/fi/FiConfig.java PRE-CREATION
src/test/aop/org/apache/sqoop/fi/ProbabilityModel.java PRE-CREATION
src/test/aop/org/apache/sqoop/mapreduce/SqlServerExportAspect.aj PRE-CREATION
src/test/aop/org/apache/sqoop/mapreduce/db/SqlServerImportAspect.aj
PRE-CREATION
src/test/fi-site.xml PRE-CREATION
Diff: https://reviews.apache.org/r/11320/diff/
Testing
-------
Tested with fault injection on SQL server integration tests from SQOOP-1035
Thanks,
Shuaishuai Nie
