-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/11320/
-----------------------------------------------------------
(Updated May 30, 2013, 5:11 p.m.)
Review request for Sqoop and Venkat Ranganathan.
Changes
-------
Change function name according to Venkat's comment
Description
-------
Implement a pluggable way for handling connection failures, and/or intermittent
errors in Sqoop. This is especially crucial in environments where the
probability of connections getting reset or throttled is high.
In case of intermittent failures in Sqoop, due to connection losses or server
throttling, Sqoop does not recover from those failures. As a result, the
running Sqoop task would eventually fail, and a new task is started. In those
cases, Sqoop does not always guarantee that tasks can safely be restarted. For
example, if part of the records is already committed to the database, then
restarting the task would result in some failures like primary key violations.
Even for Sqoop jobs which commit the records only at the end of the task, any
failures towards the end of the task would involve reprocessing the whole range
of split owned by the task, and any progress is lost. (More detail please refer
to the document attached at SQOOP-1056)
This addresses bug SQOOP-1056.
https://issues.apache.org/jira/browse/SQOOP-1056
Diffs (updated)
-----
src/java/org/apache/sqoop/manager/SQLServerManager.java eea5a1d
src/java/org/apache/sqoop/mapreduce/SQLServerAsyncDBExecThread.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/SQLServerExportDBExecThread.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/SQLServerResilientExportOutputFormat.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/SQLServerResilientUpdateOutputFormat.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/SQLServerUpdateDBExecThread.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/db/BasicRetrySQLFailureHandler.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/db/DBRecordReader.java 7b9e4d2
src/java/org/apache/sqoop/mapreduce/db/DataDrivenDBRecordReader.java 4b44244
src/java/org/apache/sqoop/mapreduce/db/SQLFailureHandler.java PRE-CREATION
src/java/org/apache/sqoop/mapreduce/db/SQLServerConnectionFailureHandler.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/db/SQLServerDBInputFormat.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/db/SQLServerDBRecordReader.java
PRE-CREATION
src/java/org/apache/sqoop/mapreduce/sqlserver/SqlServerRecordReader.java
2c08f12
Diff: https://reviews.apache.org/r/11320/diff/
Testing
-------
Tested with fault injection on SQL server integration tests from SQOOP-1035
Thanks,
Shuaishuai Nie
