[jira] [Updated] (SQOOP-604) Easy throttling feature for MySQL exports
[ https://issues.apache.org/jira/browse/SQOOP-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Toth-Czifra updated SQOOP-604: - Attachment: SQOOP-604_v6.patch Attaching the same patch on my own in order to explicitly give Apache Software Foundation (ASF) right to use that piece of code in Sqoop under ASF license. Easy throttling feature for MySQL exports - Key: SQOOP-604 URL: https://issues.apache.org/jira/browse/SQOOP-604 Project: Sqoop Issue Type: Improvement Components: connectors/mysql Affects Versions: 1.4.2 Reporter: Zoltan Toth-Czifra Assignee: Zoltan Toth-Czifra Priority: Minor Fix For: 1.4.3 Attachments: SQOOP-604_v6.patch, SQOOP-604_v6.patch Sqoop always tries to achieve the best possible throughput with exports, which might not be desirable in all cases. Sometimes we need to export large data with Sqoop to a live relational database (MySQL in our case), that is, a database that is under a high load serving random queries from the users of our product. While data consistency issues during the export can be easily solved with a staging table, there is still a problem: the performance impact caused by the heavy export. First off, the resources of MySQL dedicated to the import process can affect the performance of the live product, both on the master and on the slaves. Second, even if the servers can handle the import with no significant performance impact (mysqlimport should be relatively cheap), importing big tables (GB+) can cause serious replication lag in the cluster risking data consistency. My suggestion is quite simple. Using the already existing checkpoint feature of the MySQL exports (the export process is restarted every X bytes written), extending it with a new config value that would simply make the thread sleep for X milliseconds at the checkbpoints. With low enough byte count limit this can be a simple yet powerful throttling mechanism. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (SQOOP-604) Easy throttling feature for MySQL exports
[ https://issues.apache.org/jira/browse/SQOOP-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Toth-Czifra closed SQOOP-604. Easy throttling feature for MySQL exports - Key: SQOOP-604 URL: https://issues.apache.org/jira/browse/SQOOP-604 Project: Sqoop Issue Type: Improvement Components: connectors/mysql Affects Versions: 1.4.2 Reporter: Zoltan Toth-Czifra Assignee: Zoltan Toth-Czifra Priority: Minor Fix For: 1.4.3 Attachments: SQOOP-604_v6.patch, SQOOP-604_v6.patch Sqoop always tries to achieve the best possible throughput with exports, which might not be desirable in all cases. Sometimes we need to export large data with Sqoop to a live relational database (MySQL in our case), that is, a database that is under a high load serving random queries from the users of our product. While data consistency issues during the export can be easily solved with a staging table, there is still a problem: the performance impact caused by the heavy export. First off, the resources of MySQL dedicated to the import process can affect the performance of the live product, both on the master and on the slaves. Second, even if the servers can handle the import with no significant performance impact (mysqlimport should be relatively cheap), importing big tables (GB+) can cause serious replication lag in the cluster risking data consistency. My suggestion is quite simple. Using the already existing checkpoint feature of the MySQL exports (the export process is restarted every X bytes written), extending it with a new config value that would simply make the thread sleep for X milliseconds at the checkbpoints. With low enough byte count limit this can be a simple yet powerful throttling mechanism. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (SQOOP-683) Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports
Zoltan Toth-Czifra created SQOOP-683: Summary: Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports Key: SQOOP-683 URL: https://issues.apache.org/jira/browse/SQOOP-683 Project: Sqoop Issue Type: Sub-task Components: connectors/mysql, docs Affects Versions: 1.4.2 Reporter: Zoltan Toth-Czifra Assignee: Zoltan Toth-Czifra Priority: Trivial Documenting feature added in parent task. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-641) Add unit tests to Sqoop 2
[ https://issues.apache.org/jira/browse/SQOOP-641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Jarcec Cecho updated SQOOP-641: - Assignee: (was: Jarek Jarcec Cecho) Add unit tests to Sqoop 2 - Key: SQOOP-641 URL: https://issues.apache.org/jira/browse/SQOOP-641 Project: Sqoop Issue Type: Task Affects Versions: 2.0.0 Reporter: Jarek Jarcec Cecho Labels: newbie Fix For: 2.0.0 We already do have some unit tests in place, but our test covering is very small. We should improve that. I'm creating this ticket as a top level task, where individual test cases will be added in subsequently created sub tasks. Please do not hesitate to create sub task if you want to provide some test case to Sqoop 2 branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (SQOOP-683) Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports
[
https://issues.apache.org/jira/browse/SQOOP-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490745#comment-13490745
]
Zoltan Toth-Czifra edited comment on SQOOP-683 at 11/5/12 5:17 PM:
---
{code}
diff --git a/src/docs/user/compatibility.txt b/src/docs/user/compatibility.txt
index 3576fd7..e8218d6 100644
--- a/src/docs/user/compatibility.txt
+++ b/src/docs/user/compatibility.txt
@@ -138,9 +138,30 @@ bytes. Set _size_ to 0 to disable intermediate checkpoints,
but individual files being exported will continue to be committed
independently of one another.
+Sometimes you need to export large data with Sqoop to a live MySQL cluster that
+is under a high load serving random queries from the users of our product.
+While data consistency issues during the export can be easily solved with a
+staging table, there is still a problem: the performance impact caused by the
+heavy export.
+
+First off, the resources of MySQL dedicated to the import process can affect
+the performance of the live product, both on the master and on the slaves.
+Second, even if the servers can handle the import with no significant
+performance impact (mysqlimport should be relatively cheap), importing big
+tables can cause serious replication lag in the cluster risking data
+inconsistency.
+
+With +-D sqoop.mysql.export.sleep.ms=time+, where _time_ is a value in
+milliseconds, you can let the server relax between checkpoints and the replicas
+catch up by pausing the export process after transferring the number of bytes
+specified in +sqoop.mysql.export.checkpoint.bytes+. Experiment with different
+settings of these two parameters to archieve an export pace that doesn't
+endanger the stability of your MySQL cluster.
+
IMPORTANT: Note that any arguments to Sqoop that are of the form +-D
parameter=value+ are Hadoop _generic arguments_ and must appear before
any tool-specific arguments (for example, +\--connect+, +\--table+, etc).
+Don't forget that these parameters only work with the +\--direct+ flag set.
PostgreSQL
~~
{code}
was (Author: tcz):
diff --git a/src/docs/user/compatibility.txt
b/src/docs/user/compatibility.txt
index 3576fd7..e8218d6 100644
--- a/src/docs/user/compatibility.txt
+++ b/src/docs/user/compatibility.txt
@@ -138,9 +138,30 @@ bytes. Set _size_ to 0 to disable intermediate checkpoints,
but individual files being exported will continue to be committed
independently of one another.
+Sometimes you need to export large data with Sqoop to a live MySQL cluster that
+is under a high load serving random queries from the users of our product.
+While data consistency issues during the export can be easily solved with a
+staging table, there is still a problem: the performance impact caused by the
+heavy export.
+
+First off, the resources of MySQL dedicated to the import process can affect
+the performance of the live product, both on the master and on the slaves.
+Second, even if the servers can handle the import with no significant
+performance impact (mysqlimport should be relatively cheap), importing big
+tables can cause serious replication lag in the cluster risking data
+inconsistency.
+
+With +-D sqoop.mysql.export.sleep.ms=time+, where _time_ is a value in
+milliseconds, you can let the server relax between checkpoints and the replicas
+catch up by pausing the export process after transferring the number of bytes
+specified in +sqoop.mysql.export.checkpoint.bytes+. Experiment with different
+settings of these two parameters to archieve an export pace that doesn't
+endanger the stability of your MySQL cluster.
+
IMPORTANT: Note that any arguments to Sqoop that are of the form +-D
parameter=value+ are Hadoop _generic arguments_ and must appear before
any tool-specific arguments (for example, +\--connect+, +\--table+, etc).
+Don't forget that these parameters only work with the +\--direct+ flag set.
PostgreSQL
~~
Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct
MySQL exports
--
Key: SQOOP-683
URL: https://issues.apache.org/jira/browse/SQOOP-683
Project: Sqoop
Issue Type: Sub-task
Components: connectors/mysql, docs
Affects Versions: 1.4.2
Reporter: Zoltan Toth-Czifra
Assignee: Zoltan Toth-Czifra
Priority: Trivial
Fix For: 1.4.3
Documenting feature added in parent task.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-683) Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports
[ https://issues.apache.org/jira/browse/SQOOP-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490757#comment-13490757 ] Jarek Jarcec Cecho commented on SQOOP-683: -- Hi Zoltan, thank you very much for your time and effort. Would you mind uploading your patch as a file and and the same time uploading it to Review board? Jarcec Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports -- Key: SQOOP-683 URL: https://issues.apache.org/jira/browse/SQOOP-683 Project: Sqoop Issue Type: Sub-task Components: connectors/mysql, docs Affects Versions: 1.4.2 Reporter: Zoltan Toth-Czifra Assignee: Zoltan Toth-Czifra Priority: Trivial Fix For: 1.4.3 Documenting feature added in parent task. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: SQOOP-683 Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7880/ --- Review request for Sqoop. Description --- Code review for SQOOP-683, see https://issues.apache.org/jira/browse/SQOOP-683. Diffs - src/docs/user/compatibility.txt 3576fd7 Diff: https://reviews.apache.org/r/7880/diff/ Testing --- Converted to XML with asciidoc, the affected part: simparaSometimes you need to export large data with Sqoop to a live MySQL cluster that is under a high load serving random queries from the users of our product. While data consistency issues during the export can be easily solved with a staging table, there is still a problem: the performance impact caused by the heavy export./simpara simparaFirst off, the resources of MySQL dedicated to the import process can affect the performance of the live product, both on the master and on the slaves. Second, even if the servers can handle the import with no significant performance impact (mysqlimport should be relatively cheap), importing big tables can cause serious replication lag in the cluster risking data inconsistency./simpara simparaWith literal-D sqoop.mysql.export.sleep.ms=time/literal, where emphasistime/emphasis is a value in milliseconds, you can let the server relax between checkpoints and the replicas catch up by pausing the export process after transferring the number of bytes specified in literalsqoop.mysql.export.checkpoint.bytes/literal. Experiment with different settings of these two parameters to archieve an export pace that doesn#8217;t endanger the stability of your MySQL cluster./simpara importantsimparaNote that any arguments to Sqoop that are of the form literal-D parameter=value/literal are Hadoop emphasisgeneric arguments/emphasis and must appear before any tool-specific arguments (for example, literal--connect/literal, literal--table/literal, etc). Don#8217;t forget that these parameters only work with the literal--direct/literal flag set./simpara/important Thanks, Zoltán Tóth-Czifra
[jira] [Commented] (SQOOP-683) Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports
[ https://issues.apache.org/jira/browse/SQOOP-683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13490792#comment-13490792 ] Zoltan Toth-Czifra commented on SQOOP-683: -- Sure, not a problem: https://reviews.apache.org/r/7880 Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports -- Key: SQOOP-683 URL: https://issues.apache.org/jira/browse/SQOOP-683 Project: Sqoop Issue Type: Sub-task Components: connectors/mysql, docs Affects Versions: 1.4.2 Reporter: Zoltan Toth-Czifra Assignee: Zoltan Toth-Czifra Priority: Trivial Fix For: 1.4.3 Documenting feature added in parent task. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: SQOOP-666 Introduce execution engine
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7881/ --- Review request for Sqoop. Description --- I've abstracted execution engine to it's own module so that sqoop-core do no longer have dependencies on hadoop libraries. This patch seems more heavier than it actually is because I was moving a lot of files around - I however did not made any changes to them. This addresses bug SQOOP-666. https://issues.apache.org/jira/browse/SQOOP-666 Diffs - core/pom.xml 028c24063ffa6cd3324c83ca02d051dae7363f41 core/src/main/java/org/apache/sqoop/framework/ExecutionEngine.java PRE-CREATION core/src/main/java/org/apache/sqoop/framework/FrameworkConstants.java d6e70ca70b019a5d75abdfcb98c0d0a0390d63fc core/src/main/java/org/apache/sqoop/framework/FrameworkError.java 19d0d8757dcc2928cf3a001dd5a46bc05ab2c189 core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 604d403d4e634cac51bccd0ea7aa09fc462d387e core/src/main/java/org/apache/sqoop/framework/SubmissionEngine.java f4ad3f5a12e0871362a44832f668d554b00b4a3d core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java 27b0566c274819d5a1c131fea26a751d3099161d core/src/main/java/org/apache/sqoop/job/JobConstants.java 19ac91e00ad92f6b17bd6e210c1b8e293e8f2960 core/src/main/java/org/apache/sqoop/job/PrefixContext.java 5488b46f87dabff874e4a2da6d1b8c7a711072f2 core/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java 1235d1d90e14c35ac898f9d4dca0f969b561e47e core/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java 36aa11f7f2871e99fbc84d518bfa524d108f3cf4 core/src/main/java/org/apache/sqoop/job/io/Data.java 4ddd13209e4b581d32fc46508826e7e40683b3ee core/src/main/java/org/apache/sqoop/job/io/FieldTypes.java e96dc6e2f414841fba7c4dc49fe9b0ba9a0b18a1 core/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java 59baaf688cf9af5ff4562335f75e97fd114c5b85 core/src/main/java/org/apache/sqoop/job/mr/SqoopFileOutputFormat.java c465f10113b8c1170336bb41ba2ad7244c58ede4 core/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 8fcdc99daa08a02775ed2ba0708eccfdfdbe078b core/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 6892b4b5f48ada17185ca2bd170a8af3d87483db core/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 1242f90c8de4f267509592abf15195085151bab2 core/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 96e1533ec1899f618b5a49c7a4fabac1abbb541a core/src/main/java/org/apache/sqoop/job/mr/SqoopReducer.java d2361482771eea1a34b14c767952c5592f89c45a core/src/main/java/org/apache/sqoop/job/mr/SqoopSplit.java 7dc9541350a85e4178975b45e97619f99d1aa509 core/src/test/java/org/apache/sqoop/io/TestData.java 9fe9d4130c4c45f8d2909f2cd4213a22dac8f8d9 core/src/test/java/org/apache/sqoop/job/FileUtils.java e685883e9c540c06723d1927d52ddfc0073b3fae core/src/test/java/org/apache/sqoop/job/JobUtils.java e6ead3f265d43b7a9797461c2bed804102a562ef core/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java c74faa2bbaf9ea808e69b02a0b42f16469e17322 core/src/test/java/org/apache/sqoop/job/TestJobEngine.java 51dddb42e3840ffa7b64f186d83b71ea848b9a10 core/src/test/java/org/apache/sqoop/job/TestMapReduce.java 94ab560490cff8ccbfc24950c76ea3cb485ba37a dist/src/main/server/conf/sqoop.properties d429c3a8d000819d2ed6e6a8e91405c3937606a7 execution/mapreduce/pom.xml PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MRSubmissionRequest.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/job/PrefixContext.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/job/io/Data.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/job/io/FieldTypes.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopFileOutputFormat.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java PRE-CREATION
[jira] [Updated] (SQOOP-666) Introduce execution engine
[ https://issues.apache.org/jira/browse/SQOOP-666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Jarcec Cecho updated SQOOP-666: - Attachment: bugSQOOP-666.patch Introduce execution engine -- Key: SQOOP-666 URL: https://issues.apache.org/jira/browse/SQOOP-666 Project: Sqoop Issue Type: New Feature Affects Versions: 2.0.0 Reporter: Jarek Jarcec Cecho Assignee: Jarek Jarcec Cecho Fix For: 2.0.0 Attachments: bugSQOOP-666.patch We should introduce execution engine abstraction next to submission engine to allow multiple execution engines in the future (for example pure YARN application). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (SQOOP-679) Wrap exception to MSubmission metadata object
[ https://issues.apache.org/jira/browse/SQOOP-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Jarcec Cecho reassigned SQOOP-679: Assignee: Jarek Jarcec Cecho Wrap exception to MSubmission metadata object - Key: SQOOP-679 URL: https://issues.apache.org/jira/browse/SQOOP-679 Project: Sqoop Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Jarek Jarcec Cecho Assignee: Jarek Jarcec Cecho Fix For: 2.0.0 We should store any exception generated during executing submission into appropriate MSubmission metadata class so that it can be moved to the client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-679) Wrap exception to MSubmission metadata object
[ https://issues.apache.org/jira/browse/SQOOP-679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Jarcec Cecho updated SQOOP-679: - Attachment: bugSQOOP-679.patch Wrap exception to MSubmission metadata object - Key: SQOOP-679 URL: https://issues.apache.org/jira/browse/SQOOP-679 Project: Sqoop Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Jarek Jarcec Cecho Assignee: Jarek Jarcec Cecho Fix For: 2.0.0 Attachments: bugSQOOP-679.patch We should store any exception generated during executing submission into appropriate MSubmission metadata class so that it can be moved to the client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: SQOOP-666 Introduce execution engine
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7881/#review13111 --- Thanks for the patch. Two issues below: execution/mapreduce/src/test/java/org/apache/sqoop/job/io/TestData.java https://reviews.apache.org/r/7881/#comment28244 should be org.apache.sqoop.job.io. execution/mapreduce/src/test/java/org/apache/sqoop/job/io/TestData.java https://reviews.apache.org/r/7881/#comment28245 remove this line. - Bilung Lee On Nov. 5, 2012, 6:49 p.m., Jarek Cecho wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7881/ --- (Updated Nov. 5, 2012, 6:49 p.m.) Review request for Sqoop. Description --- I've abstracted execution engine to it's own module so that sqoop-core do no longer have dependencies on hadoop libraries. This patch seems more heavier than it actually is because I was moving a lot of files around - I however did not made any changes to them. This addresses bug SQOOP-666. https://issues.apache.org/jira/browse/SQOOP-666 Diffs - core/pom.xml 028c24063ffa6cd3324c83ca02d051dae7363f41 core/src/main/java/org/apache/sqoop/framework/ExecutionEngine.java PRE-CREATION core/src/main/java/org/apache/sqoop/framework/FrameworkConstants.java d6e70ca70b019a5d75abdfcb98c0d0a0390d63fc core/src/main/java/org/apache/sqoop/framework/FrameworkError.java 19d0d8757dcc2928cf3a001dd5a46bc05ab2c189 core/src/main/java/org/apache/sqoop/framework/FrameworkManager.java 604d403d4e634cac51bccd0ea7aa09fc462d387e core/src/main/java/org/apache/sqoop/framework/SubmissionEngine.java f4ad3f5a12e0871362a44832f668d554b00b4a3d core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java 27b0566c274819d5a1c131fea26a751d3099161d core/src/main/java/org/apache/sqoop/job/JobConstants.java 19ac91e00ad92f6b17bd6e210c1b8e293e8f2960 core/src/main/java/org/apache/sqoop/job/PrefixContext.java 5488b46f87dabff874e4a2da6d1b8c7a711072f2 core/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java 1235d1d90e14c35ac898f9d4dca0f969b561e47e core/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java 36aa11f7f2871e99fbc84d518bfa524d108f3cf4 core/src/main/java/org/apache/sqoop/job/io/Data.java 4ddd13209e4b581d32fc46508826e7e40683b3ee core/src/main/java/org/apache/sqoop/job/io/FieldTypes.java e96dc6e2f414841fba7c4dc49fe9b0ba9a0b18a1 core/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java 59baaf688cf9af5ff4562335f75e97fd114c5b85 core/src/main/java/org/apache/sqoop/job/mr/SqoopFileOutputFormat.java c465f10113b8c1170336bb41ba2ad7244c58ede4 core/src/main/java/org/apache/sqoop/job/mr/SqoopInputFormat.java 8fcdc99daa08a02775ed2ba0708eccfdfdbe078b core/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java 6892b4b5f48ada17185ca2bd170a8af3d87483db core/src/main/java/org/apache/sqoop/job/mr/SqoopNullOutputFormat.java 1242f90c8de4f267509592abf15195085151bab2 core/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java 96e1533ec1899f618b5a49c7a4fabac1abbb541a core/src/main/java/org/apache/sqoop/job/mr/SqoopReducer.java d2361482771eea1a34b14c767952c5592f89c45a core/src/main/java/org/apache/sqoop/job/mr/SqoopSplit.java 7dc9541350a85e4178975b45e97619f99d1aa509 core/src/test/java/org/apache/sqoop/io/TestData.java 9fe9d4130c4c45f8d2909f2cd4213a22dac8f8d9 core/src/test/java/org/apache/sqoop/job/FileUtils.java e685883e9c540c06723d1927d52ddfc0073b3fae core/src/test/java/org/apache/sqoop/job/JobUtils.java e6ead3f265d43b7a9797461c2bed804102a562ef core/src/test/java/org/apache/sqoop/job/TestHdfsLoad.java c74faa2bbaf9ea808e69b02a0b42f16469e17322 core/src/test/java/org/apache/sqoop/job/TestJobEngine.java 51dddb42e3840ffa7b64f186d83b71ea848b9a10 core/src/test/java/org/apache/sqoop/job/TestMapReduce.java 94ab560490cff8ccbfc24950c76ea3cb485ba37a dist/src/main/server/conf/sqoop.properties d429c3a8d000819d2ed6e6a8e91405c3937606a7 execution/mapreduce/pom.xml PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MRSubmissionRequest.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/job/JobConstants.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/job/PrefixContext.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsSequenceImportLoader.java PRE-CREATION execution/mapreduce/src/main/java/org/apache/sqoop/job/etl/HdfsTextImportLoader.java
[jira] [Updated] (SQOOP-666) Introduce execution engine
[ https://issues.apache.org/jira/browse/SQOOP-666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jarek Jarcec Cecho updated SQOOP-666: - Attachment: bugSQOOP-666.patch Introduce execution engine -- Key: SQOOP-666 URL: https://issues.apache.org/jira/browse/SQOOP-666 Project: Sqoop Issue Type: New Feature Affects Versions: 2.0.0 Reporter: Jarek Jarcec Cecho Assignee: Jarek Jarcec Cecho Fix For: 2.0.0 Attachments: bugSQOOP-666.patch, bugSQOOP-666.patch We should introduce execution engine abstraction next to submission engine to allow multiple execution engines in the future (for example pure YARN application). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
