[
https://issues.apache.org/jira/browse/SQOOP-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166282#comment-14166282
]
Hudson commented on SQOOP-1496:
-------------------------------
ABORTED: Integrated in Sqoop2-hadoop200 #547 (See
[https://builds.apache.org/job/Sqoop2-hadoop200/547/])
SQOOP-1496: Sqoop2: Revisit/Refactor the SubmissionEngine/ExecutionEngine APIs
(abraham:
https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=3d539dd4d7477324dfe62a4e57f684351769b000)
*
submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java
* core/src/main/java/org/apache/sqoop/framework/SubmissionEngine.java
*
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopDestroyerExecutor.java
* common/src/main/java/org/apache/sqoop/model/MSubmission.java
* common/src/main/java/org/apache/sqoop/json/SubmissionBean.java
* core/src/test/java/org/apache/sqoop/framework/TestSubmissionRequest.java
* core/src/test/java/org/apache/sqoop/framework/TestFrameworkValidator.java
* shell/src/main/java/org/apache/sqoop/shell/utils/SubmissionDisplayer.java
*
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormat.java
* core/src/test/java/org/apache/sqoop/repository/TestJdbcRepository.java
*
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MRSubmissionRequest.java
* spi/src/main/java/org/apache/sqoop/validation/Validator.java
* spi/src/main/java/org/apache/sqoop/job/etl/To.java
*
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MapreduceExecutionEngine.java
*
connector/connector-generic-jdbc/src/test/java/org/apache/sqoop/connector/jdbc/TestToInitializer.java
*
connector/connector-sdk/src/test/java/org/apache/sqoop/connector/idf/CSVIntermediateDataFormatTest.java
* core/src/test/java/org/apache/sqoop/framework/TestJobManager.java
*
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopFileOutputFormat.java
*
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ConfigurationUtils.java
* execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopMapper.java
* core/src/main/java/org/apache/sqoop/framework/ExecutionEngine.java
* core/src/test/java/org/apache/sqoop/framework/TestJobRequest.java
* core/src/main/java/org/apache/sqoop/connector/ConnectorManager.java
*
connector/connector-sdk/src/main/java/org/apache/sqoop/connector/idf/IntermediateDataFormat.java
* core/src/main/java/org/apache/sqoop/framework/JobRequest.java
*
connector/connector-sdk/src/test/java/org/apache/sqoop/connector/idf/TestCSVIntermediateDataFormat.java
* spi/src/main/java/org/apache/sqoop/job/etl/From.java
* spi/src/main/java/org/apache/sqoop/job/etl/Transferable.java
*
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/ProgressRunnable.java
* core/src/main/java/org/apache/sqoop/framework/SubmissionRequest.java
*
execution/mapreduce/src/main/java/org/apache/sqoop/execution/mapreduce/MRJobRequest.java
* spi/src/main/java/org/apache/sqoop/job/etl/CallbackBase.java
*
execution/mapreduce/src/main/java/org/apache/sqoop/job/mr/SqoopOutputFormatLoadExecutor.java
* core/src/main/java/org/apache/sqoop/framework/JobManager.java
* common/src/test/java/org/apache/sqoop/json/TestSubmissionBean.java
> Sqoop2: Revisit/Refactor the SubmissionEngine/ExecutionEngine APIs
> ------------------------------------------------------------------
>
> Key: SQOOP-1496
> URL: https://issues.apache.org/jira/browse/SQOOP-1496
> Project: Sqoop
> Issue Type: Improvement
> Reporter: Veena Basavaraj
> Assignee: Veena Basavaraj
> Fix For: from/to
>
> Attachments: SQOOP-1496.patch, Screen Shot 2014-09-08 at 2.35.06
> PM.png
>
>
> Here are the things this ticket should address
> 1. Refactor the submit() in JobManager ( it is way too long for any good unit
> testing to happen)
> 2. Revisit the Execution Engine and Submission Engine relationship in the
> code.
> ExecutionEngine api should probably not create a submissionRequest. It should
> be the inverse. The Submission Engine is an abstraction of a JobTracker/
> YARN/ Mesos/ OOzie. It will create a job request for the execution engine
> while it submits the job into the execution engine
> a) Suggest renaming SubmissionEngine to a JobSubmissionManager.
>
> b) SubmissionRequest is a JobRunRequest. See the java doc on the current code
> SubmissionRequest, it should already hint us that this object holds the job
> context to aid the execution engine
> c) Change the apis in Execution Engine/ SubmissionManager to reflect the
> same, i.e rename prepareSubmissionRequest to prepareJobRequest
> d) The Submission term is overloaded in the code base. In one case it is used
> to represent the JobRuns( history of the jobs) and in another place it is the
> submission engine such as JobTracker/YARN/ OOzie. The act of submitting
> creates a submission record. So submission is an artifact of
> completing//failing a submit request. Prior to that it is still a job request.
> YARN is an example of a submission engine and it can do much more than
> submissions, so at some point we should revisit and outline what the
> responsibilities of the submission engine API should be,
> e) define other responsibilities of the submission engine. For instance, if
> we are to add more monitoring/ tracking into how the job execution is
> progressing, we should be able to add those apis to this entity. There is
> more room for improvement for this api
> 3. Lastly add unit test for the JobManager!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
