[
https://issues.apache.org/jira/browse/SQOOP-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131726#comment-14131726
]
Veena Basavaraj edited comment on SQOOP-1496 at 9/12/14 4:37 PM:
-----------------------------------------------------------------
- renamed BaseCallbacks to Transferrable
- renamed SubmissionRequest to JobRequest , I am keeping the SubmissionEngine
terminology as it is, and will revisit at a later point when a new
submission/execution engine will be supported
- fixed #2 ( execution engine now creates a job request and a submission
request. submission engine submits the job request)... uuf!
- fixed #1 ( refactored to smaller methods, so that the code is readable and
testable)
- fixed #3 ( added a unit test for the job manager, one of the most important
class that has no tests)
was (Author: vybs):
- renamed BaseCallbacks to Transferrable
- renamed SubmissionRequest to JobRequest
- fixed #2 ( execution engine now creates a job request and a submission
request. submission engine submits the job request)... uuf!
- fixed #1 ( refactored to smaller methods, so that the code is readable and
testable)
- fixed #3 ( added a unit test for the job manager, one of the most important
class that has no tests)
> Sqoop2: Revisit/Refactor the SubmissionEngine/ExecutionEngine APIs
> ------------------------------------------------------------------
>
> Key: SQOOP-1496
> URL: https://issues.apache.org/jira/browse/SQOOP-1496
> Project: Sqoop
> Issue Type: Improvement
> Reporter: Veena Basavaraj
> Assignee: Veena Basavaraj
> Attachments: Screen Shot 2014-09-08 at 2.35.06 PM.png
>
>
> Here are the things this ticket should address
> 1. Refactor the submit() in JobManager ( it is way too long for any good unit
> testing to happen)
> 2. Revisit the Execution Engine and Submission Engine relationship in the
> code.
> ExecutionEngine api should probably not create a submissionRequest. It should
> be the inverse. The Submission Engine is an abstraction of a JobTracker/
> YARN/ Mesos/ OOzie. It will create a execution request for the execution
> engine while it submits the job into the execution engine
> a) Suggest renaming SubmissionEngine to a JobSubmissionManager. Rename
> MapreduceSubmissionEngine to a JobTrackerSubmisisonEngine. Since I could
> actually use the JobTracker with a non map reduce execution engine as well (
> such as Spark )
>
> b) SubmissionRequest is a JobRunRequest. See the java doc on the current code
> SubmissionRequest, it should already hint us that this object holds the job
> context to aid the execution engine
> c) Change the apis in Execution Engine/ SubmissionManager to reflect the
> same, i.e rename prepareSubmissionRequest to prepareExecutionRequest
> d) The Submission term is overloaded in the code base. In one case it is used
> to represent the JobRuns( history of the jobs) and in another place it is the
> submission engine such as JobTracker/YARN/ OOzie. These are unrelated and we
> should not be overloading this term in Sqoop
> YARN is an example of a submission engine and it can do much more than
> submissions. So our design and terminology should be much more generic than
> what it is now.
> e) define other responsibilities of the submission engine. For instance, if
> we are to add more monitoring/ tracking into how the job execution is
> progressing, we should be able to add those apis to this entity. There is
> more room for improvement for this api
> 3. Lastly add unit test for the JobManager!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
