[
https://issues.apache.org/jira/browse/SQOOP-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Veena Basavaraj updated SQOOP-1496:
-----------------------------------
Description:
Here are the things this ticket should address
1. Refactor the submit() in JobManager ( it is way too long for any good unit
testing to happen)
2. Revisit the Execution Engine and Submission Engine relationship in the code.
ExecutionEngine api should probably not create a submissionRequest. It should
be the inverse. The Submission Engine is an abstraction of a JobTracker/ YARN/
Mesos/ OOzie. It will create a execution request for the execution engine while
it submits the job into the execution engine
a) Suggest renaming SubmissionEngine to a JobSubmissionManager. Rename
MapreduceSubmissionEngine to a JobTrackerSubmisisonEngine. Since I could
actually use the JobTracker with a non map reduce execution engine as well (
such as Spark )
b) SubmissionRequest is a JobRunRequest. See the java doc on the current code
SubmissionRequest, it should already hint us that this object holds the job
context to aid the execution engine
c) Change the apis in Execution Engine/ SubmissionManager to reflect the same,
i.e rename prepareSubmissionRequest to prepareExecutionRequest
d) The Submission term is overloaded in the code base. In one case it is used
to represent the JobRuns( history of the jobs) and in another place it is the
submission engine such as JobTracker/YARN/ OOzie. These are unrelated and we
should not be overloading this term in Sqoop
YARN is an example of a submission engine and it can do much more than
submissions. So our design and terminology should be much more generic than
what it is now.
e) define other responsibilities of the submission engine. For instance, if we
are to add more monitoring/ tracking into how the job execution is progressing,
we should be able to add those apis to this entity. There is more room for
improvement for this api
3. Lastly add unit test for the JobManager!
was:
Here are the things this ticket should address
1. Refactor the submit() in JobManager ( it is way too long for any good unit
testing to happen)
2. Revisit the Execution Engine and Submission Engine relationship in the code.
ExecutionEngine api should probably not create a submissionRequest. It should
be the inverse. The Submission Engine is an abstraction of a JobTracker/ YARN/
Mesos/ OOzie. It will create a execution request for the execution engine while
it submits the job into the execution engine
a) Suggest renaming SubmissionEngine to a JobSubmissionManager. Rename
MapreduceSubmissionEngine to a JobTrackerSubmisisonEngine. Since I could
actually use the JobTracker with a non map reduce execution engine as well (
such as Spark )
b) SubmissionRequest is a JobExecutionRequest. See the java doc on the current
code SubmissionRequest, it should already hint us that this object holds the
context for the execution engine
c) Change the apis in Execution Engine/ SubmissionManager to reflect the same,
i.e rename prepareSubmissionRequest to prepareExecutionRequest
d) The Submission term is overloaded in the code base. In one case it is used
to represent the JobRuns( history of the jobs) and in another place it is the
submission engine such as JobTracker/YARN/ OOzie. These are unrelated and we
should not be overloading this term in Sqoop
YARN is an example of a submission engine and it can do much more than
submissions. So our design and terminology should be much more generic than
what it is now.
e) define other responsibilities of the submission engine. For instance, if we
are to add more monitoring/ tracking into how the job execution is progressing,
we should be able to add those apis to this entity. There is more room for
improvement for this api
3. Lastly add unit test for the JobManager!
> Sqoop2: Revisit/Refactor the SubmissionEngine/ExecutionEngine APIs
> ------------------------------------------------------------------
>
> Key: SQOOP-1496
> URL: https://issues.apache.org/jira/browse/SQOOP-1496
> Project: Sqoop
> Issue Type: Improvement
> Reporter: Veena Basavaraj
> Assignee: Veena Basavaraj
> Attachments: Screen Shot 2014-09-08 at 2.35.06 PM.png
>
>
> Here are the things this ticket should address
> 1. Refactor the submit() in JobManager ( it is way too long for any good unit
> testing to happen)
> 2. Revisit the Execution Engine and Submission Engine relationship in the
> code.
> ExecutionEngine api should probably not create a submissionRequest. It should
> be the inverse. The Submission Engine is an abstraction of a JobTracker/
> YARN/ Mesos/ OOzie. It will create a execution request for the execution
> engine while it submits the job into the execution engine
> a) Suggest renaming SubmissionEngine to a JobSubmissionManager. Rename
> MapreduceSubmissionEngine to a JobTrackerSubmisisonEngine. Since I could
> actually use the JobTracker with a non map reduce execution engine as well (
> such as Spark )
>
> b) SubmissionRequest is a JobRunRequest. See the java doc on the current code
> SubmissionRequest, it should already hint us that this object holds the job
> context to aid the execution engine
> c) Change the apis in Execution Engine/ SubmissionManager to reflect the
> same, i.e rename prepareSubmissionRequest to prepareExecutionRequest
> d) The Submission term is overloaded in the code base. In one case it is used
> to represent the JobRuns( history of the jobs) and in another place it is the
> submission engine such as JobTracker/YARN/ OOzie. These are unrelated and we
> should not be overloading this term in Sqoop
> YARN is an example of a submission engine and it can do much more than
> submissions. So our design and terminology should be much more generic than
> what it is now.
> e) define other responsibilities of the submission engine. For instance, if
> we are to add more monitoring/ tracking into how the job execution is
> progressing, we should be able to add those apis to this entity. There is
> more room for improvement for this api
> 3. Lastly add unit test for the JobManager!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
