[
https://issues.apache.org/jira/browse/HIVE-8220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-8220:
------------------------------
Description:
This is a followup for HIVE-7053. Currently the code to split the operator tree
and to generate tasks is mingled and thus hard to understand and maintain.
Logically the two seems independent. This can be improved by modulizing both.
The following might be helpful:
{code}
@Override
protected void generateTaskTree(List<Task<? extends Serializable>> rootTasks,
ParseContext pCtx,
List<Task<MoveWork>> mvTask, Set<ReadEntity> inputs, Set<WriteEntity>
outputs)
throws SemanticException {
// 1. Identify if the plan is for multi-insert and split the plan if necessary
List<Set<Operator>> operatorSets = multiInsertSplit(...);
// 2. For each operator set, generate a task.
for (Set<Operator> topOps : operatorSets) {
SparkTask task = generateTask(topOps);
...
}
// 3. wire up the tasks
...
}
{code}
was:
This is a followup for HIVE-7053. Currently the code to split the operator tree
and to generate tasks is mingled and thus hard to understand and maintain.
Logically the two seems independent. This can be improved by modulizing both.
The following might be helpful:
{code}
@Override
protected void generateTaskTree(List<Task<? extends Serializable>> rootTasks,
ParseContext pCtx,
List<Task<MoveWork>> mvTask, Set<ReadEntity> inputs, Set<WriteEntity>
outputs)
throws SemanticException {
// 1. Identify if the plan is for multi-insert and split the plan if necessary
List<Set<Operator>> operatorSets = multiInsertSplit(...);
// 2. For each operator set, generate a task.
for (Set<Operator> topOps : operatorSets) {
SparkTask task = generateTask(topOps);
...
}
// 3. wire up the tasks
...
}
{code}
> Refactor multi-insert code such that plan splitting and task generation are
> modular and reusable [Spark Branch]
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-8220
> URL: https://issues.apache.org/jira/browse/HIVE-8220
> Project: Hive
> Issue Type: Improvement
> Components: Spark
> Reporter: Xuefu Zhang
> Labels: Spark-M1
>
> This is a followup for HIVE-7053. Currently the code to split the operator
> tree and to generate tasks is mingled and thus hard to understand and
> maintain. Logically the two seems independent. This can be improved by
> modulizing both. The following might be helpful:
> {code}
> @Override
> protected void generateTaskTree(List<Task<? extends Serializable>> rootTasks,
> ParseContext pCtx,
> List<Task<MoveWork>> mvTask, Set<ReadEntity> inputs, Set<WriteEntity>
> outputs)
> throws SemanticException {
> // 1. Identify if the plan is for multi-insert and split the plan if necessary
> List<Set<Operator>> operatorSets = multiInsertSplit(...);
> // 2. For each operator set, generate a task.
> for (Set<Operator> topOps : operatorSets) {
> SparkTask task = generateTask(topOps);
> ...
> }
> // 3. wire up the tasks
> ...
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
