[
https://issues.apache.org/jira/browse/TEZ-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yingda Chen updated TEZ-4018:
-----------------------------
Description:
A high-level description is provided here for now, *will follow up with a
proper design doc later.*
We have encountered a few application scenarios for dynamic (logical) DAG here
in our system. A typical one is for a distributed query to be able to
dynamically choose among two execution paths, say among hash join and sorted
merge join.
This can be solved by allowing TEZ to execute "conditional DAG". By that we
mean that a DAG may have some conditional vertices: several conditional
vertices can form a conditional group, and inside each group, only one will be
chosen for execution at runtime.
To allow decision at runtime, each conditional group will be associated with a
“control vertex”. A control vertex can be a pure virtual component that lives
only on AM with its VertexImpl and VetexManager, but has no associated tasks
(DoP = 0). It can be also be extended to have physical tasks associated with
it, in the case where intensive computation may be required to make a control
decision.
A upstream vertex (that produces input data to vertices in downstream
conditional group) will be connected to the control vertex, as well as all
conditional vertices in the group at the same time. This allows its VMEs and
DMEs to be sent to all of them. Upon receipt of (enough) VMEs, the control
vertex would be able to gather enough runtime statistics and determine (by
user-supplied logic) which downstream vertex should be scheduled (and which
should be skipped). Such decision will effectively “uncondition” the DAG and
determines the sub-graph that is actually being executed.
Such conditional vertices can be useful to enable scenarios such as conditional
join, where a query can choose between hash join and sorted merge join at
runtime, base on the precious runtime statistics (e.g., output size) of
upstream mapper.
was:
A high-level description is provided here for now, will follow up with a proper
design doc later.
We have encountered a few application scenarios for dynamic (logical) DAG here
in our system. A typical one is for a distributed query to be able to
dynamically choose among two execution paths, say among hash join and sorted
merge join.
This can be solved by allowing TEZ to execute "conditional DAG". By that we
mean that a DAG may have some conditional vertices: several conditional
verteices can form a conditioanl group, and insided each group, only one will
be chosen for execution at runtime.
To allow decision at runtime, each conditional group will be associated with a
“control vertex”. A control vertex can be a pure virtual component that lives
only on AM with its VertexImpl and VetexManager, but has no associated tasks
(DoP = 0). It can be also be extended to have physical tasks associated with
it, in the case where intensitve compuation may be required to make a control
decision.
A upstream vertex (that produces input data to verteices in downstream
conditioanl group) will be connected to the control vertex, as well as all
conditional vertices in the group at the same time. This allows its VMEs and
DMEs to be sent to all of them. Upon receipt of (enough) VMEs, the control
vertex would be able to gather enough runtime statistics and determine (by
user-supplied logic) which downstream vertex should be scheduled (and which
should be skipped). Such decision will effectively “uncondition” the DAG and
determines the sub-graph that is actually being executed.
Such conditional verteices can be useful to enable scenarios such as
conditional join, where a query can choose between hash join and sorted merge
join at runtime, base on the precious runtime statistics (e.g., output size) of
upstream mapper.
> Allow conditional vertex in DAG execution
> -----------------------------------------
>
> Key: TEZ-4018
> URL: https://issues.apache.org/jira/browse/TEZ-4018
> Project: Apache Tez
> Issue Type: New Feature
> Reporter: Yingda Chen
> Priority: Major
>
> A high-level description is provided here for now, *will follow up with a
> proper design doc later.*
> We have encountered a few application scenarios for dynamic (logical) DAG
> here in our system. A typical one is for a distributed query to be able to
> dynamically choose among two execution paths, say among hash join and sorted
> merge join.
> This can be solved by allowing TEZ to execute "conditional DAG". By that we
> mean that a DAG may have some conditional vertices: several conditional
> vertices can form a conditional group, and inside each group, only one will
> be chosen for execution at runtime.
> To allow decision at runtime, each conditional group will be associated with
> a “control vertex”. A control vertex can be a pure virtual component that
> lives only on AM with its VertexImpl and VetexManager, but has no associated
> tasks (DoP = 0). It can be also be extended to have physical tasks associated
> with it, in the case where intensive computation may be required to make a
> control decision.
> A upstream vertex (that produces input data to vertices in downstream
> conditional group) will be connected to the control vertex, as well as all
> conditional vertices in the group at the same time. This allows its VMEs and
> DMEs to be sent to all of them. Upon receipt of (enough) VMEs, the control
> vertex would be able to gather enough runtime statistics and determine (by
> user-supplied logic) which downstream vertex should be scheduled (and which
> should be skipped). Such decision will effectively “uncondition” the DAG and
> determines the sub-graph that is actually being executed.
> Such conditional vertices can be useful to enable scenarios such as
> conditional join, where a query can choose between hash join and sorted merge
> join at runtime, base on the precious runtime statistics (e.g., output size)
> of upstream mapper.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
