[jira] [Commented] (TEZ-3998) Allow CONCURRENT edge property in DAG construction and introduce ConcurrentSchedulingType
[
https://issues.apache.org/jira/browse/TEZ-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652201#comment-16652201
]
ASF GitHub Bot commented on TEZ-3998:
-
Github user anicoara commented on the issue:
https://github.com/apache/tez/pull/33
@yingdachen since this is part of a larger end-to-end feature (Tez-3997)
would it be better for you to own a feature branch, in which you commit for the
subtasks, and later, when the feature is complete, merge into master?
It is easier to provide meaningful feedback when seeing how it all comes
together - right now, most of the changes are stubs.
> Allow CONCURRENT edge property in DAG construction and introduce
> ConcurrentSchedulingType
> -
>
> Key: TEZ-3998
> URL: https://issues.apache.org/jira/browse/TEZ-3998
> Project: Apache Tez
> Issue Type: Task
>Reporter: Yingda Chen
>Assignee: Yingda Chen
>Priority: Major
>
> This is the first task related to TEZ-3997
>
> |Note: There is no API change in this proposed change. The majority of this
> change will be lifting some existing constraints against CONCURRENT edge
> type, and addition of a VertexMangerPlugin implementation.|
>
> This includes enabling the CONCURRENT SchedulingType as a valid edge
> property, by removing all the sanity check against CONCURRENT during DAG
> construction/execution. A new VertexManagerPlugin (namely
> VertexManagerWithConcurrentInput) will be implemented for vertex with
> incoming concurrent edge(s).
> In addition, we will assume in this change that
> * A vertex *cannot* have both SEQUENTIAL and CONCURRENT incoming edges
> * No shuffle or data movement is handled by Tez framework when two vertices
> are connected through a CONCURRENT edge. Instead, runtime should be
> responsible for handling all the data-plane communications (as proposed in
> [1]).
> Note that the above assumptions are common for scenarios such as whole-DAG or
> sub-graph gang scheduling, but they may be relaxed in later implementation,
> which may allow mixture of SEQUENTIAL and CONCURRENT edges on the same vertex.
>
> Most of the (meaningful) scheduling decisions today in Tez are made based on
> the notion of (or an extended version of) source task completion. This will
> no longer be true in presence of CONCURRENT edge. Instead, events such as
> source vertex configured, or source task running will become more relevant
> when making scheduling decision for two vertices connected via a CONCURRENT
> edge. We therefore introduce a new enum *ConcurrentSchedulingType* to
> describe the “scheduling timing” for the downstream vertex in such scenarios.
> |public enum ConcurrentSchedulingType{
> /** * trigger downstream vertex tasks scheduling by "configured" event of
> upstream vertices */
> SOURCE_VERTEX_CONFIGURED,
> /** * trigger downstream vertex tasks scheduling by "running" event of
> upstream tasks */
> SOURCE_TASK_STARTED
> }|
>
> Note that in this change, we will only use SOURCE_VERTEX_CONFIGURED as the
> scheduling type, which suffice for scenarios of whole-DAG or sub-graph
> gang-scheduling, where we want (all the tasks in) the downstream vertex to be
> scheduled together with (all the tasks) in the upstream vertex. In this case,
> we can leverage the existing onVertexStateUpdated() interface of
> VextexMangerPlugin to collect relevant information to assist the scheduling
> decision, and *there is no additional API change necessary*. However, in more
> subtle case such as the parameter-server example described in Fig. 1, other
> scheduling type would be more relevant, therefore the placeholder for
> *ConcurrentSchedulingType* will be introduced in this change as part of the
> infrastructure work.
>
> Finally, since we assume that all communications between two vertices
> connected via CONCURRENT edge are handled by application runtime, a
> CONCURRENT edge will be assigned a DummyEdgeManager that basically mute all
> DME/VME handling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (TEZ-3998) Allow CONCURRENT edge property in DAG construction and introduce ConcurrentSchedulingType
[
https://issues.apache.org/jira/browse/TEZ-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652766#comment-16652766
]
ASF GitHub Bot commented on TEZ-3998:
-
Github user yingdachen commented on the issue:
https://github.com/apache/tez/pull/33
> @yingdachen since this is part of a larger end-to-end feature (Tez-3997)
would it be better for you to own a feature branch, in which you commit for the
subtasks, and later, when the feature is complete, merge into master?
> It is easier to provide meaningful feedback when seeing how it all comes
together - right now, most of the changes are stubs.
@anicoara thanks for the feedback.
couple of things
1. The design doc in 3997 was meant to provide the big picture and as the
place to collect feedback for the overall design. We believe suitable level of
details about proposed changes have been provided, and it would be great hear
from community on the design.
2. The feature was broken into 4 tasks carefully that range from basic
support and plugin addition(3998), api change in AM component (3999), change in
runtime (4000) and more complete scenario support (4001). We believe such
break-down is suitable for us as new contributors to fit in, beginning with a
simple change discussed here (3998) that is standalone in itself. We are hoping
that our changes will take a gradual path to completion, which would not
necessitate a separate feature branch.
3. In the change introduced here, the SilentEdgeManager is indeed a stub,
that is by design and would not likely to change. However, I would not
categorize the other change (such as the new VertexManagerWithConcurrentInput)
as so, since it is functional as it is and we have added UT coverage to cover
that as well.
> Allow CONCURRENT edge property in DAG construction and introduce
> ConcurrentSchedulingType
> -
>
> Key: TEZ-3998
> URL: https://issues.apache.org/jira/browse/TEZ-3998
> Project: Apache Tez
> Issue Type: Task
>Reporter: Yingda Chen
>Assignee: Yingda Chen
>Priority: Major
>
> This is the first task related to TEZ-3997
>
> |Note: There is no API change in this proposed change. The majority of this
> change will be lifting some existing constraints against CONCURRENT edge
> type, and addition of a VertexMangerPlugin implementation.|
>
> This includes enabling the CONCURRENT SchedulingType as a valid edge
> property, by removing all the sanity check against CONCURRENT during DAG
> construction/execution. A new VertexManagerPlugin (namely
> VertexManagerWithConcurrentInput) will be implemented for vertex with
> incoming concurrent edge(s).
> In addition, we will assume in this change that
> * A vertex *cannot* have both SEQUENTIAL and CONCURRENT incoming edges
> * No shuffle or data movement is handled by Tez framework when two vertices
> are connected through a CONCURRENT edge. Instead, runtime should be
> responsible for handling all the data-plane communications (as proposed in
> [1]).
> Note that the above assumptions are common for scenarios such as whole-DAG or
> sub-graph gang scheduling, but they may be relaxed in later implementation,
> which may allow mixture of SEQUENTIAL and CONCURRENT edges on the same vertex.
>
> Most of the (meaningful) scheduling decisions today in Tez are made based on
> the notion of (or an extended version of) source task completion. This will
> no longer be true in presence of CONCURRENT edge. Instead, events such as
> source vertex configured, or source task running will become more relevant
> when making scheduling decision for two vertices connected via a CONCURRENT
> edge. We therefore introduce a new enum *ConcurrentSchedulingType* to
> describe the “scheduling timing” for the downstream vertex in such scenarios.
> |public enum ConcurrentSchedulingType{
> /** * trigger downstream vertex tasks scheduling by "configured" event of
> upstream vertices */
> SOURCE_VERTEX_CONFIGURED,
> /** * trigger downstream vertex tasks scheduling by "running" event of
> upstream tasks */
> SOURCE_TASK_STARTED
> }|
>
> Note that in this change, we will only use SOURCE_VERTEX_CONFIGURED as the
> scheduling type, which suffice for scenarios of whole-DAG or sub-graph
> gang-scheduling, where we want (all the tasks in) the downstream vertex to be
> scheduled together with (all the tasks) in the upstream vertex. In this case,
> we can leverage the existing onVertexStateUpdated() interface of
> VextexMangerPlugin to collect relevant information to assist the scheduling
> decision, and *there is no additional API change necessary*. However, in more
> subtle case such as the parameter-server example described in Fig. 1,
[jira] [Created] (TEZ-4007) Zookeeper based FrameworkClient
Eric Wohlstadter created TEZ-4007: - Summary: Zookeeper based FrameworkClient Key: TEZ-4007 URL: https://issues.apache.org/jira/browse/TEZ-4007 Project: Apache Tez Issue Type: Sub-task Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter A sub-class of FrameworkClient that interacts with application state managed in Zookeeper rather than Yarn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (TEZ-3991) Unmanaged tez sessions
[ https://issues.apache.org/jira/browse/TEZ-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated TEZ-3991: - Labels: Kubernetes (was: ) > Unmanaged tez sessions > -- > > Key: TEZ-3991 > URL: https://issues.apache.org/jira/browse/TEZ-3991 > Project: Apache Tez > Issue Type: New Feature >Affects Versions: 0.10.0 >Reporter: Prasanth Jayachandran >Assignee: Eric Wohlstadter >Priority: Major > Labels: Kubernetes > > Provide an option for launching tez AM in unmanaged mode. In unmanaged mode, > tez AMs can register itself with Zookeeper which clients (like HiveServer2) > can discover via zk registry client. > HiveServer2 currently manages the lifecycle of tez AMs. The unmanaged mode > will let AM come up on their own (can be via simple java launcher) and be > discoverable for others. > Example use case for this is, HiveServer2 can discover already running AMs > and can attach to it for DAG submission and detach when done executing > queries. AMs can similarly discover LLAP daemons via task scheduler plugin > for submitting tasks. > A mode to cut off interactions with RM will also useful since for LLAP no > on-demand containers are required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-4006) Make FrameworkClient Pluggable
Eric Wohlstadter created TEZ-4006:
-
Summary: Make FrameworkClient Pluggable
Key: TEZ-4006
URL: https://issues.apache.org/jira/browse/TEZ-4006
Project: Apache Tez
Issue Type: Sub-task
Reporter: Eric Wohlstadter
Assignee: Eric Wohlstadter
Currently {{FrameworkClient.createFrameworkClient}} is hardcoded to use either
LocalClient or YarnClient.
Change it to allow specifying a FrameworkClient impl. class from configuration.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (TEZ-3991) Unmanaged tez sessions
[ https://issues.apache.org/jira/browse/TEZ-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated TEZ-3991: -- Target Version/s: 0.10.1 (was: 0.10.0) > Unmanaged tez sessions > -- > > Key: TEZ-3991 > URL: https://issues.apache.org/jira/browse/TEZ-3991 > Project: Apache Tez > Issue Type: New Feature >Affects Versions: 0.10.0 >Reporter: Prasanth Jayachandran >Assignee: Eric Wohlstadter >Priority: Major > > Provide an option for launching tez AM in unmanaged mode. In unmanaged mode, > tez AMs can register itself with Zookeeper which clients (like HiveServer2) > can discover via zk registry client. > HiveServer2 currently manages the lifecycle of tez AMs. The unmanaged mode > will let AM come up on their own (can be via simple java launcher) and be > discoverable for others. > Example use case for this is, HiveServer2 can discover already running AMs > and can attach to it for DAG submission and detach when done executing > queries. AMs can similarly discover LLAP daemons via task scheduler plugin > for submitting tasks. > A mode to cut off interactions with RM will also useful since for LLAP no > on-demand containers are required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (TEZ-3991) Unmanaged tez sessions
[ https://issues.apache.org/jira/browse/TEZ-3991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter reassigned TEZ-3991: - Assignee: Eric Wohlstadter > Unmanaged tez sessions > -- > > Key: TEZ-3991 > URL: https://issues.apache.org/jira/browse/TEZ-3991 > Project: Apache Tez > Issue Type: New Feature >Affects Versions: 0.10.0 >Reporter: Prasanth Jayachandran >Assignee: Eric Wohlstadter >Priority: Major > > Provide an option for launching tez AM in unmanaged mode. In unmanaged mode, > tez AMs can register itself with Zookeeper which clients (like HiveServer2) > can discover via zk registry client. > HiveServer2 currently manages the lifecycle of tez AMs. The unmanaged mode > will let AM come up on their own (can be via simple java launcher) and be > discoverable for others. > Example use case for this is, HiveServer2 can discover already running AMs > and can attach to it for DAG submission and detach when done executing > queries. AMs can similarly discover LLAP daemons via task scheduler plugin > for submitting tasks. > A mode to cut off interactions with RM will also useful since for LLAP no > on-demand containers are required. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEZ-4008) Pluggable AM pool registry
Eric Wohlstadter created TEZ-4008:
-
Summary: Pluggable AM pool registry
Key: TEZ-4008
URL: https://issues.apache.org/jira/browse/TEZ-4008
Project: Apache Tez
Issue Type: Sub-task
Reporter: Eric Wohlstadter
Assignee: Eric Wohlstadter
Interface and reflective plugin configuration for a pool of AMs identified by a
namespace.
The registry should allow each {{DAGClientServer}} to register/unregister
themselves from a pool.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (TEZ-4009) Zookeeper based AM Registry
Eric Wohlstadter created TEZ-4009: - Summary: Zookeeper based AM Registry Key: TEZ-4009 URL: https://issues.apache.org/jira/browse/TEZ-4009 Project: Apache Tez Issue Type: Sub-task Reporter: Eric Wohlstadter Assignee: Eric Wohlstadter Zookeeper based implementation of the interface defined by TEZ-4008. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-2222) Investigate moving to log4j2 for logging
[ https://issues.apache.org/jira/browse/TEZ-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652896#comment-16652896 ] Yingda Chen commented on TEZ-: -- I am wondering if this is still being considered? log4j2 provides quite some improvement on both performance and memory usage reduction. Both are critical for running large-scale job on Tez [https://logging.apache.org/log4j/2.x/performance.html] [https://logging.apache.org/log4j/2.x/manual/garbagefree.html] > Investigate moving to log4j2 for logging > > > Key: TEZ- > URL: https://issues.apache.org/jira/browse/TEZ- > Project: Apache Tez > Issue Type: Improvement >Reporter: Siddharth Seth >Priority: Blocker > > Via slf4j. > Some bits to keep in mind > - We have explicit code which rotates logs using direct log4j12 APIs. This > should keep working. I believe the log4j2 APIs are different here > - API compatibility between log4j12 / log4j2 can be problematic - if both end > up on the classpath (I believe the APIs are different) > - Hadoop dist includes a slf4j-log4j12 binding. Changing the default can > result in sl4j-log4j12 and slf4j-log4j2 to co-exist by default - which could > be problematic. Needs investigation. > End of the day, we will likely need an option to use either of the two. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3998) Allow CONCURRENT edge property in DAG construction and introduce ConcurrentSchedulingType
[
https://issues.apache.org/jira/browse/TEZ-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651706#comment-16651706
]
Kuhu Shukla commented on TEZ-3998:
--
I plan to review this in the next few days. Would be nice to get some comments
from [~gopalv] , [~jlowe] and [~jeagles] among others.
> Allow CONCURRENT edge property in DAG construction and introduce
> ConcurrentSchedulingType
> -
>
> Key: TEZ-3998
> URL: https://issues.apache.org/jira/browse/TEZ-3998
> Project: Apache Tez
> Issue Type: Task
>Reporter: Yingda Chen
>Assignee: Yingda Chen
>Priority: Major
>
> This is the first task related to TEZ-3997
>
> |Note: There is no API change in this proposed change. The majority of this
> change will be lifting some existing constraints against CONCURRENT edge
> type, and addition of a VertexMangerPlugin implementation.|
>
> This includes enabling the CONCURRENT SchedulingType as a valid edge
> property, by removing all the sanity check against CONCURRENT during DAG
> construction/execution. A new VertexManagerPlugin (namely
> VertexManagerWithConcurrentInput) will be implemented for vertex with
> incoming concurrent edge(s).
> In addition, we will assume in this change that
> * A vertex *cannot* have both SEQUENTIAL and CONCURRENT incoming edges
> * No shuffle or data movement is handled by Tez framework when two vertices
> are connected through a CONCURRENT edge. Instead, runtime should be
> responsible for handling all the data-plane communications (as proposed in
> [1]).
> Note that the above assumptions are common for scenarios such as whole-DAG or
> sub-graph gang scheduling, but they may be relaxed in later implementation,
> which may allow mixture of SEQUENTIAL and CONCURRENT edges on the same vertex.
>
> Most of the (meaningful) scheduling decisions today in Tez are made based on
> the notion of (or an extended version of) source task completion. This will
> no longer be true in presence of CONCURRENT edge. Instead, events such as
> source vertex configured, or source task running will become more relevant
> when making scheduling decision for two vertices connected via a CONCURRENT
> edge. We therefore introduce a new enum *ConcurrentSchedulingType* to
> describe the “scheduling timing” for the downstream vertex in such scenarios.
> |public enum ConcurrentSchedulingType{
> /** * trigger downstream vertex tasks scheduling by "configured" event of
> upstream vertices */
> SOURCE_VERTEX_CONFIGURED,
> /** * trigger downstream vertex tasks scheduling by "running" event of
> upstream tasks */
> SOURCE_TASK_STARTED
> }|
>
> Note that in this change, we will only use SOURCE_VERTEX_CONFIGURED as the
> scheduling type, which suffice for scenarios of whole-DAG or sub-graph
> gang-scheduling, where we want (all the tasks in) the downstream vertex to be
> scheduled together with (all the tasks) in the upstream vertex. In this case,
> we can leverage the existing onVertexStateUpdated() interface of
> VextexMangerPlugin to collect relevant information to assist the scheduling
> decision, and *there is no additional API change necessary*. However, in more
> subtle case such as the parameter-server example described in Fig. 1, other
> scheduling type would be more relevant, therefore the placeholder for
> *ConcurrentSchedulingType* will be introduced in this change as part of the
> infrastructure work.
>
> Finally, since we assume that all communications between two vertices
> connected via CONCURRENT edge are handled by application runtime, a
> CONCURRENT edge will be assigned a DummyEdgeManager that basically mute all
> DME/VME handling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (TEZ-3075) Revamp bad node handling
[ https://issues.apache.org/jira/browse/TEZ-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yingda Chen reassigned TEZ-3075: Assignee: Ying Han > Revamp bad node handling > > > Key: TEZ-3075 > URL: https://issues.apache.org/jira/browse/TEZ-3075 > Project: Apache Tez > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Ying Han >Priority: Major > > The current logic around that is derived from MR and does not work in all > cases. > Things to consider > 1) Have a notion of probation where machines are put out of service for a > period of time (say 5m, 15m and 30m) before being given up for good. This > allows more graceful handling of temporary glitches. > 2) Different handling for YARN marking a node as bad vs internal heuritics > 3) Bad nodes should not immediately trigger re-execution of completed work. > That should be based on presence of downstream consumers (ie existing demand > for that output) and a reasonable indication by other consumers from that > node that it cannot serve results (eg. multiple reports of read errors with > that node as a source). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-3075) Revamp bad node handling
[ https://issues.apache.org/jira/browse/TEZ-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651765#comment-16651765 ] Yingda Chen commented on TEZ-3075: -- [~Chyler] and I will be looking at this together with TEZ-3822 > Revamp bad node handling > > > Key: TEZ-3075 > URL: https://issues.apache.org/jira/browse/TEZ-3075 > Project: Apache Tez > Issue Type: Improvement >Reporter: Bikas Saha >Assignee: Ying Han >Priority: Major > > The current logic around that is derived from MR and does not work in all > cases. > Things to consider > 1) Have a notion of probation where machines are put out of service for a > period of time (say 5m, 15m and 30m) before being given up for good. This > allows more graceful handling of temporary glitches. > 2) Different handling for YARN marking a node as bad vs internal heuritics > 3) Bad nodes should not immediately trigger re-execution of completed work. > That should be based on presence of downstream consumers (ie existing demand > for that output) and a reasonable indication by other consumers from that > node that it cannot serve results (eg. multiple reports of read errors with > that node as a source). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
