[
https://issues.apache.org/jira/browse/TEZ-1233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313773#comment-14313773
]
Siddharth Seth commented on TEZ-1233:
-------------------------------------
bq. For TEZ-1788, it is probably not sufficient to specify vertex specific
disable in code. Sometimes it may be possible to know that a vertex is
non-deterministic (e.g.) and so speculation should not be done. The disabling
would likely be necessary in an operational situation where speculation is
being unnecessarily triggered. In general, configuration is used to allow the
same job to run differently based on different conditions. So enabling
something like this that requires changing code to apply configuration is kind
of inconsistent with the concept of configuration. While its probably fine to
allow programmatic access, it seems incomplete without allowing configuration
to be changed via configuration files.
There are cases where just setting configuration programmatically is
sufficient. Number of attempts for tasks in a vertex for example. Some of this
could be pushed into VertexManagers - but that seems a little overkill at this
point.
Trying to change this via a configuration property like tez-site may be fairly
difficult as well - unless the user knows all their vertex names etc up front.
We could support something like vertex name suffixes, but I think that can be
hard to use. It may be better to leave this management to individual platforms
like Hive/Pig - which could define their own policies for speculation, as an
example.
bq. From the above comments, it wasn't clear how this relates to TEZ-2003. Any
light on that aspect would be helpful to understand if the code being built
here is useful in that regard.
TEZ-2003 talks about allowing scheduling / launch / communication bits to be
configured at a per vertex level. It'll be up to the framework using Tez to set
these up properly, and define policies on how to override these via
configuration.
> Allow configuration of framework parameters per vertex
> ------------------------------------------------------
>
> Key: TEZ-1233
> URL: https://issues.apache.org/jira/browse/TEZ-1233
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Siddharth Seth
> Assignee: Jeff Zhang
> Attachments: TEZ-1233-1.patch, TEZ-1233-2.patch, TEZ-1233-3.patch
>
>
> Currently, configuration properties specific in AMConfiguration are used to
> configure tasks - e.g. memory configs, process tree, etc. These should be
> configurable at a Vertex level.
> Also, tasks end up reading these configs from the dist cache each time -
> would be simpler to send over the wire as part of the TaskSpec.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
