[
https://issues.apache.org/jira/browse/YARN-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15433409#comment-15433409
]
Carlo Curino edited comment on YARN-5323 at 8/23/16 7:05 PM:
-------------------------------------------------------------
[~wangda] thanks for reviewing.
Answering your questions:
# The AMRMProxyFederationPolicy is used in the AMRMProxyService to determine
where to send the ResourceRequests, while the RouterFederationPolicy is used by
the router to determine where to send the AM. This patch is a bit of a "leaf"
patch, other patches will use this code, so it make sense you don't get much
context from this patch alone.
# We could remove it, but the key intuition we have is that the two
AMRMProxyPolicy and RouterPolicy, while completely independent (run on
different nodes and do different things), need to work in collaboration with
each other to achieve a consistent system behavior. We are thus trying to use
the code structure (e.g., FederationPolicy) as a way to tie them together. This
should help administrators to avoid some of the obvious mistakes (e.g.,
combining a load-balancing RouterPolicy with a fault-tolerance focused
AMRMProxyPolicy)---if you want you can, but have to implement a class and
configure the system to use it, which should make you think harder about what
you are doing. Also we are using this writer + getter api as a way to handle
the entire lifecycle of ser/deser of the policy configuration through the
store. By construction we want the configuration to be rather opaque
(expandibility), but we need to keep the lifecycle management sane, this is our
attempt.
# Works for me, though I don't think is the common way for other "*Context"
objects around YARN.
# The Router routes more than the APPs, all client-to-rm protocols go through
there... Most of the decisions of where to route are not policy based, as they
are deterministic (e.g., if you send a kill should go where the app was
running). I think the policy name should say Router to indicate that is used by
the router. Also in longer term I can see more policy behaviors to be added to
the router, as we become more sophisticated and operations that are fully
manual today (capacity allocation to subclusters) get automated.
I hope this helps to clarify your concerns.
was (Author: curino):
[~wangda] thanks for reviewing.
Answering your questions:
# The AMRMProxyFederationPolicy is used in the AMRMProxyService to determine
where to send the ResourceRequests, while the RouterFederationPolicy is used by
the router to determine where to send the AM. This patch is a bit of a "leaf"
patch, other patches will use this code, so it make sense you don't get much
context from this patch alone.
# We could remove it, but the key intuition we have is that the two
AMRMProxyPolicy and RouterPolicy, while completely independent (run on
different nodes and do different things), need to work in collaboration with
each other to achieve a consistent system behavior. We are thus trying to use
the code structure (e.g., FederationPolicy) as a way to tie them together. This
should help administrators to avoid some of the obvious mistakes (e.g.,
combining a load-balancing RouterPolicy with a fault-tolerance focused
AMRMProxyPolicy)---if you want you can, but have to implement a class and
configure the system to use it, which should make you think harder about what
you are doing. Also we are using this writer + getter api as a way to handle
the entire lifecycle of ser/deser of the policy configuration through the
store. By construction we want the configuration to be rather opaque
(expandibility), but we need to keep the lifecycle management sane, this is our
attempt.
# Works for me, though I don't think is the common way for other "*Context"
objects around YARN.
# The Router routes more than the APPs, all client-to-rm protocols go through
there... Most of the decisions of where to route are not policy based, as they
are deterministic (e.g., if you send a kill should go where the app was
running). I think the policy name should say Router to indicate that is used by
the router. Also in longer term I can see more policy behaviors to be added to
the router, as we become more sophisticated and operations that are fully
manual today (capacity allocation to subclusters) get automated.
I hope this help clarify your concerns.
> Policies APIs (for Router and AMRMProxy policies)
> -------------------------------------------------
>
> Key: YARN-5323
> URL: https://issues.apache.org/jira/browse/YARN-5323
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
> Affects Versions: YARN-2915
> Reporter: Carlo Curino
> Assignee: Carlo Curino
> Attachments: YARN-5323-YARN-2915.05.patch, YARN-5323.01.patch,
> YARN-5323.02.patch, YARN-5323.03.patch, YARN-5323.04.patch
>
>
> This JIRA tracks APIs for the policies that will guide the Router and
> AMRMProxy decisions on where to fwd the jobs submission/query requests as
> well as ResourceRequests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
