[jira] [Comment Edited] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672561#comment-16672561
]
Rahul Anand edited comment on YARN-6900 at 11/2/18 4:48 AM:
Thanks [~subru] or [~elgoiri] for the comment.
So right now we have two properties in the yarn-site for setting up policy
manager and its related params i.e. *yarn.federation.policy-manager-params* and
*yarn.federation.policy-manager*. According to the code, we should set this
param property with bytes but obviously that is not what user expects. So for
discusstion, first thing is the Format of this input.
Here is how things look is the zknode if I tried to put policy configurations
in statestore for the default queue * using PriorityBroadcastPolicyManager
*get /federationstore/policies/**
{code:java}
*Xorg.apache.hadoop.yarn.server.federation.policies.manager.PriorityBroadcastPolicyManager�{"routerPolicyWeights":{"entry":[{"key":{"id":"cluster-bb"},"value":"1.0"},{"key":{"id":"cluster-ra"},"value":"3.0"}]},"amrmPolicyWeights":null,"headroomAlpha":"0.0"}
{code}
So overall we have these many things to put in the params field
1. routerpolicyweights (weight of particular subcluter to be used for router
policy)
2. amrmpolicyweights (weight of particular subcluter to be used for amrm
policy)
3. headroomAlpha
was (Author: rahulanand90):
Thanks [~subru] or [~elgoiri] for the comment.
So right now we have two properties in the yarn-site for setting up policy
manager and its related params i.e. *yarn.federation.policy-manager-params* and
*yarn.federation.policy-manager*. According to the code, we should set this
param property with bytes but obviously that is not what user expects. So for
discusstion, first thing is the Format of this input.
Here is how things look is the zknode if I tried to put policy configurations
in statestore for the default queue * using PriorityBroadcastPolicyManager
*get /federationstore/policies/**
{code:java}
*Xorg.apache.hadoop.yarn.server.federation.policies.manager.PriorityBroadcastPolicyManager�{"routerPolicyWeights":{"entry":[{"key":{"id":"cluster-bb"},"value":"1.0"},{"key":{"id":"cluster-ra"},"value":"3.0"}]},"amrmPolicyWeights":null,"headroomAlpha":"0.0"}
{code}
So overall we have these many things to put in the params field
1. amrmpolicyweights (weight of particular subcluter to be used for router
policy)
2. routerpolicyweights (weight of particular subcluter to be used for amrm
policy)
3. headroomAlpha
> ZooKeeper based implementation of the FederationStateStore
> --
>
> Key: YARN-6900
> URL: https://issues.apache.org/jira/browse/YARN-6900
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Íñigo Goiri
>Priority: Major
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6900-002.patch, YARN-6900-003.patch,
> YARN-6900-004.patch, YARN-6900-005.patch, YARN-6900-006.patch,
> YARN-6900-007.patch, YARN-6900-008.patch, YARN-6900-009.patch,
> YARN-6900-010.patch, YARN-6900-011.patch, YARN-6900-YARN-2915-000.patch,
> YARN-6900-YARN-2915-001.patch
>
>
> YARN-5408 defines the unified {{FederationStateStore}} API. Currently we only
> support SQL based stores, this JIRA tracks adding a ZooKeeper based
> implementation for simplifying deployment as it's already popularly used for
> {{RMStateStore}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637795#comment-16637795
]
Rahul Anand edited comment on YARN-6900 at 10/4/18 12:47 PM:
-
[~elgoiri] [~subru]
I would like to know how one can configure the znode so as to specify the queue
policies. I mean how can I specify the router, amrmpolicy, associated weights
etc in the znode and in which format. I am unable to find this anywhere.
Alternatively, if not that then can you help in setting me params in
yarn.federation.policy-manager-params so that I can set policies weights.
was (Author: rahulanand90):
[~elgoiri] [~subru]
I would like to know how one can configure the znode so as to specify the queue
policies. I mean how can I specify the router, amrmpolicy, associated weights
etc in the znode and in which format. I am unable to find this anywhere.
> ZooKeeper based implementation of the FederationStateStore
> --
>
> Key: YARN-6900
> URL: https://issues.apache.org/jira/browse/YARN-6900
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Íñigo Goiri
>Priority: Major
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6900-002.patch, YARN-6900-003.patch,
> YARN-6900-004.patch, YARN-6900-005.patch, YARN-6900-006.patch,
> YARN-6900-007.patch, YARN-6900-008.patch, YARN-6900-009.patch,
> YARN-6900-010.patch, YARN-6900-011.patch, YARN-6900-YARN-2915-000.patch,
> YARN-6900-YARN-2915-001.patch
>
>
> YARN-5408 defines the unified {{FederationStateStore}} API. Currently we only
> support SQL based stores, this JIRA tracks adding a ZooKeeper based
> implementation for simplifying deployment as it's already popularly used for
> {{RMStateStore}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16128122#comment-16128122
]
Subru Krishnan edited comment on YARN-6900 at 8/16/17 12:14 AM:
Thanks [~elgoiri] for updating the patch. The latest patch is pretty close,
couple of minor comments:
* Can you rebase (_get/put/createRootDirRecursively_ from {{ZKCuratorManager}})
now that HADOOP-14773 is in.
* Instead of adding in *yarn-default.xml*, I prefer to exclude in
{{TestYarnConfigurationFields}} as ZK store is not the default store anyways.
was (Author: subru):
Thanks [~elgoiri] for updating the patch. The latest patch is pretty close,
couple of minor comments:
* Can you rebase now that HADOOP-14773 is in.
* Instead of adding in *yarn-default.xml*, I prefer to exclude in
{{TestYarnConfigurationFields}} as ZK store is not the default store anyways.
> ZooKeeper based implementation of the FederationStateStore
> --
>
> Key: YARN-6900
> URL: https://issues.apache.org/jira/browse/YARN-6900
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Íñigo Goiri
> Attachments: YARN-6900-002.patch, YARN-6900-003.patch,
> YARN-6900-004.patch, YARN-6900-005.patch, YARN-6900-006.patch,
> YARN-6900-007.patch, YARN-6900-008.patch, YARN-6900-009.patch,
> YARN-6900-010.patch, YARN-6900-YARN-2915-000.patch,
> YARN-6900-YARN-2915-001.patch
>
>
> YARN-5408 defines the unified {{FederationStateStore}} API. Currently we only
> support SQL based stores, this JIRA tracks adding a ZooKeeper based
> implementation for simplifying deployment as it's already popularly used for
> {{RMStateStore}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111986#comment-16111986
]
Subru Krishnan edited comment on YARN-6900 at 8/3/17 7:46 PM:
--
Thanks [~elgoiri] for working on this. I looked at the patch, please find my
comments below:
* Looks like we have common ZooKeeper interaction functionalities like
*createAndStartCurator, getZKAuths, getZKAcls, create, delete, exists,
getNodePath*,... etc which we can share with {{ZKRMStateStore}}. I suggest
moving these to a common {{YarnZKUtils}} in _yarn-server-common_ so that it be
reused. This should also be useful for YARN-6840.
* ZK_RM_STATE_STORE_PARENT_PATH --> ZK_FEDERATION_STATE_STORE_PARENT_PATH as
this is federation state across clusters.
* Can we materialize *appsZNode, membershipZNode, policiesZNode* as we use them
a lot and they don't change post initialization?
* Nit: can we have a hierarchy layout as part of the class Javadocs like in
{{ZKRMStateStore}} for clarity?
* We should make {{ZookeeperFederationStateStore}} the default
{{FederationStateStore}} implementation.
* Update the documentation
{{hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md}}
based on the above change.
* Lastly, can you take a look at the Yetus checkstyle/whitespace warnings?
was (Author: subru):
Thanks [~elgoiri] for working on this. I looked at the patch, please find my
comments below:
* Looks like we have common ZooKeeper interaction functionalities like
*createAndStartCurator, getZKAuths, getZKAcls, create, delete, exists,
getNodePath*,... etc which we can share with {{ZKRMStateStore}}. I suggest
moving these to a common {{YarnZKUtils}} in _yarn-server-common_ so that it be
reused. This should also be useful for YARN-6840.
* ZK_RM_STATE_STORE_PARENT_PATH --> ZK_FEDERATION_STATE_STORE_PARENT_PATH as
this is federation state across clusters.
* Can we materialize *appsZNode, membershipZNode, policiesZNode* as we use them
a lot and they don't change post initialization?
* Nit: can we have a hierarchy layout as part of the class Javadocs like in
{{ZKRMStateStore}} for clarity?
* We should {{ZookeeperFederationStateStore}} the default
{{FederationStateStore}} implementation.
* Update the documentation
{{hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md}}
based on the above change.
* Lastly, can you take a look at the Yetus checkstyle/whitespace warnings?
> ZooKeeper based implementation of the FederationStateStore
> --
>
> Key: YARN-6900
> URL: https://issues.apache.org/jira/browse/YARN-6900
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Inigo Goiri
> Attachments: YARN-6900-002.patch, YARN-6900-YARN-2915-000.patch,
> YARN-6900-YARN-2915-001.patch
>
>
> YARN-5408 defines the unified {{FederationStateStore}} API. Currently we only
> support SQL based stores, this JIRA tracks adding a ZooKeeper based
> implementation for simplifying deployment as it's already popularly used for
> {{RMStateStore}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
