[jira] [Commented] (YARN-4100) Add Documentation for Distributed and Delegated-Centralized Node Labels feature
[ https://issues.apache.org/jira/browse/YARN-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065962#comment-15065962 ] Dian Fu commented on YARN-4100: --- Hi [~Naganarasimha], Thanks a lot for the quick update. LGTM. +1. > Add Documentation for Distributed and Delegated-Centralized Node Labels > feature > --- > > Key: YARN-4100 > URL: https://issues.apache.org/jira/browse/YARN-4100 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: NodeLabel.html, YARN-4100.v1.001.patch, > YARN-4100.v1.002.patch, YARN-4100.v1.003.patch > > > Add Documentation for Distributed Node Labels feature -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4100) Add Documentation for Distributed and Delegated-Centralized Node Labels feature
[
https://issues.apache.org/jira/browse/YARN-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15065637#comment-15065637
]
Dian Fu commented on YARN-4100:
---
Hi [~Naganarasimha],
Very sorry for late response. It LGTM overall. Just a few small comments as
follows:
{quote}
When "yarn.nodemanager.node-labels.provider" is configured with "config",
"Script"
{quote}
{{S}} should be lower case for {{Script}}.
{quote}
When "yarn.nodemanager.node-labels.provider" is configured with "config" then
{quote}
A comma can be added before {{then}}
{quote}
which queries the Node labels.
{quote}
{{Node}} can be {{node}}. Actually {{node label}}, {{Node Label}}, {{Node
label}}, {{node Label}} appears a lot of times in the doc, I think they should
be consistent.
{quote}
In case of multiple lines have this pattern, then last one will be considered
{quote}
A period should be added at the end.
{quote}
Configured class needs to extend
{quote}
Two white space between {{Configured}} and {{class}}
> Add Documentation for Distributed and Delegated-Centralized Node Labels
> feature
> ---
>
> Key: YARN-4100
> URL: https://issues.apache.org/jira/browse/YARN-4100
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: NodeLabel.html, YARN-4100.v1.001.patch,
> YARN-4100.v1.002.patch
>
>
> Add Documentation for Distributed Node Labels feature
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4100) Add Documentation for Distributed and Delegated-Centralized Node Labels feature
[
https://issues.apache.org/jira/browse/YARN-4100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15052411#comment-15052411
]
Dian Fu commented on YARN-4100:
---
Hi [~Naganarasimha],
Thanks a lot for patch. LGTM. Just a few minor comments:
* {{Script}} should be {{script}}.
* {{NM will send its loaded labels every x intervals configured along with
heartbeat from NM to RM.}} can be changed to {{NM will send its loaded labels
every x intervals configured along with heartbeat to RM.}}.
* {quote}
By default 2 implementations are supported, Script and Configuration based
{quote}
What about {{By default 2 implementations are supported: Script based and
Configuration based}}?
* {quote}
In case of Script,..In case of config,
{quote}
The capital of {{script}} and {{config}} should be consistent.
* {{This would helpfull}} can be changed to {{This would be helpfull}}
* {quote}"delegated-centralized" needs to be set, to fetch the labels from a
interface in RM.}} {quote}
The comma can be removed.
* {quote} When "yarn.node-labels.configuration-type" parameter in RM is
configured as "distributed", {quote}
What about changed to {{When "yarn.node-labels.configuration-type" is
configured with "distributed"}}
* {quote}If "config" is specified then "ConfigurationNodeLabelsProvider" and
"script" then "ScriptNodeLabelsProvider" will be used.{quote}
What about changed to {{If "config" is configured, then
"ConfigurationNodeLabelsProvider" will be used and if "script" is configured,
then "ScriptNodeLabelsProvider" will be used.}}
* {quote}admin might have removed the cluster label which was provided NM{quote}
Missed one word {{by}} between {{provided}} and {{NM}}.
* {quote} This configuration is to define the interval period.If -1 is
configured{quote}
Missed one white space before {{If}}.
* {{Whenthen}}, {{If...then}}
I think there should be one comma before then.
* {{will be considered for Node Labels.}}
Seems that it should be {{will be considered as Node Labels.}}
* {{ In case of multiple lines having the pattern, last one will be
considered.}}
What about {{In case that multiple lines have this pattern, the last line will
be considered.}}
> Add Documentation for Distributed and Delegated-Centralized Node Labels
> feature
> ---
>
> Key: YARN-4100
> URL: https://issues.apache.org/jira/browse/YARN-4100
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
> Attachments: NodeLabel.html, YARN-4100.v1.001.patch
>
>
> Add Documentation for Distributed Node Labels feature
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4240) Add documentation for delegated-centralized node labels feature
[ https://issues.apache.org/jira/browse/YARN-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046513#comment-15046513 ] Dian Fu commented on YARN-4240: --- Hi [~Naganarasimha], Cool! Yes, of course. Feel free to do it and thanks a lot for the efforts. > Add documentation for delegated-centralized node labels feature > --- > > Key: YARN-4240 > URL: https://issues.apache.org/jira/browse/YARN-4240 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Dian Fu >Assignee: Dian Fu > > As a follow up of YARN-3964, we should add documentation for > delegated-centralized node labels feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4240) Add documentation for delegated-centralized node labels feature
Dian Fu created YARN-4240: - Summary: Add documentation for delegated-centralized node labels feature Key: YARN-4240 URL: https://issues.apache.org/jira/browse/YARN-4240 Project: Hadoop YARN Issue Type: Sub-task Reporter: Dian Fu Assignee: Dian Fu As a follow up of YARN-3964, we should add documentation for delegated-centralized node labels feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.016.patch > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch, > YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.010.patch, > YARN-3964.011.patch, YARN-3964.012.patch, YARN-3964.013.patch, > YARN-3964.014.patch, YARN-3964.015.patch, YARN-3964.016.patch, > YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14948280#comment-14948280 ] Dian Fu commented on YARN-3964: --- Hi [~devaraj.k], Thanks a lot for your review. Updated the patch accordingly. Have also created ticket YARN-4240 for the documentation. > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch, > YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.010.patch, > YARN-3964.011.patch, YARN-3964.012.patch, YARN-3964.013.patch, > YARN-3964.014.patch, YARN-3964.015.patch, YARN-3964.016.patch, > YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4240) Add documentation for delegated-centralized node labels feature
[ https://issues.apache.org/jira/browse/YARN-4240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949881#comment-14949881 ] Dian Fu commented on YARN-4240: --- Hi [~Naganarasimha], Yes, of course. I will update documentation for "Delegated-Centralized" on top of YARN-4100 after it is committed. > Add documentation for delegated-centralized node labels feature > --- > > Key: YARN-4240 > URL: https://issues.apache.org/jira/browse/YARN-4240 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Reporter: Dian Fu >Assignee: Dian Fu > > As a follow up of YARN-3964, we should add documentation for > delegated-centralized node labels feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.015.patch Attaching a new patch to fix the findbugs warning. > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch, > YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.010.patch, > YARN-3964.011.patch, YARN-3964.012.patch, YARN-3964.013.patch, > YARN-3964.014.patch, YARN-3964.015.patch, YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.014.patch > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch, > YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.010.patch, > YARN-3964.011.patch, YARN-3964.012.patch, YARN-3964.013.patch, > YARN-3964.014.patch, YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[
https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939668#comment-14939668
]
Dian Fu commented on YARN-3964:
---
Hi [~Naganarasimha],
Thanks a lot for your review. Updated the patch accordingly.
{quote}
nodesToUpdateLabels == null is req ?
{quote}
Yes. If {{nodesToUpdateLabels}} isn't null, then {{nodesToUpdateLabels}}
contains all the cluster nodes (also including the newly registered nodes). In
this case, we don't need to add the newly registered nodes to
{{nodesToUpdateLabels}} any more.
{quote}
Hope you can share your test code for the RMNodeLabelMappingsUpdater with which
it i can test, hope you also have verified it .
{quote}
I have an implementation of {{RMNodeLabelMappingsUpdater}} which can fetch
nodes trust status from Mt. Wilson server and translate this trust status
information into node trust/untrust labels. As the Mt. Wilson is a commercial
software and the environment setup of Mt. Wilson server is a little
complicated, it would be difficult for you to use my test code directly. I
think you can write a simple implementation of {{RMNodeLabelMappingsUpdater}}
which returns hard-coded labels for the test.
> Support NodeLabelsProvider at Resource Manager side
> ---
>
> Key: YARN-3964
> URL: https://issues.apache.org/jira/browse/YARN-3964
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Dian Fu
>Assignee: Dian Fu
> Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch,
> YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch,
> YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch,
> YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.010.patch,
> YARN-3964.011.patch, YARN-3964.012.patch, YARN-3964.013.patch,
> YARN-3964.014.patch, YARN-3964.1.patch
>
>
> Currently, CLI/REST API is provided in Resource Manager to allow users to
> specify labels for nodes. For labels which may change over time, users will
> have to start a cron job to update the labels. This has the following
> limitations:
> - The cron job needs to be run in the YARN admin user.
> - This makes it a little complicate to maintain as users will have to make
> sure this service/daemon is alive.
> Adding a Node Labels Provider in Resource Manager will provide user more
> flexibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.013.patch > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch, > YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.010.patch, > YARN-3964.011.patch, YARN-3964.012.patch, YARN-3964.013.patch, > YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[
https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14936344#comment-14936344
]
Dian Fu commented on YARN-3964:
---
Hi [~leftnoteasy], [~Naganarasimha] and [~sunilg],
Thanks for your inputs. Updated the patch to eliminate the impact of the
response time of {{nodeLabelsMappingProvider.getNodeLabels}}.
> Support NodeLabelsProvider at Resource Manager side
> ---
>
> Key: YARN-3964
> URL: https://issues.apache.org/jira/browse/YARN-3964
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Dian Fu
>Assignee: Dian Fu
> Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch,
> YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch,
> YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch,
> YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.010.patch,
> YARN-3964.011.patch, YARN-3964.012.patch, YARN-3964.013.patch,
> YARN-3964.1.patch
>
>
> Currently, CLI/REST API is provided in Resource Manager to allow users to
> specify labels for nodes. For labels which may change over time, users will
> have to start a cron job to update the labels. This has the following
> limitations:
> - The cron job needs to be run in the YARN admin user.
> - This makes it a little complicate to maintain as users will have to make
> sure this service/daemon is alive.
> Adding a Node Labels Provider in Resource Manager will provide user more
> flexibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[
https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14933584#comment-14933584
]
Dian Fu commented on YARN-3964:
---
Hi [~Naganarasimha] and [~sunilg],
Thanks for your comments.
{quote}
I feel as nodes are more in cluster, and response time from
{{nodeLabelsMappingProvider.getNodeLabels}} may also be more.
{quote}
{{updateNodeLabelsInternal}} is called in two places: one place is in thread
{{RMDelegatedNodeLabelsUpdaterTimerTask}} and the other place is on register.
* When called in thread {{RMDelegatedNodeLabelsUpdaterTimerTask}}, the impact
is minor as it is working in a separate thread and so the response time of
{{RMDelegatedNodeLabelsUpdaterTimerTask}} won't impact other threads. Further
more, only one call of {{nodeLabelsMappingProvider.getNodeLabels}} can fetch
all the node labels and so the response time should not be too long.
* When called on register, the response time of
{{nodeLabelsMappingProvider.getNodeLabels}} does have impact as it may lead to
longer register time. That's the reason why patch 006 doesn't update node
labels synchronously on register. Thoughts?
{quote}
This piece of code i think should be done by the caller handle(SchedulerEvent
event) rather than updateLabelsOnNode as its synchronized method.{quote}
Make sense.
> Support NodeLabelsProvider at Resource Manager side
> ---
>
> Key: YARN-3964
> URL: https://issues.apache.org/jira/browse/YARN-3964
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Dian Fu
>Assignee: Dian Fu
> Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch,
> YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch,
> YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch,
> YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.010.patch,
> YARN-3964.011.patch, YARN-3964.012.patch, YARN-3964.1.patch
>
>
> Currently, CLI/REST API is provided in Resource Manager to allow users to
> specify labels for nodes. For labels which may change over time, users will
> have to start a cron job to update the labels. This has the following
> limitations:
> - The cron job needs to be run in the YARN admin user.
> - This makes it a little complicate to maintain as users will have to make
> sure this service/daemon is alive.
> Adding a Node Labels Provider in Resource Manager will provide user more
> flexibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[
https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14933342#comment-14933342
]
Dian Fu commented on YARN-3964:
---
Hi [~leftnoteasy],
Thanks a lot for your review. Updated the patch according to your comments.
Regarding to the issues YARN-4106 and YARN-4176, this patch doesn't have the
same issue as currently when updating node labels,
{{RMDelegatedNodeLabelsUpdater}} doesn't check if the labels for a node has
changed and just call {{RMNodeLabelsManager#replaceLabelsOnNode}}. The reason
is that for {{delegated-centralized}} type, we only update the nodes -> labels
map stored in memory and this should be very fast. Thoughts?
> Support NodeLabelsProvider at Resource Manager side
> ---
>
> Key: YARN-3964
> URL: https://issues.apache.org/jira/browse/YARN-3964
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Dian Fu
>Assignee: Dian Fu
> Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch,
> YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch,
> YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch,
> YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.010.patch,
> YARN-3964.011.patch, YARN-3964.012.patch, YARN-3964.1.patch
>
>
> Currently, CLI/REST API is provided in Resource Manager to allow users to
> specify labels for nodes. For labels which may change over time, users will
> have to start a cron job to update the labels. This has the following
> limitations:
> - The cron job needs to be run in the YARN admin user.
> - This makes it a little complicate to maintain as users will have to make
> sure this service/daemon is alive.
> Adding a Node Labels Provider in Resource Manager will provide user more
> flexibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[
https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14934534#comment-14934534
]
Dian Fu commented on YARN-3964:
---
Thanks a lot for the comments [~leftnoteasy]. What's your thought about the
impact of the response time of {{nodeLabelsMappingProvider.getNodeLabels}}?
> Support NodeLabelsProvider at Resource Manager side
> ---
>
> Key: YARN-3964
> URL: https://issues.apache.org/jira/browse/YARN-3964
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Dian Fu
>Assignee: Dian Fu
> Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch,
> YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch,
> YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch,
> YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.010.patch,
> YARN-3964.011.patch, YARN-3964.012.patch, YARN-3964.1.patch
>
>
> Currently, CLI/REST API is provided in Resource Manager to allow users to
> specify labels for nodes. For labels which may change over time, users will
> have to start a cron job to update the labels. This has the following
> limitations:
> - The cron job needs to be run in the YARN admin user.
> - This makes it a little complicate to maintain as users will have to make
> sure this service/daemon is alive.
> Adding a Node Labels Provider in Resource Manager will provide user more
> flexibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.012.patch > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch, > YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.010.patch, > YARN-3964.011.patch, YARN-3964.012.patch, YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.010.patch > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch, > YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.010.patch, > YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.011.patch > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch, > YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.010.patch, > YARN-3964.011.patch, YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909535#comment-14909535 ] Dian Fu commented on YARN-3964: --- Updated the patch to fix the test failures and change the default interval time to 30 minutes. > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch, > YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.010.patch, > YARN-3964.011.patch, YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.009.patch > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch, > YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[
https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909028#comment-14909028
]
Dian Fu commented on YARN-3964:
---
Hi [~Naganarasimha], [~sunilg],
Thanks a lot for your review. Updated the patch according to your comments with
the following exceptions:
{quote}
If updateInterval is disabled then only on register of nodes the labels are
fetched, is that fine or is it better to fetch once and not run timer ?
{quote}
You mean not only fetching the labels on register, but also fetching the labels
once for all the nodes when all nodes are registered? If it's so, I think
this's unnecessary and difficult to implement. Firstly, the labels of all the
registered nodes have been fetched once on register. Secondly, it's difficult
to determine when to fetch the labels for all the nodes as some nodes may still
not registered. Thoughts?
{quote}
createNodeLabelsMappingProvider throws exception from serviceInit if
NodeLabelsMappingProvider class is not configured.
{quote}
I would agree with [~Naganarasimha] on this point. By default, {{centralized}}
node label configuration type is enabled. If user explicitly set the node label
configuration type to {{delegated-centralized}}, we can assume that he knows
what does it mean for {{delegated-centralized}} and he knows he should provide
a {{NodeLabelsMappingProvider}}. On the other hand, providing a
{{SimpleNodeLabelsMappingProvider}} and make it as default would also be
acceptable for me. If we decide to do this, then we should log the following
with INFO to let user know which {{NodeLabelsMappingProvider}} is being used.
{noformat}
{{LOG.debug("Node labels mapping provider class is : "
157 + nodeLabelsMappingProvider.getClass().toString());}}
{noformat}Thoughts?
> Support NodeLabelsProvider at Resource Manager side
> ---
>
> Key: YARN-3964
> URL: https://issues.apache.org/jira/browse/YARN-3964
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Dian Fu
>Assignee: Dian Fu
> Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch,
> YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch,
> YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch,
> YARN-3964.008.patch, YARN-3964.009.patch, YARN-3964.1.patch
>
>
> Currently, CLI/REST API is provided in Resource Manager to allow users to
> specify labels for nodes. For labels which may change over time, users will
> have to start a cron job to update the labels. This has the following
> limitations:
> - The cron job needs to be run in the YARN admin user.
> - This makes it a little complicate to maintain as users will have to make
> sure this service/daemon is alive.
> Adding a Node Labels Provider in Resource Manager will provide user more
> flexibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[
https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907806#comment-14907806
]
Dian Fu commented on YARN-3964:
---
Hi [~Naganarasimha],
Thanks a lot for your review. Have updated the patch according to your comments.
{quote}
CommonNodeLabelsManager.isDistributedNodeLabelConfiguration is set in
distributed mode and we avoid reading and storing to FileSystemNodeLabelsStore.
But in your case its not done, hence LabelStore Editlogs goes on increasing
over time as we not checking for weather the labels are not modified for a
given node while replacing. Eventually FileSystemNodeLabelsStore.recover during
startup(/failover) might be come slow.
{quote}
In fact, in delegated-centralized mode, we should also avoid storing the
node->labels map to FileSystemNodeLabelsStore. The new patch should have fixed
this.
{quote}
During update even if one node has Labels which is not part of CLuster Labels
it fails to update for other nodes is that fine ?
{quote}
This issue also exists when updating multiple nodes labels through CLI/REST.
Maybe we can address this in a separate JIRA if we think this is a problem.
> Support NodeLabelsProvider at Resource Manager side
> ---
>
> Key: YARN-3964
> URL: https://issues.apache.org/jira/browse/YARN-3964
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Dian Fu
>Assignee: Dian Fu
> Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch,
> YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch,
> YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch,
> YARN-3964.008.patch, YARN-3964.1.patch
>
>
> Currently, CLI/REST API is provided in Resource Manager to allow users to
> specify labels for nodes. For labels which may change over time, users will
> have to start a cron job to update the labels. This has the following
> limitations:
> - The cron job needs to be run in the YARN admin user.
> - This makes it a little complicate to maintain as users will have to make
> sure this service/daemon is alive.
> Adding a Node Labels Provider in Resource Manager will provide user more
> flexibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.008.patch > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch, > YARN-3964.008.patch, YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[
https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907351#comment-14907351
]
Dian Fu commented on YARN-3964:
---
Hi [~leftnoteasy], [~Naganarasimha], [~devaraj.k],
I have run test-patch.sh in a clean environment (one newly created virtual
machine) and all the tests have passed. Most of the test failures jenkins
reported are caused by
{noformat}
java.lang.NoSuchMethodError:
org.apache.hadoop.yarn.conf.YarnConfiguration.isDelegatedCentralizedNodeLabelConfiguration(Lorg/apache/hadoop/conf/Configuration;)Z
{noformat}
But method {{YarnConfiguration.isDelegatedCentralizedNodeLabelConfiguration}}
does exist and the signature is also correct. Could you go ahead the review? I
will also continue to investigate why the tests failed in jenkins machines.
> Support NodeLabelsProvider at Resource Manager side
> ---
>
> Key: YARN-3964
> URL: https://issues.apache.org/jira/browse/YARN-3964
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Dian Fu
>Assignee: Dian Fu
> Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch,
> YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch,
> YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch,
> YARN-3964.1.patch
>
>
> Currently, CLI/REST API is provided in Resource Manager to allow users to
> specify labels for nodes. For labels which may change over time, users will
> have to start a cron job to update the labels. This has the following
> limitations:
> - The cron job needs to be run in the YARN admin user.
> - This makes it a little complicate to maintain as users will have to make
> sure this service/daemon is alive.
> Adding a Node Labels Provider in Resource Manager will provide user more
> flexibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.007.patch Hi [~leftnoteasy], Thanks a lot for the review. Updated the patch according to your review comments. > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14905609#comment-14905609 ] Dian Fu commented on YARN-3964: --- Hi [~Naganarasimha], Thanks in advance for your review. Have marked YARN-3743 as duplicate with this one. > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (YARN-3743) Allow admin specify labels from RM with node labels provider
[ https://issues.apache.org/jira/browse/YARN-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu resolved YARN-3743. --- Resolution: Duplicate Duplicate with YARN-3964. > Allow admin specify labels from RM with node labels provider > > > Key: YARN-3743 > URL: https://issues.apache.org/jira/browse/YARN-3743 > Project: Hadoop YARN > Issue Type: Task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3743.1.patch > > > As discussed in YARN-3557, providing a node label configuration mechanism > similar to YARN-2495 at RM side would ease the use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.007.patch Attaching the 007 patch to trigger the jenkins again as I have checked the test failures in my local environment and all of them have passed. > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.007.patch, YARN-3964.007.patch, > YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.006.patch Thanks [~devaraj.k] for taking look at the patch. Attaching rebased patch. > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14901718#comment-14901718 ] Dian Fu commented on YARN-3964: --- Thanks [~leftnoteasy] for your detailed review. Make sense to me and will update the patch to incorporate your comments ASAP. > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.006.patch, YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.005.patch rebase the patch. > Support NodeLabelsProvider at Resource Manager side > --- > > Key: YARN-3964 > URL: https://issues.apache.org/jira/browse/YARN-3964 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Dian Fu >Assignee: Dian Fu > Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, > YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.005.patch, > YARN-3964.1.patch > > > Currently, CLI/REST API is provided in Resource Manager to allow users to > specify labels for nodes. For labels which may change over time, users will > have to start a cron job to update the labels. This has the following > limitations: > - The cron job needs to be run in the YARN admin user. > - This makes it a little complicate to maintain as users will have to make > sure this service/daemon is alive. > Adding a Node Labels Provider in Resource Manager will provide user more > flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.004.patch Updated the patch to address the checkstyle warnings and also add more comments. The test failures aren't relevent to this patch. Support NodeLabelsProvider at Resource Manager side --- Key: YARN-3964 URL: https://issues.apache.org/jira/browse/YARN-3964 Project: Hadoop YARN Issue Type: Sub-task Reporter: Dian Fu Assignee: Dian Fu Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, YARN-3964.003.patch, YARN-3964.004.patch, YARN-3964.1.patch Currently, CLI/REST API is provided in Resource Manager to allow users to specify labels for nodes. For labels which may change over time, users will have to start a cron job to update the labels. This has the following limitations: - The cron job needs to be run in the YARN admin user. - This makes it a little complicate to maintain as users will have to make sure this service/daemon is alive. Adding a Node Labels Provider in Resource Manager will provide user more flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.003.patch Updated the patch to fix the warnings and test failures. Support NodeLabelsProvider at Resource Manager side --- Key: YARN-3964 URL: https://issues.apache.org/jira/browse/YARN-3964 Project: Hadoop YARN Issue Type: Sub-task Reporter: Dian Fu Assignee: Dian Fu Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, YARN-3964.003.patch, YARN-3964.1.patch Currently, CLI/REST API is provided in Resource Manager to allow users to specify labels for nodes. For labels which may change over time, users will have to start a cron job to update the labels. This has the following limitations: - The cron job needs to be run in the YARN admin user. - This makes it a little complicate to maintain as users will have to make sure this service/daemon is alive. Adding a Node Labels Provider in Resource Manager will provide user more flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.002.patch Support NodeLabelsProvider at Resource Manager side --- Key: YARN-3964 URL: https://issues.apache.org/jira/browse/YARN-3964 Project: Hadoop YARN Issue Type: Sub-task Reporter: Dian Fu Assignee: Dian Fu Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, YARN-3964.1.patch Currently, CLI/REST API is provided in Resource Manager to allow users to specify labels for nodes. For labels which may change over time, users will have to start a cron job to update the labels. This has the following limitations: - The cron job needs to be run in the YARN admin user. - This makes it a little complicate to maintain as users will have to make sure this service/daemon is alive. Adding a Node Labels Provider in Resource Manager will provide user more flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14681574#comment-14681574 ] Dian Fu commented on YARN-3964: --- Updated the patch with the following updates: - remove the interface modification to NodeLabelsProvider - improve Fetcher implementation to update node labels in batch Support NodeLabelsProvider at Resource Manager side --- Key: YARN-3964 URL: https://issues.apache.org/jira/browse/YARN-3964 Project: Hadoop YARN Issue Type: Sub-task Reporter: Dian Fu Assignee: Dian Fu Attachments: YARN-3964 design doc.pdf, YARN-3964.002.patch, YARN-3964.1.patch Currently, CLI/REST API is provided in Resource Manager to allow users to specify labels for nodes. For labels which may change over time, users will have to start a cron job to update the labels. This has the following limitations: - The cron job needs to be run in the YARN admin user. - This makes it a little complicate to maintain as users will have to make sure this service/daemon is alive. Adding a Node Labels Provider in Resource Manager will provide user more flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[
https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14642498#comment-14642498
]
Dian Fu commented on YARN-3964:
---
Hi [~leftnoteasy],
Really thanks for your comments.
{quote}
I took a quick look at the patch, some problems I can see now:
- It involves some unnecessary interface/parameter to NodeLabelsProvider, this
also leads to unnecessary changes to NM
{quote}
This patch tries to move {{NodeLabelsProvider}} from
{{hadoop-yarn-server-nodemanager}} to {{hadoop-yarn-server-common}} to make it
usable by both NM and RM. But it's fine to keep it untouched.
{quote}
- Fetcher implementation is polling updated labels for ALL NMs in the cluster,
if a cluster has several thousands of NMs, this can be inefficient.
{quote}
Good advice. We can solve this issue by updating the labels for ALL NMs in one
request, not one by one. Will update the patch accordingly.
{quote}
My biggest concern is still about if this change is must-to-have:
Since we already have a set of APIs to do this, I can't see a big add-on value
of doing this inside RM.
{quote}
I understand your concern and agree that with a cron job, some scripts and REST
API, we do be able to achieve the functionality. While this improvement will
have its value. It can largely decrease the amount of additional work to do and
other difficulties for integrating a label source. Also it increases the
usability of the label feature from management perspective. We know, a lot of
times, how a technology will be adapted by users depends largely on how easily
the technology can be used or integrated. Although this is not a
must-to-have, this improvement take the label feature a step further from the
integration point of view.
For large clusters, it's usually not practical to manage the label of all nodes
manually. Enterprises usually use some kind of label or label policy storage.
This improvement can help address this requirement perfectly with the minimized
additional development work. Also, this feature can be used as a different use
case than synchronizing the labels through REST API because the configuration
of a label provider mechanism at the YARN side means the management operations
(usually done by administrator) instead of REST API operation of a client,
adding the trustfulness of label source.
Further more, we will target to make this change to be simple, light weight and
strait-forward . It will not bring any additional complexity to YARN
architecture but provide a flexible functionality for label integration.
Thank you again for your feedback.
Support NodeLabelsProvider at Resource Manager side
---
Key: YARN-3964
URL: https://issues.apache.org/jira/browse/YARN-3964
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Dian Fu
Assignee: Dian Fu
Attachments: YARN-3964 design doc.pdf, YARN-3964.1.patch
Currently, CLI/REST API is provided in Resource Manager to allow users to
specify labels for nodes. For labels which may change over time, users will
have to start a cron job to update the labels. This has the following
limitations:
- The cron job needs to be run in the YARN admin user.
- This makes it a little complicate to maintain as users will have to make
sure this service/daemon is alive.
Adding a Node Labels Provider in Resource Manager will provide user more
flexibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964.1.patch Support NodeLabelsProvider at Resource Manager side --- Key: YARN-3964 URL: https://issues.apache.org/jira/browse/YARN-3964 Project: Hadoop YARN Issue Type: Sub-task Reporter: Dian Fu Assignee: Dian Fu Attachments: YARN-3964 design doc.pdf, YARN-3964.1.patch Currently, CLI/REST API is provided in Resource Manager to allow users to specify labels for nodes. For labels which may change over time, users will have to start a cron job to update the labels. This has the following limitations: - The cron job needs to be run in the YARN admin user. - This makes it a little complicate to maintain as users will have to make sure this service/daemon is alive. Adding a Node Labels Provider in Resource Manager will provide user more flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[
https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14640220#comment-14640220
]
Dian Fu commented on YARN-3964:
---
Hi [~leftnoteasy],
Thanks a lot for your review and feedback.
{quote}
According to the design doc, configuration of centralized provider also needs
to consider 1) frequency of invoking fetch script.
{quote}
User just need to configure it via configuration and a default value can be
provided if user doesn't configure.
{quote}
2) permission of script.
{quote}
Sorry, I'm not sure what's your point? You mean the execution permission? I
don't think that would be a problem. Actually the provider don't have to be
script based.
{quote}
3) a customized provider
{quote}
The implementation of a customized provider will be very simple. User just need
to implement interface {{NodeLabelsFetcher}}.
{quote}
4) configuration RM classpath to make use of it.
{quote}
Yes, that's true.
{quote}
Configuration/executing provider script also needs YARN's admin permission.
{quote}
The provider will be part of Resource Manager, we don't need to start up a
separate process to execute it.
To clearly express the thoughts, I have uploaded an initial patch.
Thanks again for your comments.
Support NodeLabelsProvider at Resource Manager side
---
Key: YARN-3964
URL: https://issues.apache.org/jira/browse/YARN-3964
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Dian Fu
Assignee: Dian Fu
Attachments: YARN-3964 design doc.pdf, YARN-3964.1.patch
Currently, CLI/REST API is provided in Resource Manager to allow users to
specify labels for nodes. For labels which may change over time, users will
have to start a cron job to update the labels. This has the following
limitations:
- The cron job needs to be run in the YARN admin user.
- This makes it a little complicate to maintain as users will have to make
sure this service/daemon is alive.
Adding a Node Labels Provider in Resource Manager will provide user more
flexibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Issue Type: Sub-task (was: Task) Parent: YARN-2492 Support NodeLabelsProvider at Resource Manager side --- Key: YARN-3964 URL: https://issues.apache.org/jira/browse/YARN-3964 Project: Hadoop YARN Issue Type: Sub-task Reporter: Dian Fu Currently, CLI/REST API is provided in Resource Manager to allow users to specify labels for nodes. For labels which may change over time, users will have to start a cron job to update the labels. This has the following limitations: - The cron job needs to be run in the YARN admin user. - This makes it a little complicate to maintain as users will have to make sure this service/daemon is alive. Adding a Node Labels Provider in Resource Manager will provide user more flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
Dian Fu created YARN-3964: - Summary: Support NodeLabelsProvider at Resource Manager side Key: YARN-3964 URL: https://issues.apache.org/jira/browse/YARN-3964 Project: Hadoop YARN Issue Type: Task Reporter: Dian Fu Currently, CLI/REST API is provided in Resource Manager to allow users to specify labels for nodes. For labels which may change over time, users will have to start a cron job to update the labels. This has the following limitations: - The cron job needs to be run in the YARN admin user. - This makes it a little complicate to maintain as users will have to make sure this service/daemon is alive. Adding a Node Labels Provider in Resource Manager will provide user more flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3964: -- Attachment: YARN-3964 design doc.pdf A simple design doc attached. Any feedback will be welcome. Support NodeLabelsProvider at Resource Manager side --- Key: YARN-3964 URL: https://issues.apache.org/jira/browse/YARN-3964 Project: Hadoop YARN Issue Type: Sub-task Reporter: Dian Fu Attachments: YARN-3964 design doc.pdf Currently, CLI/REST API is provided in Resource Manager to allow users to specify labels for nodes. For labels which may change over time, users will have to start a cron job to update the labels. This has the following limitations: - The cron job needs to be run in the YARN admin user. - This makes it a little complicate to maintain as users will have to make sure this service/daemon is alive. Adding a Node Labels Provider in Resource Manager will provide user more flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-3964) Support NodeLabelsProvider at Resource Manager side
[ https://issues.apache.org/jira/browse/YARN-3964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu reassigned YARN-3964: - Assignee: Dian Fu Support NodeLabelsProvider at Resource Manager side --- Key: YARN-3964 URL: https://issues.apache.org/jira/browse/YARN-3964 Project: Hadoop YARN Issue Type: Sub-task Reporter: Dian Fu Assignee: Dian Fu Attachments: YARN-3964 design doc.pdf Currently, CLI/REST API is provided in Resource Manager to allow users to specify labels for nodes. For labels which may change over time, users will have to start a cron job to update the labels. This has the following limitations: - The cron job needs to be run in the YARN admin user. - This makes it a little complicate to maintain as users will have to make sure this service/daemon is alive. Adding a Node Labels Provider in Resource Manager will provide user more flexibility. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3930) FileSystemNodeLabelsStore should make sure edit log file closed when exception is thrown
Dian Fu created YARN-3930:
-
Summary: FileSystemNodeLabelsStore should make sure edit log file
closed when exception is thrown
Key: YARN-3930
URL: https://issues.apache.org/jira/browse/YARN-3930
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Dian Fu
Assignee: Dian Fu
When I test the node label feature in my local environment, I encountered the
following exception:
{code}
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2426)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:2523)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2498)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:662)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:418)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.handleStoreEvent(CommonNodeLabelsManager.java:196)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager$ForwardingEventHandler.handle(CommonNodeLabelsManager.java:168)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager$ForwardingEventHandler.handle(CommonNodeLabelsManager.java:163)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:176)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
at java.lang.Thread.run(Thread.java:745)
{code}
The reason is that HDFS throws an exception when calling
{{ensureAppendEditlogFile}} because of some reason which causes the edit log
output stream isn't closed. This caused that the next time we call
{{ensureAppendEditlogFile}}, lease recovery will failed because we are just the
lease holder.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3930) FileSystemNodeLabelsStore should make sure edit log file closed when exception is thrown
[
https://issues.apache.org/jira/browse/YARN-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated YARN-3930:
--
Attachment: YARN-3930.001.patch
A simple patch attached.
FileSystemNodeLabelsStore should make sure edit log file closed when
exception is thrown
-
Key: YARN-3930
URL: https://issues.apache.org/jira/browse/YARN-3930
Project: Hadoop YARN
Issue Type: Sub-task
Components: api, client, resourcemanager
Reporter: Dian Fu
Assignee: Dian Fu
Attachments: YARN-3930.001.patch
When I test the node label feature in my local environment, I encountered the
following exception:
{code}
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2426)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:2523)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2498)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:662)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:418)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.handleStoreEvent(CommonNodeLabelsManager.java:196)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager$ForwardingEventHandler.handle(CommonNodeLabelsManager.java:168)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager$ForwardingEventHandler.handle(CommonNodeLabelsManager.java:163)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:176)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
at java.lang.Thread.run(Thread.java:745)
{code}
The reason is that HDFS throws an exception when calling
{{ensureAppendEditlogFile}} because of some reason which causes the edit log
output stream isn't closed. This caused that the next time we call
{{ensureAppendEditlogFile}}, lease recovery will failed because we are just
the lease holder.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3930) FileSystemNodeLabelsStore should make sure edit log file closed when exception is thrown
[
https://issues.apache.org/jira/browse/YARN-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630615#comment-14630615
]
Dian Fu commented on YARN-3930:
---
Thanks [~leftnoteasy] for review and commit.
FileSystemNodeLabelsStore should make sure edit log file closed when
exception is thrown
-
Key: YARN-3930
URL: https://issues.apache.org/jira/browse/YARN-3930
Project: Hadoop YARN
Issue Type: Sub-task
Components: api, client, resourcemanager
Reporter: Dian Fu
Assignee: Dian Fu
Fix For: 2.8.0
Attachments: YARN-3930.001.patch
When I test the node label feature in my local environment, I encountered the
following exception:
{code}
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2426)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInternal(FSNamesystem.java:)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFileInt(FSNamesystem.java:2523)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSNamesystem.java:2498)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.append(NameNodeRpcServer.java:662)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.append(ClientNamenodeProtocolServerSideTranslatorPB.java:418)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2174)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1666)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2168)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.handleStoreEvent(CommonNodeLabelsManager.java:196)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager$ForwardingEventHandler.handle(CommonNodeLabelsManager.java:168)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager$ForwardingEventHandler.handle(CommonNodeLabelsManager.java:163)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:176)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:108)
at java.lang.Thread.run(Thread.java:745)
{code}
The reason is that HDFS throws an exception when calling
{{ensureAppendEditlogFile}} because of some reason which causes the edit log
output stream isn't closed. This caused that the next time we call
{{ensureAppendEditlogFile}}, lease recovery will failed because we are just
the lease holder.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2923) Support configuration based NodeLabelsProvider Service in Distributed Node Label Configuration Setup
[
https://issues.apache.org/jira/browse/YARN-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14612805#comment-14612805
]
Dian Fu commented on YARN-2923:
---
{quote}But would also would like to get inputs from other folks in the
Opensource for exposing this interface in RM side... may be based on this i
would like to move into hadoop-yarn-server-common.{quote}
Yes, of course.
Support configuration based NodeLabelsProvider Service in Distributed Node
Label Configuration Setup
-
Key: YARN-2923
URL: https://issues.apache.org/jira/browse/YARN-2923
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Fix For: 2.8.0
Attachments: YARN-2923.20141204-1.patch, YARN-2923.20141210-1.patch,
YARN-2923.20150328-1.patch, YARN-2923.20150404-1.patch,
YARN-2923.20150517-1.patch
As part of Distributed Node Labels configuration we need to support Node
labels to be configured in Yarn-site.xml. And on modification of Node Labels
configuration in yarn-site.xml, NM should be able to get modified Node labels
from this NodeLabelsprovider service without NM restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2923) Support configuration based NodeLabelsProvider Service in Distributed Node Label Configuration Setup
[
https://issues.apache.org/jira/browse/YARN-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14605238#comment-14605238
]
Dian Fu commented on YARN-2923:
---
Hi [~Naganarasimha],
Sorry to come to this JIRA late. Thanks a lot for working on this. Most codes
LGTM. Just have the following rough thoughts:
1. Classes such as {{AbstractNodeLabelsProvider}} are located in directory
{{hadoop-yarn-server-nodemanager}}, I'm think if we can move it to
{{hadoop-yarn-server-common}}. This would ease the work of adding similar node
label provider mechanism at RM side in the future if possible. There may need
some minor code reactor work to do, such as it accesses NM side configuration
{{NM_NODE_LABELS_PROVIDER_FETCH_INTERVAL_MS}}.
2. For YARN-2923 and YARN-2729, both {{ScriptBasedNodeLabelsProvider}} and
{{ConfigurationNodeLabelsProvider}} have a mechanism to periodically execute a
script or read configuration to fetch labels. Is it possible to merge
{{ScriptBasedNodeLabelsProvider}} and {{ConfigurationNodeLabelsProvider}} into
one class? This class can provide the function to periodically fetch labels and
the actual work of how to fetch labels can be configured in a plugable way. The
benefits of this approach are: a) remove duplicate code in
{{ScriptBasedNodeLabelsProvider}} and {{ConfigurationNodeLabelsProvider}}; b)
ease third parts to add their own node label provider.
Not sure if the above thoughts are clear and make sense to you? I can help to
do some POC based on YARN-2923 and YARN-2729 if needed.
Support configuration based NodeLabelsProvider Service in Distributed Node
Label Configuration Setup
-
Key: YARN-2923
URL: https://issues.apache.org/jira/browse/YARN-2923
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Naganarasimha G R
Assignee: Naganarasimha G R
Fix For: 2.8.0
Attachments: YARN-2923.20141204-1.patch, YARN-2923.20141210-1.patch,
YARN-2923.20150328-1.patch, YARN-2923.20150404-1.patch,
YARN-2923.20150517-1.patch
As part of Distributed Node Labels configuration we need to support Node
labels to be configured in Yarn-site.xml. And on modification of Node Labels
configuration in yarn-site.xml, NM should be able to get modified Node labels
from this NodeLabelsprovider service without NM restart
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-3743) Allow admin specify labels from RM with node labels provider
[ https://issues.apache.org/jira/browse/YARN-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu reassigned YARN-3743: - Assignee: Dian Fu Allow admin specify labels from RM with node labels provider Key: YARN-3743 URL: https://issues.apache.org/jira/browse/YARN-3743 Project: Hadoop YARN Issue Type: Task Reporter: Dian Fu Assignee: Dian Fu Attachments: YARN-3743.1.patch As discussed in YARN-3557, providing a node label configuration mechanism similar to YARN-2495 at RM side would ease the use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3743) Allow admin specify labels from RM with node labels provider
[ https://issues.apache.org/jira/browse/YARN-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14566859#comment-14566859 ] Dian Fu commented on YARN-3743: --- Hi [~Naganarasimha], I know that you are the owner of YARN-2923 and YARN-2729, could you help to review and give some comments? Any comments would be welcome. Thanks a lot. Allow admin specify labels from RM with node labels provider Key: YARN-3743 URL: https://issues.apache.org/jira/browse/YARN-3743 Project: Hadoop YARN Issue Type: Task Reporter: Dian Fu Attachments: YARN-3743.1.patch As discussed in YARN-3557, providing a node label configuration mechanism similar to YARN-2495 at RM side would ease the use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3743) Allow admin specify labels from RM with node labels provider
Dian Fu created YARN-3743: - Summary: Allow admin specify labels from RM with node labels provider Key: YARN-3743 URL: https://issues.apache.org/jira/browse/YARN-3743 Project: Hadoop YARN Issue Type: Task Reporter: Dian Fu As discussed in YARN-3557, providing a node label configuration mechanism similar to YARN-2495 at RM side would ease the use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3743) Allow admin specify labels from RM with node labels provider
[ https://issues.apache.org/jira/browse/YARN-3743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3743: -- Attachment: YARN-3743.1.patch Uploaded a quick patch. This patch is dependent on YARN-2923 and YARN-2729. Allow admin specify labels from RM with node labels provider Key: YARN-3743 URL: https://issues.apache.org/jira/browse/YARN-3743 Project: Hadoop YARN Issue Type: Task Reporter: Dian Fu Attachments: YARN-3743.1.patch As discussed in YARN-3557, providing a node label configuration mechanism similar to YARN-2495 at RM side would ease the use. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3557) Support Intel Trusted Execution Technology(TXT) in YARN scheduler
[
https://issues.apache.org/jira/browse/YARN-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541165#comment-14541165
]
Dian Fu commented on YARN-3557:
---
Hi [~leftnoteasy],
Thanks a lot for your comments.
{quote}We can continue discuss it along with design of YARN-3409.{quote}
OK.
Support Intel Trusted Execution Technology(TXT) in YARN scheduler
-
Key: YARN-3557
URL: https://issues.apache.org/jira/browse/YARN-3557
Project: Hadoop YARN
Issue Type: New Feature
Reporter: Dian Fu
Attachments: Support TXT in YARN high level design doc.pdf
Intel TXT defines platform-level enhancements that provide the building
blocks for creating trusted platforms. A TXT aware YARN scheduler can
schedule security sensitive jobs on TXT enabled nodes only. YARN-2492
provides the capacity to restrict YARN applications to run only on cluster
nodes that have a specified node label. This is a good mechanism that be
utilized for TXT aware YARN scheduler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3557) Support Intel Trusted Execution Technology(TXT) in YARN scheduler
[ https://issues.apache.org/jira/browse/YARN-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537681#comment-14537681 ] Dian Fu commented on YARN-3557: --- Hi [~leftnoteasy], I have posted the requirements about supporting configure constraints node label from both RM and NM on YARN-3409. About support script based node label configuration at RM side, what's your thought? Support Intel Trusted Execution Technology(TXT) in YARN scheduler - Key: YARN-3557 URL: https://issues.apache.org/jira/browse/YARN-3557 Project: Hadoop YARN Issue Type: New Feature Reporter: Dian Fu Attachments: Support TXT in YARN high level design doc.pdf Intel TXT defines platform-level enhancements that provide the building blocks for creating trusted platforms. A TXT aware YARN scheduler can schedule security sensitive jobs on TXT enabled nodes only. YARN-2492 provides the capacity to restrict YARN applications to run only on cluster nodes that have a specified node label. This is a good mechanism that be utilized for TXT aware YARN scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3409) Add constraint node labels
[ https://issues.apache.org/jira/browse/YARN-3409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537676#comment-14537676 ] Dian Fu commented on YARN-3409: --- Just to post requirements discussed in YARN-3557 here: Constraint node labels should be supported to be added from both RM and NM. As some labels such as TRUSTED/UNTRUSTED described in YARN-3557 require to be added from RM and some labels such as GPU, FPGA, LINUX, WINDOWS are more suitable to be added from NM. A large cluster may have all these kinds of labels coexist. Add constraint node labels -- Key: YARN-3409 URL: https://issues.apache.org/jira/browse/YARN-3409 Project: Hadoop YARN Issue Type: Sub-task Components: api, capacityscheduler, client Reporter: Wangda Tan Assignee: Wangda Tan Specify only one label for each node (IAW, partition a cluster) is a way to determinate how resources of a special set of nodes could be shared by a group of entities (like teams, departments, etc.). Partitions of a cluster has following characteristics: - Cluster divided to several disjoint sub clusters. - ACL/priority can apply on partition (Only market team / marke team has priority to use the partition). - Percentage of capacities can apply on partition (Market team has 40% minimum capacity and Dev team has 60% of minimum capacity of the partition). Constraints are orthogonal to partition, they’re describing attributes of node’s hardware/software just for affinity. Some example of constraints: - glibc version - JDK version - Type of CPU (x86_64/i686) - Type of OS (windows, linux, etc.) With this, application can be able to ask for resource has (glibc.version = 2.20 JDK.version = 8u20 x86_64). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3557) Support Intel Trusted Execution Technology(TXT) in YARN scheduler
[ https://issues.apache.org/jira/browse/YARN-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530339#comment-14530339 ] Dian Fu commented on YARN-3557: --- Hi [~leftnoteasy], I have reconsidered the support of script based node label configuration at RM side and think this support is meaningful. I agree that users can write a script and periodically executes the script to update the labels via REST APIs. But this requires users to execute the script with admin user. Moreover, I think this requirement is common and implementing this at RM side will ease the use. As NM side already has such support, an implementation at RM side will not require too many efforts. Support Intel Trusted Execution Technology(TXT) in YARN scheduler - Key: YARN-3557 URL: https://issues.apache.org/jira/browse/YARN-3557 Project: Hadoop YARN Issue Type: New Feature Reporter: Dian Fu Attachments: Support TXT in YARN high level design doc.pdf Intel TXT defines platform-level enhancements that provide the building blocks for creating trusted platforms. A TXT aware YARN scheduler can schedule security sensitive jobs on TXT enabled nodes only. YARN-2492 provides the capacity to restrict YARN applications to run only on cluster nodes that have a specified node label. This is a good mechanism that be utilized for TXT aware YARN scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3557) Support Intel Trusted Execution Technology(TXT) in YARN scheduler
[
https://issues.apache.org/jira/browse/YARN-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14529870#comment-14529870
]
Dian Fu commented on YARN-3557:
---
Hi [~Naganarasimha],
{quote}constraints should be supported to be added from both RM and NM and
partitions (existing labels) should be allowed either by RM or NM{quote}
Agree with you that constraints should be supported to be added from both RM
and RN. TRUSTED/UNTRUSTED are also more like constraints of a node. BTW, it
seems that constraints support is already created(YARN-3409).
Support Intel Trusted Execution Technology(TXT) in YARN scheduler
-
Key: YARN-3557
URL: https://issues.apache.org/jira/browse/YARN-3557
Project: Hadoop YARN
Issue Type: New Feature
Reporter: Dian Fu
Attachments: Support TXT in YARN high level design doc.pdf
Intel TXT defines platform-level enhancements that provide the building
blocks for creating trusted platforms. A TXT aware YARN scheduler can
schedule security sensitive jobs on TXT enabled nodes only. YARN-2492
provides the capacity to restrict YARN applications to run only on cluster
nodes that have a specified node label. This is a good mechanism that be
utilized for TXT aware YARN scheduler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3557) Support Intel Trusted Execution Technology(TXT) in YARN scheduler
[ https://issues.apache.org/jira/browse/YARN-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14529701#comment-14529701 ] Dian Fu commented on YARN-3557: --- Hi [~leftnoteasy] In a large cluster, it will not only use labels TRUSTED/UNTRUSTED, but also lables GPU, FPGA, LINUX, WINDOWS, etc. If you need labels which require centralized configuration such as TRUSTED/UNTRUSTED, then all other labels will have to be configured in centralized way. But labels such as GPU, LINUX, WINDOWS are more suitable for configured in distributed way. Support Intel Trusted Execution Technology(TXT) in YARN scheduler - Key: YARN-3557 URL: https://issues.apache.org/jira/browse/YARN-3557 Project: Hadoop YARN Issue Type: New Feature Reporter: Dian Fu Attachments: Support TXT in YARN high level design doc.pdf Intel TXT defines platform-level enhancements that provide the building blocks for creating trusted platforms. A TXT aware YARN scheduler can schedule security sensitive jobs on TXT enabled nodes only. YARN-2492 provides the capacity to restrict YARN applications to run only on cluster nodes that have a specified node label. This is a good mechanism that be utilized for TXT aware YARN scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3557) Support Intel Trusted Execution Technology(TXT) in YARN scheduler
[
https://issues.apache.org/jira/browse/YARN-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526416#comment-14526416
]
Dian Fu commented on YARN-3557:
---
Hi [~sunilg],
Thanks for your comments.
{quote}Apart from CLI and REST, do u mean like exposing these configuration for
a specific user (i assume this user will have some security approval in the
cluster) so that this user can make the config via REST or api's.{quote}
Exposing these configuration for a specific user can be one option. But this
will require users to start a job which updates the labels periodically and is
complicated for users. If we can provide the similar method to YARN-2495 at RM
side, user will just need to provide a script(which takes node hostname/ip as
input and output the node labels).
Support Intel Trusted Execution Technology(TXT) in YARN scheduler
-
Key: YARN-3557
URL: https://issues.apache.org/jira/browse/YARN-3557
Project: Hadoop YARN
Issue Type: New Feature
Reporter: Dian Fu
Attachments: Support TXT in YARN high level design doc.pdf
Intel TXT defines platform-level enhancements that provide the building
blocks for creating trusted platforms. A TXT aware YARN scheduler can
schedule security sensitive jobs on TXT enabled nodes only. YARN-2492
provides the capacity to restrict YARN applications to run only on cluster
nodes that have a specified node label. This is a good mechanism that be
utilized for TXT aware YARN scheduler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3557) Support Intel Trusted Execution Technology(TXT) in YARN scheduler
[ https://issues.apache.org/jira/browse/YARN-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527699#comment-14527699 ] Dian Fu commented on YARN-3557: --- Hi [~leftnoteasy], Thanks a lot for your comments. What about the support of both distributed configuration and centralized configuration? Any thoughts about the solution I mentioned in the above comment? Support Intel Trusted Execution Technology(TXT) in YARN scheduler - Key: YARN-3557 URL: https://issues.apache.org/jira/browse/YARN-3557 Project: Hadoop YARN Issue Type: New Feature Reporter: Dian Fu Attachments: Support TXT in YARN high level design doc.pdf Intel TXT defines platform-level enhancements that provide the building blocks for creating trusted platforms. A TXT aware YARN scheduler can schedule security sensitive jobs on TXT enabled nodes only. YARN-2492 provides the capacity to restrict YARN applications to run only on cluster nodes that have a specified node label. This is a good mechanism that be utilized for TXT aware YARN scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3557) Support Intel Trusted Execution Technology(TXT) in YARN scheduler
[
https://issues.apache.org/jira/browse/YARN-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526205#comment-14526205
]
Dian Fu commented on YARN-3557:
---
Hi [~leftnoteasy] [~Naganarasimha],
Thanks a lot for your review and comments.
{quote}Scheduler doesn't need to know if a node is trusted or not, trusted
will be a generic label of a node{quote}
Yes. This scenario can be seen as a use case of label feature.
{quote}Now RM supports using CLI or REST API, are they enough for you to
configure NM's trusted status?{quote}
Yes, RM supports using CLI and REST API, but these APIs require the admin
permission and from my point of view, these APIs are more suitable for the
scenarios in which admin knows when the labels of nodes should be modified, is
this correct? While in the scenario described in this JIRA, admin don't know
when the security label of the nodes change, so periodical security check need
to be performed and the labels may change at any time.
{quote}Configure both could be problematic, see my comment:
https://issues.apache.org/jira/browse/YARN-2495?focusedCommentId=14317048page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14317048.{quote}
I agree with you that configure both may be problematic for the example you
described. I'm thinking if we can make some labels configured in centralized
way and some labels configured in distributed way? For the same label it can
only be configured in one method.
Support both configurations may be necessary as some labels may require
centralized configuration be used such as the label in this use case and some
labels may require distributed configuration. It's very likely that some use
cases may require both these two kinds of labels.
{quote}Did you mean NM here ? RM side configure is already there and NM side
(distributed) is almost done and might be available in 2.8.{quote}
I mean RM here. As in the security label use case, NM may be running on
untrustworthy nodes. So we cannot trust the security lables NM reported.
{quote}If you have selected the 2nd option RM retrieve the trust status of all
cluster nodes from OAT, then why is it dependent on YARN-2495 support to
configure centralized node label configuration or distributed node label
configuration required ? {quote}
It isn't dependent on YARN-2495. I mean we may need some methods to configure
node labels in RM side which can use similar methods in YARN-2495. Sorry to let
you misunderstand.
Support Intel Trusted Execution Technology(TXT) in YARN scheduler
-
Key: YARN-3557
URL: https://issues.apache.org/jira/browse/YARN-3557
Project: Hadoop YARN
Issue Type: New Feature
Reporter: Dian Fu
Attachments: Support TXT in YARN high level design doc.pdf
Intel TXT defines platform-level enhancements that provide the building
blocks for creating trusted platforms. A TXT aware YARN scheduler can
schedule security sensitive jobs on TXT enabled nodes only. YARN-2492
provides the capacity to restrict YARN applications to run only on cluster
nodes that have a specified node label. This is a good mechanism that be
utilized for TXT aware YARN scheduler.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3557) Support Intel Trusted Execution Technology(TXT) in YARN scheduler
[ https://issues.apache.org/jira/browse/YARN-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3557: -- Attachment: Support TXT in YARN high level design doc.pdf A high level design doc attached. Support Intel Trusted Execution Technology(TXT) in YARN scheduler - Key: YARN-3557 URL: https://issues.apache.org/jira/browse/YARN-3557 Project: Hadoop YARN Issue Type: New Feature Reporter: Dian Fu Attachments: Support TXT in YARN high level design doc.pdf Intel TXT defines platform-level enhancements that provide the building blocks for creating trusted platforms. A TXT aware YARN scheduler can schedule security sensitive jobs on TXT enabled nodes only. YARN-2492 provides the capacity to restrict YARN applications to run only on cluster nodes that have a specified node label. This is a good mechanism that be utilized for TXT aware YARN scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3557) Support Intel Trusted Execution Technology(TXT) in YARN scheduler
[ https://issues.apache.org/jira/browse/YARN-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518573#comment-14518573 ] Dian Fu commented on YARN-3557: --- Will post a design doc ASAP. Support Intel Trusted Execution Technology(TXT) in YARN scheduler - Key: YARN-3557 URL: https://issues.apache.org/jira/browse/YARN-3557 Project: Hadoop YARN Issue Type: New Feature Reporter: Dian Fu Intel TXT defines platform-level enhancements that provide the building blocks for creating trusted platforms. A TXT aware YARN scheduler can schedule security sensitive jobs on TXT enabled nodes only. YARN-2492 provides the capacity to restrict YARN applications to run only on cluster nodes that have a specified node label. This is a good mechanism that be utilized for TXT aware YARN scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3557) Support Intel Trusted Execution Technology(TXT) in YARN scheduler
[ https://issues.apache.org/jira/browse/YARN-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated YARN-3557: -- Issue Type: New Feature (was: Bug) Support Intel Trusted Execution Technology(TXT) in YARN scheduler - Key: YARN-3557 URL: https://issues.apache.org/jira/browse/YARN-3557 Project: Hadoop YARN Issue Type: New Feature Reporter: Dian Fu Intel TXT defines platform-level enhancements that provide the building blocks for creating trusted platforms. A TXT aware YARN scheduler can schedule security sensitive jobs on TXT enabled nodes only. YARN-2492 provides the capacity to restrict YARN applications to run only on cluster nodes that have a specified node label. This is a good mechanism that be utilized for TXT aware YARN scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-3557) Support Intel Trusted Execution Technology(TXT) in YARN scheduler
Dian Fu created YARN-3557: - Summary: Support Intel Trusted Execution Technology(TXT) in YARN scheduler Key: YARN-3557 URL: https://issues.apache.org/jira/browse/YARN-3557 Project: Hadoop YARN Issue Type: Bug Reporter: Dian Fu Intel TXT defines platform-level enhancements that provide the building blocks for creating trusted platforms. A TXT aware YARN scheduler can schedule security sensitive jobs on TXT enabled nodes only. YARN-2492 provides the capacity to restrict YARN applications to run only on cluster nodes that have a specified node label. This is a good mechanism that be utilized for TXT aware YARN scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
