[jira] [Updated] (YARN-3841) [Storage implementation] Adding retry semantics to HDFS backing storage
[
https://issues.apache.org/jira/browse/YARN-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Modi updated YARN-3841:
Attachment: YARN-3841.004.patch
> [Storage implementation] Adding retry semantics to HDFS backing storage
> ---
>
> Key: YARN-3841
> URL: https://issues.apache.org/jira/browse/YARN-3841
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Tsuyoshi Ozawa
>Assignee: Abhishek Modi
>Priority: Major
> Labels: YARN-5355
> Attachments: YARN-3841-YARN-7055.002.patch, YARN-3841.001.patch,
> YARN-3841.002.patch, YARN-3841.003.patch, YARN-3841.004.patch
>
>
> HDFS backing storage is useful for following scenarios.
> 1. For Hadoop clusters which don't run HBase.
> 2. For fallback from HBase when HBase cluster is temporary unavailable.
> Quoting ATS design document of YARN-2928:
> {quote}
> In the case the HBase
> storage is not available, the plugin should buffer the writes temporarily
> (e.g. HDFS), and flush
> them once the storage comes back online. Reading and writing to hdfs as the
> the backup storage
> could potentially use the HDFS writer plugin unless the complexity of
> generalizing the HDFS
> writer plugin for this purpose exceeds the benefits of reusing it here.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3879) [Storage implementation] Create HDFS backing storage implementation for ATS reads
[ https://issues.apache.org/jira/browse/YARN-3879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-3879: Attachment: YARN-3879.004.patch > [Storage implementation] Create HDFS backing storage implementation for ATS > reads > - > > Key: YARN-3879 > URL: https://issues.apache.org/jira/browse/YARN-3879 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Tsuyoshi Ozawa >Assignee: Abhishek Modi >Priority: Major > Labels: YARN-5355 > Attachments: YARN-3879-YARN-7055.001.patch, YARN-3879.001.patch, > YARN-3879.002.patch, YARN-3879.003.patch, YARN-3879.004.patch > > > Reader version of YARN-3841 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7953) [GQ] Data structures for federation global queues calculations
[ https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565095#comment-16565095 ] Abhishek Modi commented on YARN-7953: - I think we need to rebase YARN-7402 with latest master. [~botong] could you please do it. Thanks. > [GQ] Data structures for federation global queues calculations > -- > > Key: YARN-7953 > URL: https://issues.apache.org/jira/browse/YARN-7953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7953-YARN-7402.v1.patch, > YARN-7953-YARN-7402.v2.patch, YARN-7953-YARN-7402.v3.patch, > YARN-7953-YARN-7402.v4.patch, YARN-7953-YARN-7402.v5.patch, > YARN-7953-YARN-7402.v6.patch, YARN-7953-YARN-7402.v7.patch, YARN-7953.v1.patch > > > This Jira tracks data structures and helper classes used by the core > algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7953) [GQ] Data structures for federation global queues calculations
[ https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-7953: Attachment: YARN-7953-YARN-7402.v7.patch > [GQ] Data structures for federation global queues calculations > -- > > Key: YARN-7953 > URL: https://issues.apache.org/jira/browse/YARN-7953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7953-YARN-7402.v1.patch, > YARN-7953-YARN-7402.v2.patch, YARN-7953-YARN-7402.v3.patch, > YARN-7953-YARN-7402.v4.patch, YARN-7953-YARN-7402.v5.patch, > YARN-7953-YARN-7402.v6.patch, YARN-7953-YARN-7402.v7.patch, YARN-7953.v1.patch > > > This Jira tracks data structures and helper classes used by the core > algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8155) Improve ATSv2 client logging in RM and NM publisher
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565001#comment-16565001 ] Abhishek Modi commented on YARN-8155: - [~rohithsharma] attached patch for branch-2. Could you please review it. Thanks. > Improve ATSv2 client logging in RM and NM publisher > --- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.2.0, 3.1.1, 3.0.4 > > Attachments: YARN-8155-branch-2.002.patch, > YARN-8155-branch-2.v1.patch, YARN-8155-branch-2.v3.patch, > YARN-8155.001.patch, YARN-8155.002.patch, YARN-8155.003.patch, > YARN-8155.004.patch, YARN-8155.005.patch, YARN-8155.006.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8155) Improve ATSv2 client logging in RM and NM publisher
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8155: Attachment: YARN-8155-branch-2.v3.patch > Improve ATSv2 client logging in RM and NM publisher > --- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.2.0, 3.1.1, 3.0.4 > > Attachments: YARN-8155-branch-2.002.patch, > YARN-8155-branch-2.v1.patch, YARN-8155-branch-2.v3.patch, > YARN-8155.001.patch, YARN-8155.002.patch, YARN-8155.003.patch, > YARN-8155.004.patch, YARN-8155.005.patch, YARN-8155.006.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8175) Add support for Node Labels in SLS
[ https://issues.apache.org/jira/browse/YARN-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563958#comment-16563958 ] Abhishek Modi commented on YARN-8175: - Thanks [~elgoiri] for review and committing it to trunk. > Add support for Node Labels in SLS > -- > > Key: YARN-8175 > URL: https://issues.apache.org/jira/browse/YARN-8175 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.1.0 >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8175.001.patch, YARN-8175.002.patch, > YARN-8175.003.patch, YARN-8175.004.patch, YARN-8175.005.patch, > YARN-8175.006.patch, YARN-8175.007.patch, YARN-8175.008.patch, > YARN-8175.009.patch > > > Currently, SLS doesn't support node labels. With this jira, we are planning > to add support for node labels in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8175) Add support for Node Labels in SLS
[ https://issues.apache.org/jira/browse/YARN-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8175: Attachment: YARN-8175.009.patch > Add support for Node Labels in SLS > -- > > Key: YARN-8175 > URL: https://issues.apache.org/jira/browse/YARN-8175 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8175.001.patch, YARN-8175.002.patch, > YARN-8175.003.patch, YARN-8175.004.patch, YARN-8175.005.patch, > YARN-8175.006.patch, YARN-8175.007.patch, YARN-8175.008.patch, > YARN-8175.009.patch > > > Currently, SLS doesn't support node labels. With this jira, we are planning > to add support for node labels in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8175) Add support for Node Labels in SLS
[ https://issues.apache.org/jira/browse/YARN-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561775#comment-16561775 ] Abhishek Modi commented on YARN-8175: - [~elgoiri] This test is unrelated to this patch. I locally checked and test is passing. Thanks. > Add support for Node Labels in SLS > -- > > Key: YARN-8175 > URL: https://issues.apache.org/jira/browse/YARN-8175 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8175.001.patch, YARN-8175.002.patch, > YARN-8175.003.patch, YARN-8175.004.patch, YARN-8175.005.patch, > YARN-8175.006.patch, YARN-8175.007.patch, YARN-8175.008.patch > > > Currently, SLS doesn't support node labels. With this jira, we are planning > to add support for node labels in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8175) Add support for Node Labels in SLS
[ https://issues.apache.org/jira/browse/YARN-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8175: Attachment: YARN-8175.008.patch > Add support for Node Labels in SLS > -- > > Key: YARN-8175 > URL: https://issues.apache.org/jira/browse/YARN-8175 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8175.001.patch, YARN-8175.002.patch, > YARN-8175.003.patch, YARN-8175.004.patch, YARN-8175.005.patch, > YARN-8175.006.patch, YARN-8175.007.patch, YARN-8175.008.patch > > > Currently, SLS doesn't support node labels. With this jira, we are planning > to add support for node labels in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8155) Improve ATSv2 client logging in RM and NM publisher
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8155: Attachment: YARN-8155-branch-2.002.patch > Improve ATSv2 client logging in RM and NM publisher > --- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.2.0, 3.1.1, 3.0.4 > > Attachments: YARN-8155-branch-2.002.patch, > YARN-8155-branch-2.v1.patch, YARN-8155.001.patch, YARN-8155.002.patch, > YARN-8155.003.patch, YARN-8155.004.patch, YARN-8155.005.patch, > YARN-8155.006.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8180) Remove yarn.federation.blacklist-subclusters from yarn federation doc
[ https://issues.apache.org/jira/browse/YARN-8180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16530169#comment-16530169 ] Abhishek Modi commented on YARN-8180: - [~giovanni.fumarola] Updated the Jira title and description. > Remove yarn.federation.blacklist-subclusters from yarn federation doc > - > > Key: YARN-8180 > URL: https://issues.apache.org/jira/browse/YARN-8180 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Reporter: Shen Yinjie >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8180.001.patch > > > Property "yarn.federation.blacklist-subclusters" was added in yarn-federation > doc by mistake and is not applicable. This Jira is to remove this property > from the doc. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8180) Remove yarn.federation.blacklist-subclusters from yarn federation doc
[
https://issues.apache.org/jira/browse/YARN-8180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Modi updated YARN-8180:
Description:
Property "yarn.federation.blacklist-subclusters" was added in yarn-federation
doc by mistake and is not applicable. This Jira is to remove this property from
the doc.
was:
Property "yarn.federation.blacklist-subclusters" is defined in yarn-fedeartion
doc,but it has not been defined and implemented in Java code.
In FederationClientInterceptor#submitApplication()
{code:java}
List blacklist = new ArrayList();
for (int i = 0; i < numSubmitRetries; ++i) {
SubClusterId subClusterId = policyFacade.getHomeSubcluster(
request.getApplicationSubmissionContext(), blacklist);
{code}
> Remove yarn.federation.blacklist-subclusters from yarn federation doc
> -
>
> Key: YARN-8180
> URL: https://issues.apache.org/jira/browse/YARN-8180
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Reporter: Shen Yinjie
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-8180.001.patch
>
>
> Property "yarn.federation.blacklist-subclusters" was added in yarn-federation
> doc by mistake and is not applicable. This Jira is to remove this property
> from the doc.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8180) Remove yarn.federation.blacklist-subclusters from yarn federation doc
[
https://issues.apache.org/jira/browse/YARN-8180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Modi updated YARN-8180:
Summary: Remove yarn.federation.blacklist-subclusters from yarn federation
doc (was: YARN Federation has not implemented blacklist sub-cluster for AM
routing)
> Remove yarn.federation.blacklist-subclusters from yarn federation doc
> -
>
> Key: YARN-8180
> URL: https://issues.apache.org/jira/browse/YARN-8180
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Reporter: Shen Yinjie
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-8180.001.patch
>
>
> Property "yarn.federation.blacklist-subclusters" is defined in
> yarn-fedeartion doc,but it has not been defined and implemented in Java code.
> In FederationClientInterceptor#submitApplication()
> {code:java}
> List blacklist = new ArrayList();
> for (int i = 0; i < numSubmitRetries; ++i) {
> SubClusterId subClusterId = policyFacade.getHomeSubcluster(
> request.getApplicationSubmissionContext(), blacklist);
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8180) YARN Federation has not implemented blacklist sub-cluster for AM routing
[
https://issues.apache.org/jira/browse/YARN-8180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516994#comment-16516994
]
Abhishek Modi commented on YARN-8180:
-
[~giovanni.fumarola] [~elgoiri] could you please review it.
> YARN Federation has not implemented blacklist sub-cluster for AM routing
>
>
> Key: YARN-8180
> URL: https://issues.apache.org/jira/browse/YARN-8180
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Reporter: Shen Yinjie
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-8180.001.patch
>
>
> Property "yarn.federation.blacklist-subclusters" is defined in
> yarn-fedeartion doc,but it has not been defined and implemented in Java code.
> In FederationClientInterceptor#submitApplication()
> {code:java}
> List blacklist = new ArrayList();
> for (int i = 0; i < numSubmitRetries; ++i) {
> SubClusterId subClusterId = policyFacade.getHomeSubcluster(
> request.getApplicationSubmissionContext(), blacklist);
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8155) Improve ATSv2 client logging in RM and NM publisher
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8155: Attachment: YARN-8155.006.patch > Improve ATSv2 client logging in RM and NM publisher > --- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.2.0, 3.1.1, 3.0.4 > > Attachments: YARN-8155-branch-2.v1.patch, YARN-8155.001.patch, > YARN-8155.002.patch, YARN-8155.003.patch, YARN-8155.004.patch, > YARN-8155.005.patch, YARN-8155.006.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8155) Improve ATSv2 client logging in RM and NM publisher
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8155: Attachment: YARN-8155-branch-2.v1.patch > Improve ATSv2 client logging in RM and NM publisher > --- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.2.0, 3.1.1, 3.0.4 > > Attachments: YARN-8155-branch-2.v1.patch, YARN-8155.001.patch, > YARN-8155.002.patch, YARN-8155.003.patch, YARN-8155.004.patch, > YARN-8155.005.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8431) Add whitelist/blacklist support for ATSv2 events.
Abhishek Modi created YARN-8431: --- Summary: Add whitelist/blacklist support for ATSv2 events. Key: YARN-8431 URL: https://issues.apache.org/jira/browse/YARN-8431 Project: Hadoop YARN Issue Type: Improvement Reporter: Abhishek Modi Assignee: Abhishek Modi In this jira, we will add functionality in ATSv2 to blacklist events at cluster level. Blacklisting of events should not require restart of any of the services and should apply dynamically. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8155) Improve ATSv2 client logging in RM and NM publisher
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512195#comment-16512195 ] Abhishek Modi commented on YARN-8155: - [~rohithsharma] sure, I will upload a patch for branch-2. > Improve ATSv2 client logging in RM and NM publisher > --- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.2.0, 3.1.1, 3.0.4 > > Attachments: YARN-8155.001.patch, YARN-8155.002.patch, > YARN-8155.003.patch, YARN-8155.004.patch, YARN-8155.005.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6931) Make the aggregation interval in AppLevelTimelineCollector configurable
[ https://issues.apache.org/jira/browse/YARN-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509892#comment-16509892 ] Abhishek Modi commented on YARN-6931: - Thanks [~haibochen] for review and committing it to trunk. > Make the aggregation interval in AppLevelTimelineCollector configurable > --- > > Key: YARN-6931 > URL: https://issues.apache.org/jira/browse/YARN-6931 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Abhishek Modi >Priority: Minor > Fix For: 3.2.0 > > Attachments: YARN-6931.001.patch, YARN-6931.002.patch > > > We do application-level metrics aggregation in AppLevelTimelineCollector, but > the interval is hardcoded. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8304) Provide generic log servlet for serving logs
[ https://issues.apache.org/jira/browse/YARN-8304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi reassigned YARN-8304: --- Assignee: Abhishek Modi > Provide generic log servlet for serving logs > > > Key: YARN-8304 > URL: https://issues.apache.org/jira/browse/YARN-8304 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Critical > > AHSWebService has log serving REST APIs i.e getContainerLog* and getLogs > which is used to view container logs from UI. It is tightly coupled with > ApplicationBaseProtocol. And these APIs are exist in AHS. But ATSv2 is > designed only with REST APIs. > Proposal is to add generic log servlet which could be plugged to ATSV1.5 or > ATSv2.0 Reader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8303) YarnClient should contact TimelineReader for application/attempt/container report
[ https://issues.apache.org/jira/browse/YARN-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509666#comment-16509666 ] Abhishek Modi commented on YARN-8303: - Thanks [~rohithsharma]. I will start working on it soon. > YarnClient should contact TimelineReader for application/attempt/container > report > - > > Key: YARN-8303 > URL: https://issues.apache.org/jira/browse/YARN-8303 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Critical > > YarnClient get app/attempt/container information from RM. If RM doesn't have > then queried to ahsClient. When ATSv2 is only enabled, yarnClient will result > empty. > YarnClient is used by many users which result in empty information for > app/attempt/container report. > Proposal is to have adapter from yarn client so that app/attempt/container > reports can be generated from AHSv2Client which does REST API to > TimelineReader and get the entity and convert it into app/attempt/container > report. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6931) Make the aggregation interval in AppLevelTimelineCollector configurable
[ https://issues.apache.org/jira/browse/YARN-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509663#comment-16509663 ] Abhishek Modi commented on YARN-6931: - Thanks [~haibochen] for the review. I have submitted v2 patch with above mentioned change. > Make the aggregation interval in AppLevelTimelineCollector configurable > --- > > Key: YARN-6931 > URL: https://issues.apache.org/jira/browse/YARN-6931 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Abhishek Modi >Priority: Minor > Attachments: YARN-6931.001.patch, YARN-6931.002.patch > > > We do application-level metrics aggregation in AppLevelTimelineCollector, but > the interval is hardcoded. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6931) Make the aggregation interval in AppLevelTimelineCollector configurable
[ https://issues.apache.org/jira/browse/YARN-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-6931: Attachment: YARN-6931.002.patch > Make the aggregation interval in AppLevelTimelineCollector configurable > --- > > Key: YARN-6931 > URL: https://issues.apache.org/jira/browse/YARN-6931 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Abhishek Modi >Priority: Minor > Attachments: YARN-6931.001.patch, YARN-6931.002.patch > > > We do application-level metrics aggregation in AppLevelTimelineCollector, but > the interval is hardcoded. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8303) YarnClient should contact TimelineReader for application/attempt/container report
[ https://issues.apache.org/jira/browse/YARN-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507252#comment-16507252 ] Abhishek Modi commented on YARN-8303: - Thanks [~rohithsharma] for filing this. I am assigning it to myself. If you have already started work on this feel free to assign back. Thanks. > YarnClient should contact TimelineReader for application/attempt/container > report > - > > Key: YARN-8303 > URL: https://issues.apache.org/jira/browse/YARN-8303 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Priority: Critical > > YarnClient get app/attempt/container information from RM. If RM doesn't have > then queried to ahsClient. When ATSv2 is only enabled, yarnClient will result > empty. > YarnClient is used by many users which result in empty information for > app/attempt/container report. > Proposal is to have adapter from yarn client so that app/attempt/container > reports can be generated from AHSv2Client which does REST API to > TimelineReader and get the entity and convert it into app/attempt/container > report. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8303) YarnClient should contact TimelineReader for application/attempt/container report
[ https://issues.apache.org/jira/browse/YARN-8303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi reassigned YARN-8303: --- Assignee: Abhishek Modi > YarnClient should contact TimelineReader for application/attempt/container > report > - > > Key: YARN-8303 > URL: https://issues.apache.org/jira/browse/YARN-8303 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Critical > > YarnClient get app/attempt/container information from RM. If RM doesn't have > then queried to ahsClient. When ATSv2 is only enabled, yarnClient will result > empty. > YarnClient is used by many users which result in empty information for > app/attempt/container report. > Proposal is to have adapter from yarn client so that app/attempt/container > reports can be generated from AHSv2Client which does REST API to > TimelineReader and get the entity and convert it into app/attempt/container > report. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7953) [GQ] Data structures for federation global queues calculations
[ https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507123#comment-16507123 ] Abhishek Modi commented on YARN-7953: - Thanks [~botong] for quick review. Attached v6 patch with the fixes. > [GQ] Data structures for federation global queues calculations > -- > > Key: YARN-7953 > URL: https://issues.apache.org/jira/browse/YARN-7953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7953-YARN-7402.v1.patch, > YARN-7953-YARN-7402.v2.patch, YARN-7953-YARN-7402.v3.patch, > YARN-7953-YARN-7402.v4.patch, YARN-7953-YARN-7402.v5.patch, > YARN-7953-YARN-7402.v6.patch, YARN-7953.v1.patch > > > This Jira tracks data structures and helper classes used by the core > algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7953) [GQ] Data structures for federation global queues calculations
[ https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-7953: Attachment: YARN-7953-YARN-7402.v6.patch > [GQ] Data structures for federation global queues calculations > -- > > Key: YARN-7953 > URL: https://issues.apache.org/jira/browse/YARN-7953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7953-YARN-7402.v1.patch, > YARN-7953-YARN-7402.v2.patch, YARN-7953-YARN-7402.v3.patch, > YARN-7953-YARN-7402.v4.patch, YARN-7953-YARN-7402.v5.patch, > YARN-7953-YARN-7402.v6.patch, YARN-7953.v1.patch > > > This Jira tracks data structures and helper classes used by the core > algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8180) YARN Federation has not implemented blacklist sub-cluster for AM routing
[
https://issues.apache.org/jira/browse/YARN-8180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507088#comment-16507088
]
Abhishek Modi commented on YARN-8180:
-
Thanks [~giovanni.fumarola] and [~shenyinjie].
I have submitted a patch removing this from documentation. [~giovanni.fumarola]
could you please review. Thanks.
> YARN Federation has not implemented blacklist sub-cluster for AM routing
>
>
> Key: YARN-8180
> URL: https://issues.apache.org/jira/browse/YARN-8180
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Reporter: Shen Yinjie
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-8180.001.patch
>
>
> Property "yarn.federation.blacklist-subclusters" is defined in
> yarn-fedeartion doc,but it has not been defined and implemented in Java code.
> In FederationClientInterceptor#submitApplication()
> {code:java}
> List blacklist = new ArrayList();
> for (int i = 0; i < numSubmitRetries; ++i) {
> SubClusterId subClusterId = policyFacade.getHomeSubcluster(
> request.getApplicationSubmissionContext(), blacklist);
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8180) YARN Federation has not implemented blacklist sub-cluster for AM routing
[
https://issues.apache.org/jira/browse/YARN-8180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Modi updated YARN-8180:
Attachment: YARN-8180.001.patch
> YARN Federation has not implemented blacklist sub-cluster for AM routing
>
>
> Key: YARN-8180
> URL: https://issues.apache.org/jira/browse/YARN-8180
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Reporter: Shen Yinjie
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-8180.001.patch
>
>
> Property "yarn.federation.blacklist-subclusters" is defined in
> yarn-fedeartion doc,but it has not been defined and implemented in Java code.
> In FederationClientInterceptor#submitApplication()
> {code:java}
> List blacklist = new ArrayList();
> for (int i = 0; i < numSubmitRetries; ++i) {
> SubClusterId subClusterId = policyFacade.getHomeSubcluster(
> request.getApplicationSubmissionContext(), blacklist);
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7953) [GQ] Data structures for federation global queues calculations
[ https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-7953: Attachment: YARN-7953-YARN-7402.v5.patch > [GQ] Data structures for federation global queues calculations > -- > > Key: YARN-7953 > URL: https://issues.apache.org/jira/browse/YARN-7953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7953-YARN-7402.v1.patch, > YARN-7953-YARN-7402.v2.patch, YARN-7953-YARN-7402.v3.patch, > YARN-7953-YARN-7402.v4.patch, YARN-7953-YARN-7402.v5.patch, YARN-7953.v1.patch > > > This Jira tracks data structures and helper classes used by the core > algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7953) [GQ] Data structures for federation global queues calculations
[
https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507027#comment-16507027
]
Abhishek Modi commented on YARN-7953:
-
Thanks [~botong] for the review. I have attached v5 patch fixing these.
{quote}line 160, super.clone() is redundant.
{quote}
super.clone needs to be called, else checkstyle check gives warning.
{quote}line 47, this is a tree of queues rather than a single queue right?
{quote}
It represents details about one queue and contains references of children
queues. Since it is inspired from Capacity Scheduler and Fair Scheduler queues
and that also has similar concept, in my opinion it's better to keep it as a
queue.
> [GQ] Data structures for federation global queues calculations
> --
>
> Key: YARN-7953
> URL: https://issues.apache.org/jira/browse/YARN-7953
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Carlo Curino
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-7953-YARN-7402.v1.patch,
> YARN-7953-YARN-7402.v2.patch, YARN-7953-YARN-7402.v3.patch,
> YARN-7953-YARN-7402.v4.patch, YARN-7953.v1.patch
>
>
> This Jira tracks data structures and helper classes used by the core
> algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8155) Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16505669#comment-16505669 ] Abhishek Modi commented on YARN-8155: - [~rohithsharma] [~vrushalic] could you please commit it if it looks good to you guys. Thanks. > Improve the logging in NMTimelinePublisher and TimelineCollectorWebService > -- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8155.001.patch, YARN-8155.002.patch, > YARN-8155.003.patch, YARN-8155.004.patch, YARN-8155.005.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8175) Add support for Node Labels in SLS
[ https://issues.apache.org/jira/browse/YARN-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504567#comment-16504567 ] Abhishek Modi commented on YARN-8175: - Thanks [~elgoiri]. I have added an updated patch with a new UT covering node labels. Could you please take a look. > Add support for Node Labels in SLS > -- > > Key: YARN-8175 > URL: https://issues.apache.org/jira/browse/YARN-8175 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8175.001.patch, YARN-8175.002.patch, > YARN-8175.003.patch, YARN-8175.004.patch, YARN-8175.005.patch, > YARN-8175.006.patch, YARN-8175.007.patch > > > Currently, SLS doesn't support node labels. With this jira, we are planning > to add support for node labels in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8175) Add support for Node Labels in SLS
[ https://issues.apache.org/jira/browse/YARN-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8175: Attachment: YARN-8175.007.patch > Add support for Node Labels in SLS > -- > > Key: YARN-8175 > URL: https://issues.apache.org/jira/browse/YARN-8175 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8175.001.patch, YARN-8175.002.patch, > YARN-8175.003.patch, YARN-8175.004.patch, YARN-8175.005.patch, > YARN-8175.006.patch, YARN-8175.007.patch > > > Currently, SLS doesn't support node labels. With this jira, we are planning > to add support for node labels in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8155) Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8155: Attachment: YARN-8155.005.patch > Improve the logging in NMTimelinePublisher and TimelineCollectorWebService > -- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8155.001.patch, YARN-8155.002.patch, > YARN-8155.003.patch, YARN-8155.004.patch, YARN-8155.005.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8155) Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
[
https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504133#comment-16504133
]
Abhishek Modi commented on YARN-8155:
-
Thanks [~vrushalic] for review.
{quote}In NMTimelinePublisher, do we think it might be a good idea to log the
exception in debug mode as is being done in TimelineServiceV2Publisher? On the
NM side, in case of issues, it may be good idea to see the stack trace?
{quote}
Sure I will add debug logs with full stack trace.
{quote}In TimelineServiceV2Publisher, looks like we are updating Exception to
IOException. So I am wondering what would happen if putEntity might throw some
other exception than IOException?
{quote}
Right now putEntity throws IOException only as managed Exception. That's why I
changed to IOException only. If you think, we should capture all Exceptions, I
can change it back to Exception.
> Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
> --
>
> Key: YARN-8155
> URL: https://issues.apache.org/jira/browse/YARN-8155
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-8155.001.patch, YARN-8155.002.patch,
> YARN-8155.003.patch, YARN-8155.004.patch
>
>
> We see that NM logs are filled with larger stack trace of NotFoundException
> if collector is removed from one of the NM and other NMs are still publishing
> the entities.
>
> This Jira is to improve the logging in NM so that we log with informative
> message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6904) [ATSv2] Fix findbugs warnings
[ https://issues.apache.org/jira/browse/YARN-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi resolved YARN-6904. - Resolution: Won't Fix These findbugs warning has already been fixed while merging that branch to trunk. Closing this. > [ATSv2] Fix findbugs warnings > - > > Key: YARN-6904 > URL: https://issues.apache.org/jira/browse/YARN-6904 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: YARN-5355 >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > > Many extant findbugs warnings are reported branch YARN-5355 > [Jenkins|https://issues.apache.org/jira/browse/YARN-6130?focusedCommentId=16105786=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16105786] > This need to be investigated and fix one by one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6904) [ATSv2] Fix findbugs warnings
[ https://issues.apache.org/jira/browse/YARN-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16503068#comment-16503068 ] Abhishek Modi commented on YARN-6904: - [~rohithsharma] these links are expired. Is there any way I can generate these warnings again? > [ATSv2] Fix findbugs warnings > - > > Key: YARN-6904 > URL: https://issues.apache.org/jira/browse/YARN-6904 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: YARN-5355 >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > > Many extant findbugs warnings are reported branch YARN-5355 > [Jenkins|https://issues.apache.org/jira/browse/YARN-6130?focusedCommentId=16105786=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16105786] > This need to be investigated and fix one by one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7953) [GQ] Data structures for federation global queues calculations
[ https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-7953: Attachment: YARN-7953-YARN-7402.v4.patch > [GQ] Data structures for federation global queues calculations > -- > > Key: YARN-7953 > URL: https://issues.apache.org/jira/browse/YARN-7953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7953-YARN-7402.v1.patch, > YARN-7953-YARN-7402.v2.patch, YARN-7953-YARN-7402.v3.patch, > YARN-7953-YARN-7402.v4.patch, YARN-7953.v1.patch > > > This Jira tracks data structures and helper classes used by the core > algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7953) [GQ] Data structures for federation global queues calculations
[ https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-7953: Attachment: YARN-7953-YARN-7402.v3.patch > [GQ] Data structures for federation global queues calculations > -- > > Key: YARN-7953 > URL: https://issues.apache.org/jira/browse/YARN-7953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7953-YARN-7402.v1.patch, > YARN-7953-YARN-7402.v2.patch, YARN-7953-YARN-7402.v3.patch, YARN-7953.v1.patch > > > This Jira tracks data structures and helper classes used by the core > algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8155) Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501753#comment-16501753 ] Abhishek Modi commented on YARN-8155: - Thanks [~rohithsharma]. Submitted an updated patch with the fix. > Improve the logging in NMTimelinePublisher and TimelineCollectorWebService > -- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8155.001.patch, YARN-8155.002.patch, > YARN-8155.003.patch, YARN-8155.004.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8155) Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8155: Attachment: YARN-8155.004.patch > Improve the logging in NMTimelinePublisher and TimelineCollectorWebService > -- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8155.001.patch, YARN-8155.002.patch, > YARN-8155.003.patch, YARN-8155.004.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8155) Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500529#comment-16500529 ] Abhishek Modi commented on YARN-8155: - The test that failed is unrelated to this patch. > Improve the logging in NMTimelinePublisher and TimelineCollectorWebService > -- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8155.001.patch, YARN-8155.002.patch, > YARN-8155.003.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6904) [ATSv2] Fix findbugs warnings
[ https://issues.apache.org/jira/browse/YARN-6904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi reassigned YARN-6904: --- Assignee: Abhishek Modi > [ATSv2] Fix findbugs warnings > - > > Key: YARN-6904 > URL: https://issues.apache.org/jira/browse/YARN-6904 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: YARN-5355 >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > > Many extant findbugs warnings are reported branch YARN-5355 > [Jenkins|https://issues.apache.org/jira/browse/YARN-6130?focusedCommentId=16105786=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16105786] > This need to be investigated and fix one by one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6121) Launch app collectors for unmanaged AMs'
[ https://issues.apache.org/jira/browse/YARN-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500274#comment-16500274 ] Abhishek Modi commented on YARN-6121: - Thanks [~varun_saxena] for filing this. I will start working on this. > Launch app collectors for unmanaged AMs' > > > Key: YARN-6121 > URL: https://issues.apache.org/jira/browse/YARN-6121 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Abhishek Modi >Priority: Major > > Currently an app collector is launched whenever an AM container is launched > on a NM. This means for an unmanaged AM, app collector is never launched. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6121) Launch app collectors for unmanaged AMs'
[ https://issues.apache.org/jira/browse/YARN-6121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi reassigned YARN-6121: --- Assignee: Abhishek Modi > Launch app collectors for unmanaged AMs' > > > Key: YARN-6121 > URL: https://issues.apache.org/jira/browse/YARN-6121 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Varun Saxena >Assignee: Abhishek Modi >Priority: Major > > Currently an app collector is launched whenever an AM container is launched > on a NM. This means for an unmanaged AM, app collector is never launched. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6989) Ensure timeline service v2 codebase gets UGI from HttpServletRequest in a consistent way
[
https://issues.apache.org/jira/browse/YARN-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Modi reassigned YARN-6989:
---
Assignee: Abhishek Modi
> Ensure timeline service v2 codebase gets UGI from HttpServletRequest in a
> consistent way
>
>
> Key: YARN-6989
> URL: https://issues.apache.org/jira/browse/YARN-6989
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Vrushali C
>Assignee: Abhishek Modi
>Priority: Major
>
> As noticed during discussions in YARN-6820, the webservices in timeline
> service v2 get the UGI created from the user obtained by invoking
> getRemoteUser on the HttpServletRequest .
> It will be good to use getUserPrincipal instead of invoking getRemoteUser on
> the HttpServletRequest.
> Filing jira to update the code.
> Per Java EE documentations for 6 and 7, the behavior around getRemoteUser and
> getUserPrincipal is listed at:
> http://docs.oracle.com/javaee/6/tutorial/doc/gjiie.html#bncba
> https://docs.oracle.com/javaee/7/tutorial/security-webtier003.htm
> {code}
> getRemoteUser, which determines the user name with which the client
> authenticated. The getRemoteUser method returns the name of the remote user
> (the caller) associated by the container with the request. If no user has
> been authenticated, this method returns null.
> getUserPrincipal, which determines the principal name of the current user and
> returns a java.security.Principal object. If no user has been authenticated,
> this method returns null. Calling the getName method on the Principal
> returned by getUserPrincipal returns the name of the remote user.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6931) Make the aggregation interval in AppLevelTimelineCollector configurable
[ https://issues.apache.org/jira/browse/YARN-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi reassigned YARN-6931: --- Assignee: Abhishek Modi > Make the aggregation interval in AppLevelTimelineCollector configurable > --- > > Key: YARN-6931 > URL: https://issues.apache.org/jira/browse/YARN-6931 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Abhishek Modi >Priority: Minor > > We do application-level metrics aggregation in AppLevelTimelineCollector, but > the interval is hardcoded. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8155) Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8155: Attachment: YARN-8155.003.patch > Improve the logging in NMTimelinePublisher and TimelineCollectorWebService > -- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8155.001.patch, YARN-8155.002.patch, > YARN-8155.003.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8155) Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
[
https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500234#comment-16500234
]
Abhishek Modi commented on YARN-8155:
-
{quote}Catching notfoundexception and converting into web application exception
changing return code. We should still retain return code not found right?
{quote}
I don't think we should pass WebApplicationException with Not Found code as
that is meant when web application doesn't handle that page request. IMO
WebApplicationException should be passed only with Internal Error only.
[~rohithsharma] thoughts?
> Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
> --
>
> Key: YARN-8155
> URL: https://issues.apache.org/jira/browse/YARN-8155
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-8155.001.patch, YARN-8155.002.patch
>
>
> We see that NM logs are filled with larger stack trace of NotFoundException
> if collector is removed from one of the NM and other NMs are still publishing
> the entities.
>
> This Jira is to improve the logging in NM so that we log with informative
> message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8155) Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16500230#comment-16500230 ] Abhishek Modi commented on YARN-8155: - Thanks [~rohithsharma] for quick review. I will submit the updated patch. > Improve the logging in NMTimelinePublisher and TimelineCollectorWebService > -- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8155.001.patch, YARN-8155.002.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8155) Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499717#comment-16499717 ] Abhishek Modi commented on YARN-8155: - [~rohithsharma] [~vrushalic] could you please review it. > Improve the logging in NMTimelinePublisher and TimelineCollectorWebService > -- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8155.001.patch, YARN-8155.002.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8155) Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8155: Attachment: YARN-8155.002.patch > Improve the logging in NMTimelinePublisher and TimelineCollectorWebService > -- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8155.001.patch, YARN-8155.002.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8155) Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8155: Attachment: YARN-8155.001.patch > Improve the logging in NMTimelinePublisher and TimelineCollectorWebService > -- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8155.001.patch > > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8175) Add support for Node Labels in SLS.
[ https://issues.apache.org/jira/browse/YARN-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498814#comment-16498814 ] Abhishek Modi commented on YARN-8175: - Thanks [~botong] for quick review. I have attached YARN-8175.006.patch fixing review comments. [~elgoiri] could you please review and commit it if it looks good. > Add support for Node Labels in SLS. > --- > > Key: YARN-8175 > URL: https://issues.apache.org/jira/browse/YARN-8175 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8175.001.patch, YARN-8175.002.patch, > YARN-8175.003.patch, YARN-8175.004.patch, YARN-8175.005.patch, > YARN-8175.006.patch > > > Currently, SLS doesn't support node labels. With this jira, we are planning > to add support for node labels in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8175) Add support for Node Labels in SLS.
[ https://issues.apache.org/jira/browse/YARN-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8175: Attachment: YARN-8175.006.patch > Add support for Node Labels in SLS. > --- > > Key: YARN-8175 > URL: https://issues.apache.org/jira/browse/YARN-8175 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8175.001.patch, YARN-8175.002.patch, > YARN-8175.003.patch, YARN-8175.004.patch, YARN-8175.005.patch, > YARN-8175.006.patch > > > Currently, SLS doesn't support node labels. With this jira, we are planning > to add support for node labels in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7953) [GQ] Data structures for federation global queues calculations
[ https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-7953: Attachment: YARN-7953-YARN-7402.v2.patch > [GQ] Data structures for federation global queues calculations > -- > > Key: YARN-7953 > URL: https://issues.apache.org/jira/browse/YARN-7953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7953-YARN-7402.v1.patch, > YARN-7953-YARN-7402.v2.patch, YARN-7953.v1.patch > > > This Jira tracks data structures and helper classes used by the core > algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7953) [GQ] Data structures for federation global queues calculations
[ https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495336#comment-16495336 ] Abhishek Modi commented on YARN-7953: - [~botong] [~subru] [~curino] [~kkaranasos] Could you please review YARN-7953-YARN-7402.v1.patch. > [GQ] Data structures for federation global queues calculations > -- > > Key: YARN-7953 > URL: https://issues.apache.org/jira/browse/YARN-7953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7953-YARN-7402.v1.patch, YARN-7953.v1.patch > > > This Jira tracks data structures and helper classes used by the core > algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7953) [GQ] Data structures for federation global queues calculations
[ https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-7953: Attachment: YARN-7953-YARN-7402.v1.patch > [GQ] Data structures for federation global queues calculations > -- > > Key: YARN-7953 > URL: https://issues.apache.org/jira/browse/YARN-7953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7953-YARN-7402.v1.patch, YARN-7953.v1.patch > > > This Jira tracks data structures and helper classes used by the core > algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8175) Add support for Node Labels in SLS.
[ https://issues.apache.org/jira/browse/YARN-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8175: Attachment: YARN-8175.005.patch > Add support for Node Labels in SLS. > --- > > Key: YARN-8175 > URL: https://issues.apache.org/jira/browse/YARN-8175 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8175.001.patch, YARN-8175.002.patch, > YARN-8175.003.patch, YARN-8175.004.patch, YARN-8175.005.patch > > > Currently, SLS doesn't support node labels. With this jira, we are planning > to add support for node labels in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8180) YARN Federation has not implemented blacklist sub-cluster for AM routing
[
https://issues.apache.org/jira/browse/YARN-8180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Modi reassigned YARN-8180:
---
Assignee: Abhishek Modi
> YARN Federation has not implemented blacklist sub-cluster for AM routing
>
>
> Key: YARN-8180
> URL: https://issues.apache.org/jira/browse/YARN-8180
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Reporter: Shen Yinjie
>Assignee: Abhishek Modi
>Priority: Major
>
> Property "yarn.federation.blacklist-subclusters" is defined in
> yarn-fedeartion doc,but it has not been defined and implemented in Java code.
> In FederationClientInterceptor#submitApplication()
> {code:java}
> List blacklist = new ArrayList();
> for (int i = 0; i < numSubmitRetries; ++i) {
> SubClusterId subClusterId = policyFacade.getHomeSubcluster(
> request.getApplicationSubmissionContext(), blacklist);
> {code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8175) Add support for Node Labels in SLS.
[ https://issues.apache.org/jira/browse/YARN-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8175: Attachment: YARN-8175.004.patch > Add support for Node Labels in SLS. > --- > > Key: YARN-8175 > URL: https://issues.apache.org/jira/browse/YARN-8175 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8175.001.patch, YARN-8175.002.patch, > YARN-8175.003.patch, YARN-8175.004.patch > > > Currently, SLS doesn't support node labels. With this jira, we are planning > to add support for node labels in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8175) Add support for Node Labels in SLS.
[ https://issues.apache.org/jira/browse/YARN-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8175: Attachment: YARN-8175.003.patch > Add support for Node Labels in SLS. > --- > > Key: YARN-8175 > URL: https://issues.apache.org/jira/browse/YARN-8175 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8175.001.patch, YARN-8175.002.patch, > YARN-8175.003.patch > > > Currently, SLS doesn't support node labels. With this jira, we are planning > to add support for node labels in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8163) Add support for Node Labels in opportunistic scheduling.
[ https://issues.apache.org/jira/browse/YARN-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464366#comment-16464366 ] Abhishek Modi commented on YARN-8163: - [~elgoiri] I have updated Unit tests to cover PB conversion. Could you please review it. > Add support for Node Labels in opportunistic scheduling. > > > Key: YARN-8163 > URL: https://issues.apache.org/jira/browse/YARN-8163 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8163.002.patch, YARN-8163.003.patch, > YARN-8163.004.patch, YARN-8163.patch > > > Currently, the Opportunistic Scheduler doesn't honor node labels constraints > and schedule containers based on locality and load constraints. This Jira is > to add support in opportunistic scheduling to honor node labels in resource > requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8163) Add support for Node Labels in opportunistic scheduling.
[ https://issues.apache.org/jira/browse/YARN-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8163: Attachment: YARN-8163.004.patch > Add support for Node Labels in opportunistic scheduling. > > > Key: YARN-8163 > URL: https://issues.apache.org/jira/browse/YARN-8163 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8163.002.patch, YARN-8163.003.patch, > YARN-8163.004.patch, YARN-8163.patch > > > Currently, the Opportunistic Scheduler doesn't honor node labels constraints > and schedule containers based on locality and load constraints. This Jira is > to add support in opportunistic scheduling to honor node labels in resource > requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8163) Add support for Node Labels in opportunistic scheduling.
[ https://issues.apache.org/jira/browse/YARN-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464091#comment-16464091 ] Abhishek Modi commented on YARN-8163: - Thanks [~elgoiri]. I will check that and if required add more unit tests to cover PB side. > Add support for Node Labels in opportunistic scheduling. > > > Key: YARN-8163 > URL: https://issues.apache.org/jira/browse/YARN-8163 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8163.002.patch, YARN-8163.003.patch, YARN-8163.patch > > > Currently, the Opportunistic Scheduler doesn't honor node labels constraints > and schedule containers based on locality and load constraints. This Jira is > to add support in opportunistic scheduling to honor node labels in resource > requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8163) Add support for Node Labels in opportunistic scheduling.
[
https://issues.apache.org/jira/browse/YARN-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462874#comment-16462874
]
Abhishek Modi commented on YARN-8163:
-
Submitted v3 patch fixing CR comments. [~giovanni.fumarola] regarding
{quote}In _OpportunisticContainerAllocatorAMService#allocate_
_partitionedAsks.getOpportunistic()_ can be null. Please add null pointer check.
{quote}
I think it should never be null as we are always initializing opportunistic
with an empty ArrayList.
> Add support for Node Labels in opportunistic scheduling.
>
>
> Key: YARN-8163
> URL: https://issues.apache.org/jira/browse/YARN-8163
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-8163.002.patch, YARN-8163.003.patch, YARN-8163.patch
>
>
> Currently, the Opportunistic Scheduler doesn't honor node labels constraints
> and schedule containers based on locality and load constraints. This Jira is
> to add support in opportunistic scheduling to honor node labels in resource
> requests.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8163) Add support for Node Labels in opportunistic scheduling.
[ https://issues.apache.org/jira/browse/YARN-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8163: Attachment: YARN-8163.003.patch > Add support for Node Labels in opportunistic scheduling. > > > Key: YARN-8163 > URL: https://issues.apache.org/jira/browse/YARN-8163 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8163.002.patch, YARN-8163.003.patch, YARN-8163.patch > > > Currently, the Opportunistic Scheduler doesn't honor node labels constraints > and schedule containers based on locality and load constraints. This Jira is > to add support in opportunistic scheduling to honor node labels in resource > requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8163) Add support for Node Labels in opportunistic scheduling.
[ https://issues.apache.org/jira/browse/YARN-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461772#comment-16461772 ] Abhishek Modi commented on YARN-8163: - Thanks [~giovanni.fumarola] for review. I will address the review comments and submit an updated patch. > Add support for Node Labels in opportunistic scheduling. > > > Key: YARN-8163 > URL: https://issues.apache.org/jira/browse/YARN-8163 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8163.002.patch, YARN-8163.patch > > > Currently, the Opportunistic Scheduler doesn't honor node labels constraints > and schedule containers based on locality and load constraints. This Jira is > to add support in opportunistic scheduling to honor node labels in resource > requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8175) Add support for Node Labels in SLS.
[ https://issues.apache.org/jira/browse/YARN-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447258#comment-16447258 ] Abhishek Modi commented on YARN-8175: - [~elgoiri] and [~giovanni.fumarola] could you please review it. Thanks > Add support for Node Labels in SLS. > --- > > Key: YARN-8175 > URL: https://issues.apache.org/jira/browse/YARN-8175 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8175.001.patch, YARN-8175.002.patch > > > Currently, SLS doesn't support node labels. With this jira, we are planning > to add support for node labels in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8175) Add support for Node Labels in SLS.
[ https://issues.apache.org/jira/browse/YARN-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8175: Attachment: YARN-8175.002.patch > Add support for Node Labels in SLS. > --- > > Key: YARN-8175 > URL: https://issues.apache.org/jira/browse/YARN-8175 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8175.001.patch, YARN-8175.002.patch > > > Currently, SLS doesn't support node labels. With this jira, we are planning > to add support for node labels in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8175) Add support for Node Labels in SLS.
[ https://issues.apache.org/jira/browse/YARN-8175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8175: Attachment: YARN-8175.001.patch > Add support for Node Labels in SLS. > --- > > Key: YARN-8175 > URL: https://issues.apache.org/jira/browse/YARN-8175 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8175.001.patch > > > Currently, SLS doesn't support node labels. With this jira, we are planning > to add support for node labels in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8137) Parallelize node addition in SLS
[ https://issues.apache.org/jira/browse/YARN-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16446154#comment-16446154 ] Abhishek Modi commented on YARN-8137: - Thanks [~elgoiri] and [~giovanni.fumarola]. > Parallelize node addition in SLS > > > Key: YARN-8137 > URL: https://issues.apache.org/jira/browse/YARN-8137 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-8137.001.patch, YARN-8137.002.patch, > YARN-8137.003.patch > > > Right now, nodes are added sequentially and it can take a long time if there > are large number of nodes. With this change nodes will be added in parallel > and thus reduce the node addition time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8137) Parallelize node addition in SLS
[ https://issues.apache.org/jira/browse/YARN-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445388#comment-16445388 ] Abhishek Modi commented on YARN-8137: - Added a new patch after rebasing with trunk. cc [~elgoiri] > Parallelize node addition in SLS > > > Key: YARN-8137 > URL: https://issues.apache.org/jira/browse/YARN-8137 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8137.001.patch, YARN-8137.002.patch, > YARN-8137.003.patch > > > Right now, nodes are added sequentially and it can take a long time if there > are large number of nodes. With this change nodes will be added in parallel > and thus reduce the node addition time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8137) Parallelize node addition in SLS
[ https://issues.apache.org/jira/browse/YARN-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8137: Attachment: YARN-8137.003.patch > Parallelize node addition in SLS > > > Key: YARN-8137 > URL: https://issues.apache.org/jira/browse/YARN-8137 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8137.001.patch, YARN-8137.002.patch, > YARN-8137.003.patch > > > Right now, nodes are added sequentially and it can take a long time if there > are large number of nodes. With this change nodes will be added in parallel > and thus reduce the node addition time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8137) Parallelize node addition in SLS
[ https://issues.apache.org/jira/browse/YARN-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443003#comment-16443003 ] Abhishek Modi commented on YARN-8137: - [~elgoiri] Yes existing tests cover number of generated nodes is correct. There is a check for waitForNodesRunning which will stuck infinitely if number of running nodes doesn't match number of generated nodes. > Parallelize node addition in SLS > > > Key: YARN-8137 > URL: https://issues.apache.org/jira/browse/YARN-8137 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8137.001.patch, YARN-8137.002.patch > > > Right now, nodes are added sequentially and it can take a long time if there > are large number of nodes. With this change nodes will be added in parallel > and thus reduce the node addition time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8175) Add support for Node Labels in SLS.
Abhishek Modi created YARN-8175: --- Summary: Add support for Node Labels in SLS. Key: YARN-8175 URL: https://issues.apache.org/jira/browse/YARN-8175 Project: Hadoop YARN Issue Type: Sub-task Reporter: Abhishek Modi Assignee: Abhishek Modi Currently, SLS doesn't support node labels. With this jira, we are planning to add support for node labels in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8137) Parallelize node addition in SLS
[ https://issues.apache.org/jira/browse/YARN-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442807#comment-16442807 ] Abhishek Modi commented on YARN-8137: - [~elgoiri] Could you please review and commit it. Thanks > Parallelize node addition in SLS > > > Key: YARN-8137 > URL: https://issues.apache.org/jira/browse/YARN-8137 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8137.001.patch, YARN-8137.002.patch > > > Right now, nodes are added sequentially and it can take a long time if there > are large number of nodes. With this change nodes will be added in parallel > and thus reduce the node addition time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8134) Support specifying node resources in SLS
[ https://issues.apache.org/jira/browse/YARN-8134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440708#comment-16440708 ] Abhishek Modi commented on YARN-8134: - Thanks [~giovanni.fumarola] for review. I only kept only one node with explicit resources in nodes-with-resources to verify mix of both nodes-with-resources and nodes without explicit resources. > Support specifying node resources in SLS > > > Key: YARN-8134 > URL: https://issues.apache.org/jira/browse/YARN-8134 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8134.002.patch, YARN-8134.003.patch, YARN-8134.patch > > > At present, all nodes have same resources in SLS. We need to add capability > to add different resources to different nodes in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8137) Parallelize node addition in SLS
[ https://issues.apache.org/jira/browse/YARN-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8137: Attachment: YARN-8137.002.patch > Parallelize node addition in SLS > > > Key: YARN-8137 > URL: https://issues.apache.org/jira/browse/YARN-8137 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8137.001.patch, YARN-8137.002.patch > > > Right now, nodes are added sequentially and it can take a long time if there > are large number of nodes. With this change nodes will be added in parallel > and thus reduce the node addition time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8137) Parallelize node addition in SLS
[
https://issues.apache.org/jira/browse/YARN-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440700#comment-16440700
]
Abhishek Modi commented on YARN-8137:
-
Thanks [~giovanni.fumarola] for reviewing the patch.
{quote}You have this instruction {{int threadPoolSize = Math.max(poolSize,
10);}} instead of 10 you should have
{{SLSConfiguration.RUNNER_POOL_SIZE_DEFAULT;}}
{quote}
Will fix it.
{quote}{{executorService.awaitTermination(1, TimeUnit.HOURS);}} I think the
awaiting time is too long. How did you come up with this value?
{quote}
I tried with 5000 nodes and it was taking around similar time without this fix.
I will reduce it to 10 minutes with this fix.
> Parallelize node addition in SLS
>
>
> Key: YARN-8137
> URL: https://issues.apache.org/jira/browse/YARN-8137
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-8137.001.patch
>
>
> Right now, nodes are added sequentially and it can take a long time if there
> are large number of nodes. With this change nodes will be added in parallel
> and thus reduce the node addition time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8163) Add support for Node Labels in opportunistic scheduling.
[ https://issues.apache.org/jira/browse/YARN-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8163: Attachment: YARN-8163.002.patch > Add support for Node Labels in opportunistic scheduling. > > > Key: YARN-8163 > URL: https://issues.apache.org/jira/browse/YARN-8163 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8163.002.patch, YARN-8163.patch > > > Currently, the Opportunistic Scheduler doesn't honor node labels constraints > and schedule containers based on locality and load constraints. This Jira is > to add support in opportunistic scheduling to honor node labels in resource > requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8163) Add support for Node Labels in opportunistic scheduling.
[ https://issues.apache.org/jira/browse/YARN-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438630#comment-16438630 ] Abhishek Modi commented on YARN-8163: - Submitting initial patch. > Add support for Node Labels in opportunistic scheduling. > > > Key: YARN-8163 > URL: https://issues.apache.org/jira/browse/YARN-8163 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8163.patch > > > Currently, the Opportunistic Scheduler doesn't honor node labels constraints > and schedule containers based on locality and load constraints. This Jira is > to add support in opportunistic scheduling to honor node labels in resource > requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8163) Add support for Node Labels in opportunistic scheduling.
[ https://issues.apache.org/jira/browse/YARN-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8163: Attachment: YARN-8163.patch > Add support for Node Labels in opportunistic scheduling. > > > Key: YARN-8163 > URL: https://issues.apache.org/jira/browse/YARN-8163 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8163.patch > > > Currently, the Opportunistic Scheduler doesn't honor node labels constraints > and schedule containers based on locality and load constraints. This Jira is > to add support in opportunistic scheduling to honor node labels in resource > requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8163) Add support for Node Labels in opportunistic scheduling.
[ https://issues.apache.org/jira/browse/YARN-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438628#comment-16438628 ] Abhishek Modi commented on YARN-8163: - cc [~asuresh] > Add support for Node Labels in opportunistic scheduling. > > > Key: YARN-8163 > URL: https://issues.apache.org/jira/browse/YARN-8163 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > > Currently, the Opportunistic Scheduler doesn't honor node labels constraints > and schedule containers based on locality and load constraints. This Jira is > to add support in opportunistic scheduling to honor node labels in resource > requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8163) Add support for Node Labels in opportunistic scheduling.
Abhishek Modi created YARN-8163: --- Summary: Add support for Node Labels in opportunistic scheduling. Key: YARN-8163 URL: https://issues.apache.org/jira/browse/YARN-8163 Project: Hadoop YARN Issue Type: Improvement Reporter: Abhishek Modi Assignee: Abhishek Modi Currently, the Opportunistic Scheduler doesn't honor node labels constraints and schedule containers based on locality and load constraints. This Jira is to add support in opportunistic scheduling to honor node labels in resource requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7930) Add configuration to initialize RM with configured labels.
[ https://issues.apache.org/jira/browse/YARN-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16436859#comment-16436859 ] Abhishek Modi commented on YARN-7930: - Thanks [~asuresh] and [~leftnoteasy] for review. in case of RM restart, it will reinitialize the labels from the config and then will rerun the state from node label store. So in the case you mentioned, after RM restart, labels in the RM would be b, c and d. This should also be desired behavior. I can create another config to delete the previously stored state and that flag can be used to restart RM with fresh config. Thoughts? > Add configuration to initialize RM with configured labels. > -- > > Key: YARN-7930 > URL: https://issues.apache.org/jira/browse/YARN-7930 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7930.001.patch, YARN-7930.002.patch, > YARN-7930.003.patch, YARN-7930.004.patch, YARN-7930.005.patch > > > At present, the only way to create labels is using admin API. Sometimes, > there is a requirement to start the cluster with pre-configured node labels. > This Jira introduces yarn configurations to start RM with predefined node > labels. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8155) Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435040#comment-16435040 ] Abhishek Modi commented on YARN-8155: - [~rohithsharma] I would like to work on this and assigning it to myself. Please let me know if you have already started working on this. > Improve the logging in NMTimelinePublisher and TimelineCollectorWebService > -- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Priority: Major > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8155) Improve the logging in NMTimelinePublisher and TimelineCollectorWebService
[ https://issues.apache.org/jira/browse/YARN-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi reassigned YARN-8155: --- Assignee: Abhishek Modi > Improve the logging in NMTimelinePublisher and TimelineCollectorWebService > -- > > Key: YARN-8155 > URL: https://issues.apache.org/jira/browse/YARN-8155 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Major > > We see that NM logs are filled with larger stack trace of NotFoundException > if collector is removed from one of the NM and other NMs are still publishing > the entities. > > This Jira is to improve the logging in NM so that we log with informative > message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8134) Support specifying node resources in SLS
[ https://issues.apache.org/jira/browse/YARN-8134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8134: Attachment: YARN-8134.003.patch > Support specifying node resources in SLS > > > Key: YARN-8134 > URL: https://issues.apache.org/jira/browse/YARN-8134 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8134.002.patch, YARN-8134.003.patch, YARN-8134.patch > > > At present, all nodes have same resources in SLS. We need to add capability > to add different resources to different nodes in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8139) Skip node hostname resolution when running SLS.
Abhishek Modi created YARN-8139: --- Summary: Skip node hostname resolution when running SLS. Key: YARN-8139 URL: https://issues.apache.org/jira/browse/YARN-8139 Project: Hadoop YARN Issue Type: Bug Reporter: Abhishek Modi Assignee: Abhishek Modi Currently depending on the time taken in resolution of hostname, metrics of SLS gets skewed. To avoid this, in this fix we are introducing a flag which can be used to disable hostname resolutions. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8134) Support specifying node resources in SLS
[ https://issues.apache.org/jira/browse/YARN-8134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8134: Attachment: YARN-8134.002.patch > Support specifying node resources in SLS > > > Key: YARN-8134 > URL: https://issues.apache.org/jira/browse/YARN-8134 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8134.002.patch, YARN-8134.patch > > > At present, all nodes have same resources in SLS. We need to add capability > to add different resources to different nodes in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7930) Add configuration to initialize RM with configured labels.
[ https://issues.apache.org/jira/browse/YARN-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-7930: Attachment: YARN-7930.005.patch > Add configuration to initialize RM with configured labels. > -- > > Key: YARN-7930 > URL: https://issues.apache.org/jira/browse/YARN-7930 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7930.001.patch, YARN-7930.002.patch, > YARN-7930.003.patch, YARN-7930.004.patch, YARN-7930.005.patch > > > At present, the only way to create labels is using admin API. Sometimes, > there is a requirement to start the cluster with pre-configured node labels. > This Jira introduces yarn configurations to start RM with predefined node > labels. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7930) Add configuration to initialize RM with configured labels.
[ https://issues.apache.org/jira/browse/YARN-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-7930: Attachment: YARN-7930.004.patch > Add configuration to initialize RM with configured labels. > -- > > Key: YARN-7930 > URL: https://issues.apache.org/jira/browse/YARN-7930 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7930.001.patch, YARN-7930.002.patch, > YARN-7930.003.patch, YARN-7930.004.patch > > > At present, the only way to create labels is using admin API. Sometimes, > there is a requirement to start the cluster with pre-configured node labels. > This Jira introduces yarn configurations to start RM with predefined node > labels. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8137) Parallelize node addition in SLS
[ https://issues.apache.org/jira/browse/YARN-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8137: Issue Type: Sub-task (was: Bug) Parent: YARN-5065 > Parallelize node addition in SLS > > > Key: YARN-8137 > URL: https://issues.apache.org/jira/browse/YARN-8137 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8137.001.patch > > > Right now, nodes are added sequentially and it can take a long time if there > are large number of nodes. With this change nodes will be added in parallel > and thus reduce the node addition time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8137) Parallelize node addition in SLS
[ https://issues.apache.org/jira/browse/YARN-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8137: Attachment: YARN-8137.001.patch > Parallelize node addition in SLS > > > Key: YARN-8137 > URL: https://issues.apache.org/jira/browse/YARN-8137 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8137.001.patch > > > Right now, nodes are added sequentially and it can take a long time if there > are large number of nodes. With this change nodes will be added in parallel > and thus reduce the node addition time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8137) Parallelize node addition in SLS
Abhishek Modi created YARN-8137: --- Summary: Parallelize node addition in SLS Key: YARN-8137 URL: https://issues.apache.org/jira/browse/YARN-8137 Project: Hadoop YARN Issue Type: Bug Reporter: Abhishek Modi Assignee: Abhishek Modi Right now, nodes are added sequentially and it can take a long time if there are large number of nodes. With this change nodes will be added in parallel and thus reduce the node addition time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7930) Add configuration to initialize RM with configured labels.
[ https://issues.apache.org/jira/browse/YARN-7930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-7930: Attachment: YARN-7930.003.patch > Add configuration to initialize RM with configured labels. > -- > > Key: YARN-7930 > URL: https://issues.apache.org/jira/browse/YARN-7930 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7930.001.patch, YARN-7930.002.patch, > YARN-7930.003.patch > > > At present, the only way to create labels is using admin API. Sometimes, > there is a requirement to start the cluster with pre-configured node labels. > This Jira introduces yarn configurations to start RM with predefined node > labels. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8134) Support specifying node resources in SLS
[ https://issues.apache.org/jira/browse/YARN-8134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8134: Attachment: YARN-8134.patch > Support specifying node resources in SLS > > > Key: YARN-8134 > URL: https://issues.apache.org/jira/browse/YARN-8134 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8134.patch > > > At present, all nodes have same resources in SLS. We need to add capability > to add different resources to different nodes in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
