[jira] [Commented] (YARN-11243) Upgrade JUnit from 4 to 5 in hadoop-yarn-applications
[ https://issues.apache.org/jira/browse/YARN-11243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817541#comment-17817541 ] Ashutosh Gupta commented on YARN-11243: --- Hi [~BilwaST], Thanks for asking. I am working on it. I am have few PRs pending to be reviewed for other modules as well. I will continue to work on this. > Upgrade JUnit from 4 to 5 in hadoop-yarn-applications > - > > Key: YARN-11243 > URL: https://issues.apache.org/jira/browse/YARN-11243 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > > Upgrade JUnit from 4 to 5 in hadoop-yarn-applications -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11463) Node Labels root directory creation doesn't have a retry logic
[ https://issues.apache.org/jira/browse/YARN-11463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta reassigned YARN-11463: - Assignee: Ashutosh Gupta > Node Labels root directory creation doesn't have a retry logic > -- > > Key: YARN-11463 > URL: https://issues.apache.org/jira/browse/YARN-11463 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Benjamin Teke >Assignee: Ashutosh Gupta >Priority: Major > > When CS is initialized, it'll [try to create the configured node labels root > dir|https://github.com/apache/hadoop/blob/7169ec450957e5602775c3cd6fe1bf0b95773dfb/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/nodelabels/store/AbstractFSNodeStore.java#L69]. > This however doesn't implement any kind of retry logic (in contrast to the > RM FS state store or ZK state store), hence if the distributed file system is > unavailable at the exact moment CS tries to start it'll fail. A retry logic > could be implemented to improve the robustness of the startup process. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11364) Docker Container to accept docker Image name with sha256 digest
[
https://issues.apache.org/jira/browse/YARN-11364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta updated YARN-11364:
--
Summary: Docker Container to accept docker Image name with sha256 digest
(was: Docker Container to accept docker Image name with sha digest)
> Docker Container to accept docker Image name with sha256 digest
> ---
>
> Key: YARN-11364
> URL: https://issues.apache.org/jira/browse/YARN-11364
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
>
> Docker Container to accept docker Image name with sha digest.
> While using docker and passing the using sha digest as docker image. It
> should be valid and not throw error. eg -
> {code:java}
> 123456789123.dkr.ecr.us-east-1.amazonaws.com/emr-docker-examples:pyspark-example@sha256:f1d4ae3f7261a72e98c6ebefe9985cf10a0ea5bd762585a43e0700ed99863807{code}
> should be valid and not throw error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11364) Docker Container to accept docker Image name with sha256 digest
[
https://issues.apache.org/jira/browse/YARN-11364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta updated YARN-11364:
--
Description:
Docker Container to accept docker Image name with sha256 digest.
While using docker and passing the using sha digest as docker image. It should
be valid and not throw error. eg -
{code:java}
123456789123.dkr.ecr.us-east-1.amazonaws.com/emr-docker-examples:pyspark-example@sha256:f1d4ae3f7261a72e98c6ebefe9985cf10a0ea5bd762585a43e0700ed99863807{code}
should be valid and not throw error.
was:
Docker Container to accept docker Image name with sha digest.
While using docker and passing the using sha digest as docker image. It should
be valid and not throw error. eg -
{code:java}
123456789123.dkr.ecr.us-east-1.amazonaws.com/emr-docker-examples:pyspark-example@sha256:f1d4ae3f7261a72e98c6ebefe9985cf10a0ea5bd762585a43e0700ed99863807{code}
should be valid and not throw error.
> Docker Container to accept docker Image name with sha256 digest
> ---
>
> Key: YARN-11364
> URL: https://issues.apache.org/jira/browse/YARN-11364
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
>
> Docker Container to accept docker Image name with sha256 digest.
> While using docker and passing the using sha digest as docker image. It
> should be valid and not throw error. eg -
> {code:java}
> 123456789123.dkr.ecr.us-east-1.amazonaws.com/emr-docker-examples:pyspark-example@sha256:f1d4ae3f7261a72e98c6ebefe9985cf10a0ea5bd762585a43e0700ed99863807{code}
> should be valid and not throw error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11364) Docker Container to accept docker Image name with sha digest
[
https://issues.apache.org/jira/browse/YARN-11364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta updated YARN-11364:
--
Description:
Docker Container to accept docker Image name with sha digest.
While using docker and passing the using sha digest as docker image. It should
be valid and not throw error. eg -
{code:java}
123456789123.dkr.ecr.us-east-1.amazonaws.com/emr-docker-examples:pyspark-example@sha256:f1d4ae3f7261a72e98c6ebefe9985cf10a0ea5bd762585a43e0700ed99863807{code}
should be valid and not throw error.
was:
Docker Container to accept docker Image name with sha digest.
While using docker and passing the using sha digest as docker image. It should
be valid and not throw error
eg -
```
123456789123.dkr.ecr.us-east-1.amazonaws.com/emr-docker-examples:pyspark-example@sha256:f1d4ae3f7261a72e98c6ebefe9985cf10a0ea5bd762585a43e0700ed99863807
```
should be valid and not throw error.
> Docker Container to accept docker Image name with sha digest
>
>
> Key: YARN-11364
> URL: https://issues.apache.org/jira/browse/YARN-11364
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
>
> Docker Container to accept docker Image name with sha digest.
> While using docker and passing the using sha digest as docker image. It
> should be valid and not throw error. eg -
> {code:java}
> 123456789123.dkr.ecr.us-east-1.amazonaws.com/emr-docker-examples:pyspark-example@sha256:f1d4ae3f7261a72e98c6ebefe9985cf10a0ea5bd762585a43e0700ed99863807{code}
> should be valid and not throw error.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11364) Docker Container to accept docker Image name with sha digest
[ https://issues.apache.org/jira/browse/YARN-11364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated YARN-11364: -- Description: Docker Container to accept docker Image name with sha digest. While using docker and passing the using sha digest as docker image. It should be valid and not throw error eg - ``` 123456789123.dkr.ecr.us-east-1.amazonaws.com/emr-docker-examples:pyspark-example@sha256:f1d4ae3f7261a72e98c6ebefe9985cf10a0ea5bd762585a43e0700ed99863807 ``` should be valid and not throw error. was:While using docker and passing the sha 256 of the image as an image uri. The sha256 is not getting picked up and it throws invalid image uri > Docker Container to accept docker Image name with sha digest > > > Key: YARN-11364 > URL: https://issues.apache.org/jira/browse/YARN-11364 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > > Docker Container to accept docker Image name with sha digest. > While using docker and passing the using sha digest as docker image. It > should be valid and not throw error > eg - > ``` > 123456789123.dkr.ecr.us-east-1.amazonaws.com/emr-docker-examples:pyspark-example@sha256:f1d4ae3f7261a72e98c6ebefe9985cf10a0ea5bd762585a43e0700ed99863807 > ``` > should be valid and not throw error. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11364) Docker Container to accept docker Image name with sha digest
[ https://issues.apache.org/jira/browse/YARN-11364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated YARN-11364: -- Summary: Docker Container to accept docker Image name with sha digest (was: Docker Container to accept docker Image name with sha256) > Docker Container to accept docker Image name with sha digest > > > Key: YARN-11364 > URL: https://issues.apache.org/jira/browse/YARN-11364 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > > While using docker and passing the sha 256 of the image as an image uri. The > sha256 is not getting picked up and it throws invalid image uri -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11364) Docker Container to accept docker Image name with sha256
[ https://issues.apache.org/jira/browse/YARN-11364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated YARN-11364: -- Description: While using docker and passing the sha 256 of the image as an image uri. The sha256 is not getting picked up and it throws invalid image uri (was: While using docker and passing the sha 256 of the image as an image uri. The sha256 is not getting picked up and emr throws invalid image uri) > Docker Container to accept docker Image name with sha256 > > > Key: YARN-11364 > URL: https://issues.apache.org/jira/browse/YARN-11364 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > > While using docker and passing the sha 256 of the image as an image uri. The > sha256 is not getting picked up and it throws invalid image uri -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11364) Docker Container to accept docker Image name with sha256
[ https://issues.apache.org/jira/browse/YARN-11364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated YARN-11364: -- Summary: Docker Container to accept docker Image name with sha256 (was: Docker Container to accept Docker Image name with sha256) > Docker Container to accept docker Image name with sha256 > > > Key: YARN-11364 > URL: https://issues.apache.org/jira/browse/YARN-11364 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > > While using docker and passing the sha 256 of the image as an image uri. The > sha256 is not getting picked up and emr throws invalid image uri -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11364) Docker Container to accept Docker Image name with sha256
[ https://issues.apache.org/jira/browse/YARN-11364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated YARN-11364: -- Summary: Docker Container to accept Docker Image name with sha256 (was: Docker to accept Image name with sha256) > Docker Container to accept Docker Image name with sha256 > > > Key: YARN-11364 > URL: https://issues.apache.org/jira/browse/YARN-11364 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > > While using docker and passing the sha 256 of the image as an image uri. The > sha256 is not getting picked up and emr throws invalid image uri -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11364) Docker to accept Image name with sha256
Ashutosh Gupta created YARN-11364: - Summary: Docker to accept Image name with sha256 Key: YARN-11364 URL: https://issues.apache.org/jira/browse/YARN-11364 Project: Hadoop YARN Issue Type: Improvement Components: yarn Reporter: Ashutosh Gupta Assignee: Ashutosh Gupta While using docker and passing the sha 256 of the image as an image uri. The sha256 is not getting picked up and emr throws invalid image uri -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11363) Remove unused TimelineVersionWatcher and TimelineVersion from hadoop-yarn-server-tests
Ashutosh Gupta created YARN-11363: - Summary: Remove unused TimelineVersionWatcher and TimelineVersion from hadoop-yarn-server-tests Key: YARN-11363 URL: https://issues.apache.org/jira/browse/YARN-11363 Project: Hadoop YARN Issue Type: Improvement Components: test, yarn Affects Versions: 3.3.4, 3.3.3 Reporter: Ashutosh Gupta Assignee: Ashutosh Gupta Verify and remove unused TimelineVersionWatcher and TimelineVersion from hadoop-yarn-server-tests -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11353) Change debug logs in FSDownload.java to info logs for better escalations debugging
[ https://issues.apache.org/jira/browse/YARN-11353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624563#comment-17624563 ] Ashutosh Gupta commented on YARN-11353: --- Thanks [~aajisaka] . Welcome [~SanjayKumarSahu] > Change debug logs in FSDownload.java to info logs for better escalations > debugging > -- > > Key: YARN-11353 > URL: https://issues.apache.org/jira/browse/YARN-11353 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, yarn >Affects Versions: 3.4.0 >Reporter: Sanjay Kumar Sahu >Assignee: Sanjay Kumar Sahu >Priority: Major > Labels: pull-request-available > > AM was stuck in Preparing Local resources step and it timed out and never > started the driver. This happened in one of the customer's cluster and got > resolved when this cluster got deleted and the customer started using another > cluster . The logs were not enough to look into the issue. Adding more info > logs will help to understand when did the download of the files start and > when did it end, or whether it actually reached that step like adding the > containerId here to know who is downloading. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11353) Change debug logs in FSDownload.java to info logs for better escalations debugging
[ https://issues.apache.org/jira/browse/YARN-11353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17624379#comment-17624379 ] Ashutosh Gupta commented on YARN-11353: --- [~aajisaka] [~prabhujoseph] - Can you please add [~SanjayKumarSahu] as contributor. Thanks > Change debug logs in FSDownload.java to info logs for better escalations > debugging > -- > > Key: YARN-11353 > URL: https://issues.apache.org/jira/browse/YARN-11353 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, yarn >Affects Versions: 3.4.0 >Reporter: Sanjay Kumar Sahu >Priority: Major > Labels: pull-request-available > > AM was stuck in Preparing Local resources step and it timed out and never > started the driver. This happened in one of the customer's cluster and got > resolved when this cluster got deleted and the customer started using another > cluster . The logs were not enough to look into the issue. Adding more info > logs will help to understand when did the download of the files start and > when did it end, or whether it actually reached that step like adding the > containerId here to know who is downloading. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11360) Add number of decommissioning nodes to YARN cluster metrics.
[
https://issues.apache.org/jira/browse/YARN-11360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17622420#comment-17622420
]
Ashutosh Gupta commented on YARN-11360:
---
Thanks [~cnauroth], I think you are not removing or renaming command line
options. Adding a new one without altering the order of previous arguments
should be okay.
> Add number of decommissioning nodes to YARN cluster metrics.
>
>
> Key: YARN-11360
> URL: https://issues.apache.org/jira/browse/YARN-11360
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, resourcemanager
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
>Priority: Major
> Labels: pull-request-available
>
> YARN cluster metrics expose counts of NodeManagers in various states
> including active and decommissioned. However, these metrics don't expose
> NodeManagers that are currently in the process of decommissioning. This can
> look a little spooky to a consumer of these metrics. First, the node drops
> out of the active count, so it seems like a node just vanished. Then, later
> (possibly hours later with consideration of graceful decommission), it comes
> back into existence in the decommissioned count.
> This issue tracks adding the decommissioning count to the metrics
> ResourceManager RPC. This also enables exposing it in the {{yarn top}}
> output. This metric is already visible through the REST API, so there isn't
> any change required there.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11353) Change debug logs in FSDownload.java to info logs for better escalations debugging
[ https://issues.apache.org/jira/browse/YARN-11353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated YARN-11353: -- Issue Type: Improvement (was: New Feature) > Change debug logs in FSDownload.java to info logs for better escalations > debugging > -- > > Key: YARN-11353 > URL: https://issues.apache.org/jira/browse/YARN-11353 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager, yarn >Reporter: Sanjay Kumar Sahu >Priority: Major > > AM was stuck in Preparing Local resources step and it timed out and never > started the driver. This happened in one of the customer's cluster and got > resolved when this cluster got deleted and the customer started using another > cluster . The logs were not enough to look into the issue. Adding more info > logs will help to understand when did the download of the files start and > when did it end, or whether it actually reached that step like adding the > containerId here to know who is downloading. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11353) Change debug logs in FSDownload.java to info logs for better escalations debugging
[ https://issues.apache.org/jira/browse/YARN-11353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17620306#comment-17620306 ] Ashutosh Gupta commented on YARN-11353: --- Thanks [~SanjayKumarSahu] for reporting this, please update the Affects Version/s sections as well. > Change debug logs in FSDownload.java to info logs for better escalations > debugging > -- > > Key: YARN-11353 > URL: https://issues.apache.org/jira/browse/YARN-11353 > Project: Hadoop YARN > Issue Type: New Feature > Components: resourcemanager, yarn >Reporter: Sanjay Kumar Sahu >Priority: Major > > AM was stuck in Preparing Local resources step and it timed out and never > started the driver. This happened in one of the customer's cluster and got > resolved when this cluster got deleted and the customer started using another > cluster . The logs were not enough to look into the issue. Adding more info > logs will help to understand when did the download of the files start and > when did it end, or whether it actually reached that step like adding the > containerId here to know who is downloading. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11259) Upgrade JUnit from 4 to 5 in hadoop-yarn-server-applicationhistoryservice
[ https://issues.apache.org/jira/browse/YARN-11259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta resolved YARN-11259. --- Resolution: Duplicate > Upgrade JUnit from 4 to 5 in hadoop-yarn-server-applicationhistoryservice > - > > Key: YARN-11259 > URL: https://issues.apache.org/jira/browse/YARN-11259 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.3.4 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11339) Upgrade Junit 4 to 5 in hadoop-yarn-services-api
Ashutosh Gupta created YARN-11339: - Summary: Upgrade Junit 4 to 5 in hadoop-yarn-services-api Key: YARN-11339 URL: https://issues.apache.org/jira/browse/YARN-11339 Project: Hadoop YARN Issue Type: Sub-task Components: test Affects Versions: 3.3.4, 3.3.3 Reporter: Ashutosh Gupta Assignee: Ashutosh Gupta Upgrade Junit 4 to 5 in hadoop-yarn-services-api -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11338) Upgrade Junit 4 to 5 in hadoop-yarn-applications-unmanaged-am-launcher
Ashutosh Gupta created YARN-11338: - Summary: Upgrade Junit 4 to 5 in hadoop-yarn-applications-unmanaged-am-launcher Key: YARN-11338 URL: https://issues.apache.org/jira/browse/YARN-11338 Project: Hadoop YARN Issue Type: Sub-task Components: test Affects Versions: 3.3.4 Reporter: Ashutosh Gupta Assignee: Ashutosh Gupta Upgrade Junit 4 to 5 in hadoop-yarn-applications-unmanaged-am-launcher -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11337) Upgrade Junit 4 to 5 in hadoop-yarn-applications-mawo
Ashutosh Gupta created YARN-11337: - Summary: Upgrade Junit 4 to 5 in hadoop-yarn-applications-mawo Key: YARN-11337 URL: https://issues.apache.org/jira/browse/YARN-11337 Project: Hadoop YARN Issue Type: Sub-task Components: test Affects Versions: 3.3.4, 3.3.3 Reporter: Ashutosh Gupta Assignee: Ashutosh Gupta Fix For: 3.4.0 Upgrade Junit 4 to 5 in hadoop-yarn-applications-mawo -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11336) Upgrade Junit 4 to 5 in hadoop-yarn-applications-catalog-webapp
Ashutosh Gupta created YARN-11336: - Summary: Upgrade Junit 4 to 5 in hadoop-yarn-applications-catalog-webapp Key: YARN-11336 URL: https://issues.apache.org/jira/browse/YARN-11336 Project: Hadoop YARN Issue Type: Sub-task Components: test Affects Versions: 3.3.4, 3.3.3 Reporter: Ashutosh Gupta Assignee: Ashutosh Gupta Upgrade Junit 4 to 5 in hadoop-yarn-applications-catalog-webapp -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11303) Upgrade jquery ui to 1.13.2
[ https://issues.apache.org/jira/browse/YARN-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17608642#comment-17608642 ] Ashutosh Gupta commented on YARN-11303: --- Sure [~ste...@apache.org]. I will raise the PR for branch-3.3 too > Upgrade jquery ui to 1.13.2 > --- > > Key: YARN-11303 > URL: https://issues.apache.org/jira/browse/YARN-11303 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The current jquery-ui version used(1.13.1) in the trunk has the following > vulnerability > [CVE-2022-31160|https://nvd.nist.gov/vuln/detail/CVE-2022-31160] so we need > to upgrade to at least 1.13.2. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11309) datatables@1.10.17 sonatype-2020-0988 vulnerability
[ https://issues.apache.org/jira/browse/YARN-11309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta reassigned YARN-11309: - Assignee: Ashutosh Gupta > datatables@1.10.17 sonatype-2020-0988 vulnerability > > > Key: YARN-11309 > URL: https://issues.apache.org/jira/browse/YARN-11309 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.3.4 >Reporter: Eugene Shinn (Truveta) >Assignee: Ashutosh Gupta >Priority: Major > > Our static analysis security tool detected that YARN's UI currently includes > a vulnerable version of datatables detected by Sonatype (sonatype-2020-0988). > From the vulnerability description: > _"The `datatables.net` package is vulnerable to Prototype Pollution. The > `setData` function in `jquery.dataTables.js` fails to protect prototype > attributes when objects are created during the application's execution. A > remote attacker can exploit this to modify the behavior of object prototypes > which, depending on their use in the application, may result in a Denial of > Service (DoS), Remote Code Execution (RCE), or other unexpected execution > flow."_ > This issue was addressed in v 1.11.5 (ref: [Fix: Protect developers from > inadvertantely introducing prototype pol… · > DataTables/Dist-DataTables@e2e19ea > (github.com)).|https://github.com/DataTables/Dist-DataTables/commit/e2e19eac7e5a6f140d7eefca5c7deba165b357eb#diff-e7d8309f017dd2ef6385fa8cdc1539a2R2765] > [HDFS-16777] datatables@1.10.17 sonatype-2020-0988 vulnerability - ASF JIRA > (apache.org) was filed to address the identical issue in HDFS' UI. > h4. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11303) Upgrade jquery ui to 1.13.2
[ https://issues.apache.org/jira/browse/YARN-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta reassigned YARN-11303: - Assignee: Ashutosh Gupta (was: D M Murali Krishna Reddy) > Upgrade jquery ui to 1.13.2 > --- > > Key: YARN-11303 > URL: https://issues.apache.org/jira/browse/YARN-11303 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > > The current jquery-ui version used(1.13.1) in the trunk has the following > vulnerability > [CVE-2022-31160|https://nvd.nist.gov/vuln/detail/CVE-2022-31160] so we need > to upgrade to at least 1.13.2. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11255) Support loading alternative docker client config from system environment
[
https://issues.apache.org/jira/browse/YARN-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta updated YARN-11255:
--
Description:
When using YARN docker support, although the hadoop shell supported
{code:java}
-docker_client_config{code}
to pass the client config file that contains security token to generate the
docker config for each job as a temporary file.
For other applications that submit jobs to YARN, e.g. Spark, which loads the
docker setting via system environment e.g.
{code:java}
spark.executorEnv.* {code}
will not be able to add those authorization token because this system
environment isn't considered in YARN.
Add genetic solution to handle these kind of cases without making changes in
spark code or others
Eg
When using remote container registry, the
{{YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG}} must reference the config.json
file containing the credentials used to authenticate.
{code:java}
DOCKER_IMAGE_NAME=hadoop-docker
DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
spark-submit --master yarn \
--deploy-mode cluster \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME
\
--conf
spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
--conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
sparkR.R{code}
was:
When using YARN docker support, although the hadoop shell supported
{code:java}
-docker_client_config{code}
to pass the client config file that contains security token to generate the
docker config for each job as a temporary file.
For other applications that submit jobs to YARN, e.g. Spark, which loads the
docker setting via system environment e.g.
{code:java}
spark.executorEnv.* {code}
will not be able to add those authorization token because this system
environment isn't considered in YARN.
Add genetic solution to handle these kind of cases without making changes in
spark code or others
Eg
When using remote container registry, the
{{YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG}} must reference the config.json
file containing the credentials used to authenticate.
DOCKER_IMAGE_NAME=hadoop-docker
DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
spark-submit --master yarn \
--deploy-mode cluster \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME
\
--conf
spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
--conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
sparkR.R
> Support loading alternative docker client config from system environment
>
>
> Key: YARN-11255
> URL: https://issues.apache.org/jira/browse/YARN-11255
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
>
> When using YARN docker support, although the hadoop shell supported
> {code:java}
> -docker_client_config{code}
> to pass the client config file that contains security token to generate the
> docker config for each job as a temporary file.
> For other applications that submit jobs to YARN, e.g. Spark, which loads the
> docker setting via system environment e.g.
> {code:java}
> spark.executorEnv.* {code}
> will not be able to add those authorization token because this system
> environment isn't considered in YARN.
> Add genetic solution to handle these kind of cases without making changes in
> spark code or others
> Eg
> When using remote container registry, the
> {{YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG}} must reference the config.json
> file containing the credentials used to authenticate.
> {code:java}
> DOCKER_IMAGE_NAME=hadoop-docker
> DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
> spark-submit --master yarn \
> --deploy-mode cluster \
> --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
> --conf
> spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
> --conf
> spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
> \
> --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
> --conf
> spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME
> \
> --conf
[jira] [Updated] (YARN-11255) Support loading alternative docker client config from system environment
[
https://issues.apache.org/jira/browse/YARN-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta updated YARN-11255:
--
Description:
When using YARN docker support, although the hadoop shell supported
{code:java}
-docker_client_config{code}
to pass the client config file that contains security token to generate the
docker config for each job as a temporary file.
For other applications that submit jobs to YARN, e.g. Spark, which loads the
docker setting via system environment e.g.
{code:java}
spark.executorEnv.* {code}
will not be able to add those authorization token because this system
environment isn't considered in YARN.
Add genetic solution to handle these kind of cases without making changes in
spark code or others
Eg
When using remote container registry, the
{{YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG}} must reference the config.json
file containing the credentials used to authenticate.
DOCKER_IMAGE_NAME=hadoop-docker
DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
spark-submit --master yarn \
--deploy-mode cluster \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME
\
--conf
spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
--conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
sparkR.R{{}}
was:
When using YARN docker support, although the hadoop shell supported
{code:java}
-docker_client_config{code}
to pass the client config file that contains security token to generate the
docker config for each job as a temporary file.
For other applications that submit jobs to YARN, e.g. Spark, which loads the
docker setting via system environment e.g.
{code:java}
spark.executorEnv.* {code}
will not be able to add those authorization token because this system
environment isn't considered in YARN.
Add genetic solution to handle these kind of cases without making changes in
spark code or others
> Support loading alternative docker client config from system environment
>
>
> Key: YARN-11255
> URL: https://issues.apache.org/jira/browse/YARN-11255
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
>
> When using YARN docker support, although the hadoop shell supported
> {code:java}
> -docker_client_config{code}
> to pass the client config file that contains security token to generate the
> docker config for each job as a temporary file.
> For other applications that submit jobs to YARN, e.g. Spark, which loads the
> docker setting via system environment e.g.
> {code:java}
> spark.executorEnv.* {code}
> will not be able to add those authorization token because this system
> environment isn't considered in YARN.
> Add genetic solution to handle these kind of cases without making changes in
> spark code or others
> Eg
> When using remote container registry, the
> {{YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG}} must reference the config.json
> file containing the credentials used to authenticate.
> DOCKER_IMAGE_NAME=hadoop-docker
> DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
> spark-submit --master yarn \
> --deploy-mode cluster \
> --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
> --conf
> spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
> --conf
> spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
> \
> --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
> --conf
> spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME
> \
> --conf
> spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
> \
> sparkR.R{{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11255) Support loading alternative docker client config from system environment
[
https://issues.apache.org/jira/browse/YARN-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta updated YARN-11255:
--
Description:
When using YARN docker support, although the hadoop shell supported
{code:java}
-docker_client_config{code}
to pass the client config file that contains security token to generate the
docker config for each job as a temporary file.
For other applications that submit jobs to YARN, e.g. Spark, which loads the
docker setting via system environment e.g.
{code:java}
spark.executorEnv.* {code}
will not be able to add those authorization token because this system
environment isn't considered in YARN.
Add genetic solution to handle these kind of cases without making changes in
spark code or others
Eg
When using remote container registry, the
{{YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG}} must reference the config.json
file containing the credentials used to authenticate.
DOCKER_IMAGE_NAME=hadoop-docker
DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
spark-submit --master yarn \
--deploy-mode cluster \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME
\
--conf
spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
--conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
sparkR.R
was:
When using YARN docker support, although the hadoop shell supported
{code:java}
-docker_client_config{code}
to pass the client config file that contains security token to generate the
docker config for each job as a temporary file.
For other applications that submit jobs to YARN, e.g. Spark, which loads the
docker setting via system environment e.g.
{code:java}
spark.executorEnv.* {code}
will not be able to add those authorization token because this system
environment isn't considered in YARN.
Add genetic solution to handle these kind of cases without making changes in
spark code or others
Eg
When using remote container registry, the
{{YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG}} must reference the config.json
file containing the credentials used to authenticate.
DOCKER_IMAGE_NAME=hadoop-docker
DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
spark-submit --master yarn \
--deploy-mode cluster \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME
\
--conf
spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
--conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
sparkR.R{{}}
> Support loading alternative docker client config from system environment
>
>
> Key: YARN-11255
> URL: https://issues.apache.org/jira/browse/YARN-11255
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
>
> When using YARN docker support, although the hadoop shell supported
> {code:java}
> -docker_client_config{code}
> to pass the client config file that contains security token to generate the
> docker config for each job as a temporary file.
> For other applications that submit jobs to YARN, e.g. Spark, which loads the
> docker setting via system environment e.g.
> {code:java}
> spark.executorEnv.* {code}
> will not be able to add those authorization token because this system
> environment isn't considered in YARN.
> Add genetic solution to handle these kind of cases without making changes in
> spark code or others
> Eg
> When using remote container registry, the
> {{YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG}} must reference the config.json
> file containing the credentials used to authenticate.
> DOCKER_IMAGE_NAME=hadoop-docker
> DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
> spark-submit --master yarn \
> --deploy-mode cluster \
> --conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
> --conf
> spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
> --conf
> spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
> \
> --conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
> --conf
> spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME
> \
> --conf
>
[jira] [Commented] (YARN-9195) RM Queue's pending container number might get decreased unexpectedly or even become negative once RM failover
[ https://issues.apache.org/jira/browse/YARN-9195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568184#comment-17568184 ] Ashutosh Gupta commented on YARN-9195: -- Hi [~ssy] [~cheersyang] - The seems to be critical and pending for a while and we are facing similar issue internally. Taking it forward. Thanks. > RM Queue's pending container number might get decreased unexpectedly or even > become negative once RM failover > - > > Key: YARN-9195 > URL: https://issues.apache.org/jira/browse/YARN-9195 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 3.1.0 >Reporter: MalcolmSanders >Assignee: MalcolmSanders >Priority: Critical > Attachments: YARN-9195.001.patch, YARN-9195.002.patch, > YARN-9195.003.patch, cases_to_recreate_negative_pending_requests_scenario.diff > > > Hi, all: > Previously we have encountered a serious problem in ResourceManager, we found > that pending container number of one RM queue became negative after RM failed > over. Since queues in RM are managed in hierarchical structure, the root > queue's pending containers became negative at last, thus the scheduling > process of the whole cluster became affected. > The version of both our RM server and AMRM client in our application are > based on yarn 3.1, and we uses AMRMClientAsync#addSchedulingRequests() method > in our application to request resources from RM. > After investigation, we found that the direct cause was numAllocations of > some AMs' requests became negative after RM failed over. And there are at > lease three necessary conditions: > (1) Use schedulingRequests in AMRM client, and the application set zero to > the numAllocations for a schedulingRequest. In our batch job scenario, the > numAllocations of a schedulingRequest could turn to zero because > theoretically we can run a full batch job using only one container. > (2) RM failovers. > (3) Before AM reregisters itself to RM after RM restarts, RM has already > recovered some of the application's containers assigned before. > Here are some more details about the implementation: > (1) After RM recovers, RM will send all alive containers to AM once it > re-register itself through > RegisterApplicationMasterResponse#getContainersFromPreviousAttempts. > (2) During registerApplicationMaster, AMRMClientImpl will > removeFromOutstandingSchedulingRequests once AM gets > ContainersFromPreviousAttempts without checking whether these containers have > been assigned before. As a consequence, its outstanding requests might be > decreased unexpectedly even if it may not become negative. > (3) There is no sanity check in RM to validate requests from AMs. > For better illustrating this case, I've written a test case based on the > latest hadoop trunk, posted in the attachment. You may try case > testAMRMClientWithNegativePendingRequestsOnRMRestart and > testAMRMClientOnUnexpectedlyDecreasedPendingRequestsOnRMRestart . > To solve this issue, I propose to filter allocated containers before > removeFromOutstandingSchedulingRequests in AMRMClientImpl during > registerApplicationMaster, and some sanity checks are also needed to prevent > things from getting worse. > More comments and suggestions are welcomed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9195) RM Queue's pending container number might get decreased unexpectedly or even become negative once RM failover
[ https://issues.apache.org/jira/browse/YARN-9195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta reassigned YARN-9195: Assignee: Ashutosh Gupta (was: MalcolmSanders) > RM Queue's pending container number might get decreased unexpectedly or even > become negative once RM failover > - > > Key: YARN-9195 > URL: https://issues.apache.org/jira/browse/YARN-9195 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 3.1.0 >Reporter: MalcolmSanders >Assignee: Ashutosh Gupta >Priority: Critical > Attachments: YARN-9195.001.patch, YARN-9195.002.patch, > YARN-9195.003.patch, cases_to_recreate_negative_pending_requests_scenario.diff > > > Hi, all: > Previously we have encountered a serious problem in ResourceManager, we found > that pending container number of one RM queue became negative after RM failed > over. Since queues in RM are managed in hierarchical structure, the root > queue's pending containers became negative at last, thus the scheduling > process of the whole cluster became affected. > The version of both our RM server and AMRM client in our application are > based on yarn 3.1, and we uses AMRMClientAsync#addSchedulingRequests() method > in our application to request resources from RM. > After investigation, we found that the direct cause was numAllocations of > some AMs' requests became negative after RM failed over. And there are at > lease three necessary conditions: > (1) Use schedulingRequests in AMRM client, and the application set zero to > the numAllocations for a schedulingRequest. In our batch job scenario, the > numAllocations of a schedulingRequest could turn to zero because > theoretically we can run a full batch job using only one container. > (2) RM failovers. > (3) Before AM reregisters itself to RM after RM restarts, RM has already > recovered some of the application's containers assigned before. > Here are some more details about the implementation: > (1) After RM recovers, RM will send all alive containers to AM once it > re-register itself through > RegisterApplicationMasterResponse#getContainersFromPreviousAttempts. > (2) During registerApplicationMaster, AMRMClientImpl will > removeFromOutstandingSchedulingRequests once AM gets > ContainersFromPreviousAttempts without checking whether these containers have > been assigned before. As a consequence, its outstanding requests might be > decreased unexpectedly even if it may not become negative. > (3) There is no sanity check in RM to validate requests from AMs. > For better illustrating this case, I've written a test case based on the > latest hadoop trunk, posted in the attachment. You may try case > testAMRMClientWithNegativePendingRequestsOnRMRestart and > testAMRMClientOnUnexpectedlyDecreasedPendingRequestsOnRMRestart . > To solve this issue, I propose to filter allocated containers before > removeFromOutstandingSchedulingRequests in AMRMClientImpl during > registerApplicationMaster, and some sanity checks are also needed to prevent > things from getting worse. > More comments and suggestions are welcomed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10125) In Federation, kill application from client does not kill Unmanaged AM's and containers launched by Unmanaged AM
[ https://issues.apache.org/jira/browse/YARN-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17564508#comment-17564508 ] Ashutosh Gupta commented on YARN-10125: --- This seems to an important fix but been missing to be fixed for a while now. [~dmmkr] - Can you raise a CR and make any changes if required or can I take this up to get it fixed. > In Federation, kill application from client does not kill Unmanaged AM's and > containers launched by Unmanaged AM > > > Key: YARN-10125 > URL: https://issues.apache.org/jira/browse/YARN-10125 > Project: Hadoop YARN > Issue Type: Bug > Components: client, federation, router >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > Attachments: YARN-10125.001.patch, YARN-10125.002.patch > > > In Federation, killing an application from client using "bin/yarn application > -kill ", kills the containers only of the home subcluster, > the Unmanaged AM and the containers launched in other subcluster are not > being killed causing blocking of resources. > The containers get killed after the task gets completed and The unmanaged AM > gets killed after 10 minutes of killing the application, killing any > remaining running containers in that subcluster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10745) Change Log level from info to debug for few logs and remove unnecessary debuglog checks
[ https://issues.apache.org/jira/browse/YARN-10745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17564504#comment-17564504 ] Ashutosh Gupta commented on YARN-10745: --- Hi [~dmmkr] - Are you working on it or as suggested by [~brahmareddy] - I can break it up into subtasks for each module and take things forward ? > Change Log level from info to debug for few logs and remove unnecessary > debuglog checks > --- > > Key: YARN-10745 > URL: https://issues.apache.org/jira/browse/YARN-10745 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Minor > Attachments: YARN-10745.001.patch, YARN-10745.002.patch, > YARN-10745.003.patch, YARN-10745.004.patch, YARN-10745.005.patch, > YARN-10745.006.patch > > > Change the info log level to debug for few logs so that the load on the > logger decreases in large cluster and improves the performance. > Remove the unnecessary isDebugEnabled() checks for printing strings without > any string concatenation -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9342) Moving log4j1 to log4j2 in hadoop-yarn
[
https://issues.apache.org/jira/browse/YARN-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17563428#comment-17563428
]
Ashutosh Gupta commented on YARN-9342:
--
Hi [~abmodi] - Can you please some info about the process that you guys opted
for switching to Log4j2 ? It will be really helpful.
> Moving log4j1 to log4j2 in hadoop-yarn
> --
>
> Key: YARN-9342
> URL: https://issues.apache.org/jira/browse/YARN-9342
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 3.1.2
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>
> 1. Log4j2 Asynchronous Logging will give significant improvement in the
> performance.
> 2. Log4j2 does not have below locking issue which Log4j1 has.
> {code}
> "Thread-16" #40 daemon prio=5 os_prio=0 tid=0x7f181f9bb800 nid=0x125
> waiting for monitor entry [0x7ef163bab000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.log4j.Category.callAppenders(Category.java:204)
> - locked <0x7ef2d803e2b8> (a org.apache.log4j.spi.RootLogger)
> at org.apache.log4j.Category.forcedLog(Category.java:391)
> at org.apache.log4j.Category.log(Category.java:856)
> at
> org.apache.commons.logging.impl.Log4JLogger.info(Log4JLogger.java:176)
> {code}
> https://bz.apache.org/bugzilla/show_bug.cgi?id=57714
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10121) In Federation executing yarn queue status command throws an exception
[
https://issues.apache.org/jira/browse/YARN-10121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17562905#comment-17562905
]
Ashutosh Gupta edited comment on YARN-10121 at 7/6/22 12:30 AM:
Hi [~slfan1989] , I think we should mark resoution as no fix required or not a
problem as its already fixed.
was (Author: groot):
Hi [~slfan1989] , I think we should mark resoution as no fix required or
resolved
> In Federation executing yarn queue status command throws an exception
> -
>
> Key: YARN-10121
> URL: https://issues.apache.org/jira/browse/YARN-10121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, yarn
>Reporter: Sushanta Sen
>Assignee: fanshilun
>Priority: Major
> Fix For: 3.4.0
>
>
> yarn queue status is failing, prompting an error
> “org.apache.commons.lang.NotImplementedException: Code is not implemented”.
> {noformat}
> ./yarn queue -status default
> Exception in thread "main" org.apache.commons.lang.NotImplementedException:
> Code is not implemented
> at
> org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor.getQueueInfo(FederationClientInterceptor.java:715)
> at
> org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService.getQueueInfo(RouterClientRMService.java:246)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getQueueInfo(ApplicationClientProtocolPBServiceImpl.java:328)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:591)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:928)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:863)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2793)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getQueueInfo(ApplicationClientProtocolPBClientImpl.java:341)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy8.getQueueInfo(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfo(YarnClientImpl.java:650)
> at
> org.apache.hadoop.yarn.client.cli.QueueCLI.listQueue(QueueCLI.java:111)
> at org.apache.hadoop.yarn.client.cli.QueueCLI.run(QueueCLI.java:78)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.yarn.client.cli.QueueCLI.main(QueueCLI.java:50)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For
[jira] [Resolved] (YARN-10121) In Federation executing yarn queue status command throws an exception
[
https://issues.apache.org/jira/browse/YARN-10121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta resolved YARN-10121.
---
Resolution: Not A Problem
> In Federation executing yarn queue status command throws an exception
> -
>
> Key: YARN-10121
> URL: https://issues.apache.org/jira/browse/YARN-10121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, yarn
>Reporter: Sushanta Sen
>Assignee: fanshilun
>Priority: Major
> Fix For: 3.4.0
>
>
> yarn queue status is failing, prompting an error
> “org.apache.commons.lang.NotImplementedException: Code is not implemented”.
> {noformat}
> ./yarn queue -status default
> Exception in thread "main" org.apache.commons.lang.NotImplementedException:
> Code is not implemented
> at
> org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor.getQueueInfo(FederationClientInterceptor.java:715)
> at
> org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService.getQueueInfo(RouterClientRMService.java:246)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getQueueInfo(ApplicationClientProtocolPBServiceImpl.java:328)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:591)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:928)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:863)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2793)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getQueueInfo(ApplicationClientProtocolPBClientImpl.java:341)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy8.getQueueInfo(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfo(YarnClientImpl.java:650)
> at
> org.apache.hadoop.yarn.client.cli.QueueCLI.listQueue(QueueCLI.java:111)
> at org.apache.hadoop.yarn.client.cli.QueueCLI.run(QueueCLI.java:78)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.yarn.client.cli.QueueCLI.main(QueueCLI.java:50)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-10121) In Federation executing yarn queue status command throws an exception
[
https://issues.apache.org/jira/browse/YARN-10121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta reopened YARN-10121:
---
> In Federation executing yarn queue status command throws an exception
> -
>
> Key: YARN-10121
> URL: https://issues.apache.org/jira/browse/YARN-10121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, yarn
>Reporter: Sushanta Sen
>Assignee: fanshilun
>Priority: Major
> Fix For: 3.4.0
>
>
> yarn queue status is failing, prompting an error
> “org.apache.commons.lang.NotImplementedException: Code is not implemented”.
> {noformat}
> ./yarn queue -status default
> Exception in thread "main" org.apache.commons.lang.NotImplementedException:
> Code is not implemented
> at
> org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor.getQueueInfo(FederationClientInterceptor.java:715)
> at
> org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService.getQueueInfo(RouterClientRMService.java:246)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getQueueInfo(ApplicationClientProtocolPBServiceImpl.java:328)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:591)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:928)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:863)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2793)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getQueueInfo(ApplicationClientProtocolPBClientImpl.java:341)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy8.getQueueInfo(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfo(YarnClientImpl.java:650)
> at
> org.apache.hadoop.yarn.client.cli.QueueCLI.listQueue(QueueCLI.java:111)
> at org.apache.hadoop.yarn.client.cli.QueueCLI.run(QueueCLI.java:78)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.yarn.client.cli.QueueCLI.main(QueueCLI.java:50)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10121) In Federation executing yarn queue status command throws an exception
[
https://issues.apache.org/jira/browse/YARN-10121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17562905#comment-17562905
]
Ashutosh Gupta commented on YARN-10121:
---
Hi [~slfan1989] , I think we should mark resoution as no fix required or
resolved
> In Federation executing yarn queue status command throws an exception
> -
>
> Key: YARN-10121
> URL: https://issues.apache.org/jira/browse/YARN-10121
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, yarn
>Reporter: Sushanta Sen
>Assignee: fanshilun
>Priority: Major
> Fix For: 3.4.0
>
>
> yarn queue status is failing, prompting an error
> “org.apache.commons.lang.NotImplementedException: Code is not implemented”.
> {noformat}
> ./yarn queue -status default
> Exception in thread "main" org.apache.commons.lang.NotImplementedException:
> Code is not implemented
> at
> org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor.getQueueInfo(FederationClientInterceptor.java:715)
> at
> org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService.getQueueInfo(RouterClientRMService.java:246)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getQueueInfo(ApplicationClientProtocolPBServiceImpl.java:328)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:591)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:530)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:928)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:863)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2793)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateRuntimeException(RPCUtil.java:85)
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:122)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getQueueInfo(ApplicationClientProtocolPBClientImpl.java:341)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
> at com.sun.proxy.$Proxy8.getQueueInfo(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfo(YarnClientImpl.java:650)
> at
> org.apache.hadoop.yarn.client.cli.QueueCLI.listQueue(QueueCLI.java:111)
> at org.apache.hadoop.yarn.client.cli.QueueCLI.run(QueueCLI.java:78)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.yarn.client.cli.QueueCLI.main(QueueCLI.java:50)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9403) GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table
[
https://issues.apache.org/jira/browse/YARN-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta reassigned YARN-9403:
Assignee: Ashutosh Gupta (was: Prabhu Joseph)
> GET /apps/{appid}/entities/YARN_APPLICATION accesses application table
> instead of entity table
> --
>
> Key: YARN-9403
> URL: https://issues.apache.org/jira/browse/YARN-9403
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Major
> Attachments: YARN-9403-001.patch, YARN-9403-002.patch,
> YARN-9403-003.patch, YARN-9403-004.patch
>
>
> {noformat}"GET /apps/{appid}/entities/YARN_APPLICATION"{noformat} accesses
> application table instead of entity table. As per the doc, With this API, you
> can query generic entities identified by cluster ID, application ID and
> per-framework entity type. But it also provides all the apps when entityType
> is set to YARN_APPLICATION. It should only access Entity Table through
> {{GenericEntityReader}}.
> Wrong Output: With YARN_APPLICATION entityType, all applications listed from
> application tables.
> {code}
> [hbase@yarn-ats-3 centos]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/YARN_APPLICATION?user.name=hbase=hbase=word%20count;
> | jq .
> [
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553258922721,
> "idprefix": 0,
> "isrelatedto": {},
> "relatesto": {},
> "info": {
> "UID": "ats!application_1553258815132_0002",
> "FROM_ID": "ats!hbase!word
> count!1553258922721!application_1553258815132_0002"
> },
> "configs": {},
> "type": "YARN_APPLICATION",
> "id": "application_1553258815132_0002"
> },
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553258825918,
> "idprefix": 0,
> "isrelatedto": {},
> "relatesto": {},
> "info": {
> "UID": "ats!application_1553258815132_0001",
> "FROM_ID": "ats!hbase!word
> count!1553258825918!application_1553258815132_0001"
> },
> "configs": {},
> "type": "YARN_APPLICATION",
> "id": "application_1553258815132_0001"
> }
> ]
> {code}
> Right Output: With correct entity type (MAPREDUCE_JOB) it accesses entity
> table for given applicationId and entityType.
> {code}
> [hbase@yarn-ats-3 centos]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/MAPREDUCE_JOB?user.name=hbase=hbase=word%20count;
> | jq .
> [
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553258926667,
> "idprefix": 0,
> "isrelatedto": {},
> "relatesto": {},
> "info": {
> "UID":
> "ats!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002",
> "FROM_ID": "ats!hbase!word
> count!1553258922721!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002"
> },
> "configs": {},
> "type": "MAPREDUCE_JOB",
> "id": "job_1553258815132_0002"
> }
> ]
> {code}
> Flow Activity and Flow Run tables can also be accessed using similar way.
> {code}
> GET /apps/{appid}/entities/YARN_FLOW_ACTIVITY
> GET /apps/{appid}/entities/YARN_FLOW_RUN
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9403) GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table
[
https://issues.apache.org/jira/browse/YARN-9403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17560680#comment-17560680
]
Ashutosh Gupta commented on YARN-9403:
--
Taking it up
> GET /apps/{appid}/entities/YARN_APPLICATION accesses application table
> instead of entity table
> --
>
> Key: YARN-9403
> URL: https://issues.apache.org/jira/browse/YARN-9403
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Major
> Attachments: YARN-9403-001.patch, YARN-9403-002.patch,
> YARN-9403-003.patch, YARN-9403-004.patch
>
>
> {noformat}"GET /apps/{appid}/entities/YARN_APPLICATION"{noformat} accesses
> application table instead of entity table. As per the doc, With this API, you
> can query generic entities identified by cluster ID, application ID and
> per-framework entity type. But it also provides all the apps when entityType
> is set to YARN_APPLICATION. It should only access Entity Table through
> {{GenericEntityReader}}.
> Wrong Output: With YARN_APPLICATION entityType, all applications listed from
> application tables.
> {code}
> [hbase@yarn-ats-3 centos]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/YARN_APPLICATION?user.name=hbase=hbase=word%20count;
> | jq .
> [
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553258922721,
> "idprefix": 0,
> "isrelatedto": {},
> "relatesto": {},
> "info": {
> "UID": "ats!application_1553258815132_0002",
> "FROM_ID": "ats!hbase!word
> count!1553258922721!application_1553258815132_0002"
> },
> "configs": {},
> "type": "YARN_APPLICATION",
> "id": "application_1553258815132_0002"
> },
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553258825918,
> "idprefix": 0,
> "isrelatedto": {},
> "relatesto": {},
> "info": {
> "UID": "ats!application_1553258815132_0001",
> "FROM_ID": "ats!hbase!word
> count!1553258825918!application_1553258815132_0001"
> },
> "configs": {},
> "type": "YARN_APPLICATION",
> "id": "application_1553258815132_0001"
> }
> ]
> {code}
> Right Output: With correct entity type (MAPREDUCE_JOB) it accesses entity
> table for given applicationId and entityType.
> {code}
> [hbase@yarn-ats-3 centos]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0002/entities/MAPREDUCE_JOB?user.name=hbase=hbase=word%20count;
> | jq .
> [
> {
> "metrics": [],
> "events": [],
> "createdtime": 1553258926667,
> "idprefix": 0,
> "isrelatedto": {},
> "relatesto": {},
> "info": {
> "UID":
> "ats!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002",
> "FROM_ID": "ats!hbase!word
> count!1553258922721!application_1553258815132_0002!MAPREDUCE_JOB!0!job_1553258815132_0002"
> },
> "configs": {},
> "type": "MAPREDUCE_JOB",
> "id": "job_1553258815132_0002"
> }
> ]
> {code}
> Flow Activity and Flow Run tables can also be accessed using similar way.
> {code}
> GET /apps/{appid}/entities/YARN_FLOW_ACTIVITY
> GET /apps/{appid}/entities/YARN_FLOW_RUN
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10287) Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping
[
https://issues.apache.org/jira/browse/YARN-10287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17560677#comment-17560677
]
Ashutosh Gupta commented on YARN-10287:
---
Taking it up
> Update scheduler-conf corrupts the CS configuration when removing queue which
> is referred in queue mapping
> --
>
> Key: YARN-10287
> URL: https://issues.apache.org/jira/browse/YARN-10287
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Akhil PB
>Assignee: Ashutosh Gupta
>Priority: Major
> Attachments: YARN-10287-001.patch
>
>
> Update scheduler-conf corrupts the CS configuration when removing queue which
> is referred in queue mapping. The deletion is failed with below error
> message but the queue got removed from CS configuration and job submission
> failed and not removed from the backend ZKConfigurationStore. On subsequent
> modify using scheduler-conf, the queue appears again from ZKConfigurationStore
> {code}
> 2020-05-22 12:38:38,252 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices: Exception
> thrown when modifying configuration.
> java.io.IOException: Failed to re-init queues : mapping contains invalid or
> non-leaf queue Prod
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:478)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:430)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices$13.run(RMWebServices.java:2389)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices$13.run(RMWebServices.java:2377)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.updateSchedulerConfiguration(RMWebServices.java:2377)
> {code}
> *Repro:*
> {code}
> 1. Setup Queue Mapping
> yarn.scheduler.capacity.root.queues=default,dummy
> yarn.scheduler.capacity.queue-mappings=g:hadoop:dummy
> 2. Stop the root.dummy queue
>
>root.dummy
>
>
>state
>STOPPED
>
>
>
>
>
> 3. Delete the root.dummy queue
> curl --negotiate -u : -X PUT -d @abc.xml -H "Content-type: application/xml"
> 'http://:8088/ws/v1/cluster/scheduler-conf?user.name=yarn'
>
>
> root.default
>
>
> capacity
> 100
>
>
>
> root.dummy
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10287) Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping
[
https://issues.apache.org/jira/browse/YARN-10287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta reassigned YARN-10287:
-
Assignee: Ashutosh Gupta (was: Prabhu Joseph)
> Update scheduler-conf corrupts the CS configuration when removing queue which
> is referred in queue mapping
> --
>
> Key: YARN-10287
> URL: https://issues.apache.org/jira/browse/YARN-10287
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.3.0
>Reporter: Akhil PB
>Assignee: Ashutosh Gupta
>Priority: Major
> Attachments: YARN-10287-001.patch
>
>
> Update scheduler-conf corrupts the CS configuration when removing queue which
> is referred in queue mapping. The deletion is failed with below error
> message but the queue got removed from CS configuration and job submission
> failed and not removed from the backend ZKConfigurationStore. On subsequent
> modify using scheduler-conf, the queue appears again from ZKConfigurationStore
> {code}
> 2020-05-22 12:38:38,252 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices: Exception
> thrown when modifying configuration.
> java.io.IOException: Failed to re-init queues : mapping contains invalid or
> non-leaf queue Prod
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:478)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:430)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices$13.run(RMWebServices.java:2389)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices$13.run(RMWebServices.java:2377)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.updateSchedulerConfiguration(RMWebServices.java:2377)
> {code}
> *Repro:*
> {code}
> 1. Setup Queue Mapping
> yarn.scheduler.capacity.root.queues=default,dummy
> yarn.scheduler.capacity.queue-mappings=g:hadoop:dummy
> 2. Stop the root.dummy queue
>
>root.dummy
>
>
>state
>STOPPED
>
>
>
>
>
> 3. Delete the root.dummy queue
> curl --negotiate -u : -X PUT -d @abc.xml -H "Content-type: application/xml"
> 'http://:8088/ws/v1/cluster/scheduler-conf?user.name=yarn'
>
>
> root.default
>
>
> capacity
> 100
>
>
>
> root.dummy
>
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10538) Add recommissioning nodes to the list of updated nodes returned to the AM
[ https://issues.apache.org/jira/browse/YARN-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558941#comment-17558941 ] Ashutosh Gupta commented on YARN-10538: --- Thanks [~aajisaka] for review and merging to branch-2.10 > Add recommissioning nodes to the list of updated nodes returned to the AM > - > > Key: YARN-10538 > URL: https://issues.apache.org/jira/browse/YARN-10538 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.9.1, 3.1.1 >Reporter: Srinivas S T >Assignee: Srinivas S T >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.1, 3.2.3 > > Time Spent: 2h > Remaining Estimate: 0h > > YARN-6483 introduced nodes that transitioned to DECOMMISSIONING state to the > list of updated nodes returned to the AM. This allows the Spark application > master to gracefully decommission its containers on the decommissioning node. > But if the node were to be recommissioned, the Spark application master would > not be aware of this. We propose to add recommissioned node to the list of > updated nodes sent to the AM when a recommission node transition occurs. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11197) Backport YARN-9608 - DecommissioningNodesWatcher should get lists of running applications on node from RMNode.
[ https://issues.apache.org/jira/browse/YARN-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558650#comment-17558650 ] Ashutosh Gupta commented on YARN-11197: --- Hi [~cnauroth] . Thanks for checking - I am actively working on it. > Backport YARN-9608 - DecommissioningNodesWatcher should get lists of running > applications on node from RMNode. > -- > > Key: YARN-11197 > URL: https://issues.apache.org/jira/browse/YARN-11197 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.10.1, 2.10.2 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > > There has been ask in community and internally as well to have YARN-9608 for > hadoop-2.10 as well. > Evaluate and create patch for the same. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11197) Backport YARN-9608 - DecommissioningNodesWatcher should get lists of running applications on node from RMNode.
[ https://issues.apache.org/jira/browse/YARN-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated YARN-11197: -- Description: There has been ask in community and internally as well to have YARN-9608 for hadoop-2.10 as well. Evaluate and create patch for the same. was: There has been ask in community and internally as well to have YARN-9608 for hadoop-2.10 as well. Evaluate and creating patch for the same. > Backport YARN-9608 - DecommissioningNodesWatcher should get lists of running > applications on node from RMNode. > -- > > Key: YARN-11197 > URL: https://issues.apache.org/jira/browse/YARN-11197 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.10.1, 2.10.2 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > > There has been ask in community and internally as well to have YARN-9608 for > hadoop-2.10 as well. > Evaluate and create patch for the same. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11197) Backport YARN 9608 DecommissioningNodesWatcher should get lists of running applications on node from RMNode.
Ashutosh Gupta created YARN-11197: - Summary: Backport YARN 9608 DecommissioningNodesWatcher should get lists of running applications on node from RMNode. Key: YARN-11197 URL: https://issues.apache.org/jira/browse/YARN-11197 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 2.10.2, 2.10.1 Reporter: Ashutosh Gupta Assignee: Ashutosh Gupta There has been ask in community and internally as well to have YARN-9608 for hadoop-2.10 as well. Evaluate and creating patch for the same. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11197) Backport YARN-9608 - DecommissioningNodesWatcher should get lists of running applications on node from RMNode.
[ https://issues.apache.org/jira/browse/YARN-11197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated YARN-11197: -- Summary: Backport YARN-9608 - DecommissioningNodesWatcher should get lists of running applications on node from RMNode. (was: Backport YARN 9608 DecommissioningNodesWatcher should get lists of running applications on node from RMNode.) > Backport YARN-9608 - DecommissioningNodesWatcher should get lists of running > applications on node from RMNode. > -- > > Key: YARN-11197 > URL: https://issues.apache.org/jira/browse/YARN-11197 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.10.1, 2.10.2 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > > There has been ask in community and internally as well to have YARN-9608 for > hadoop-2.10 as well. > Evaluate and creating patch for the same. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11092) Upgrade jquery ui to 1.13.1
[ https://issues.apache.org/jira/browse/YARN-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17558058#comment-17558058 ] Ashutosh Gupta commented on YARN-11092: --- It makes sense. Thanks for taking care of conflict and updating. > Upgrade jquery ui to 1.13.1 > --- > > Key: YARN-11092 > URL: https://issues.apache.org/jira/browse/YARN-11092 > Project: Hadoop YARN > Issue Type: Bug >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.4 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The current jquery-ui version used(1.12.1) in the trunk has the following > vulnerabilities CVE-2021-41182, CVE-2021-41183, CVE-2021-41184, so we need to > upgrade to at least 1.13.0. > > Also currently for the UI2 we are using the shims repo which is not being > maintained as per the discussion > [https://github.com/components/jqueryui/issues/70] , so if possible we should > move to the main jquery repo [https://github.com/jquery/jquery-ui] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11193) jquery js file lacks approved header
[ https://issues.apache.org/jira/browse/YARN-11193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557533#comment-17557533 ] Ashutosh Gupta commented on YARN-11193: --- Also jquery and jquery-ui are separate upgrades > jquery js file lacks approved header > > > Key: YARN-11193 > URL: https://issues.apache.org/jira/browse/YARN-11193 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.4.0, 3.3.9, 3.3.4 >Reporter: Steve Loughran >Priority: Blocker > > the updated jquery.js file gets rejected by rat. it either needs a valid > header or added to the exclusion list. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11193) jquery js file lacks approved header
[ https://issues.apache.org/jira/browse/YARN-11193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557532#comment-17557532 ] Ashutosh Gupta commented on YARN-11193: --- Jquery should be updated to 3.6.0 in 3.3.4 release as part of this JIRA https://issues.apache.org/jira/browse/HADOOP-18044 and 3.5.1 should't be there. > jquery js file lacks approved header > > > Key: YARN-11193 > URL: https://issues.apache.org/jira/browse/YARN-11193 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.4.0, 3.3.9, 3.3.4 >Reporter: Steve Loughran >Priority: Blocker > > the updated jquery.js file gets rejected by rat. it either needs a valid > header or added to the exclusion list. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-11092) Upgrade jquery ui to 1.13.1
[ https://issues.apache.org/jira/browse/YARN-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557524#comment-17557524 ] Ashutosh Gupta edited comment on YARN-11092 at 6/22/22 3:32 PM: This commit seems to be missing [https://github.com/apache/hadoop/pull/3791/commits/88552f2f2175956a25a728dd6b1c69b0c718b116] which upgraded jquery-3.5.1.min.js to jquery-3.6.0.min.js is missing in branch-3.3.4 was (Author: groot): This commit seems to be missing [https://github.com/apache/hadoop/pull/3791/commits/88552f2f2175956a25a728dd6b1c69b0c718b116] which upgraded jquery-3.5.1.min.js to jquery-3.6.0.min.js is missing in branch-3.3.3 and eventually in branch-3.3.4 as well > Upgrade jquery ui to 1.13.1 > --- > > Key: YARN-11092 > URL: https://issues.apache.org/jira/browse/YARN-11092 > Project: Hadoop YARN > Issue Type: Bug >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.9 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The current jquery-ui version used(1.12.1) in the trunk has the following > vulnerabilities CVE-2021-41182, CVE-2021-41183, CVE-2021-41184, so we need to > upgrade to at least 1.13.0. > > Also currently for the UI2 we are using the shims repo which is not being > maintained as per the discussion > [https://github.com/components/jqueryui/issues/70] , so if possible we should > move to the main jquery repo [https://github.com/jquery/jquery-ui] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-11092) Upgrade jquery ui to 1.13.1
[ https://issues.apache.org/jira/browse/YARN-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557524#comment-17557524 ] Ashutosh Gupta edited comment on YARN-11092 at 6/22/22 3:27 PM: This commit seems to be missing [https://github.com/apache/hadoop/pull/3791/commits/88552f2f2175956a25a728dd6b1c69b0c718b116] which upgraded jquery-3.5.1.min.js to jquery-3.6.0.min.js is missing in branch-3.3.3 and eventually in branch-3.3.4 as well was (Author: groot): This commit [https://github.com/apache/hadoop/pull/3791/commits/88552f2f2175956a25a728dd6b1c69b0c718b116] which upgraded jquery-3.5.1.min.js to jquery-3.6.0.min.js is missing in branch-3.3.3 and eventually in branch-3.3.4 as well > Upgrade jquery ui to 1.13.1 > --- > > Key: YARN-11092 > URL: https://issues.apache.org/jira/browse/YARN-11092 > Project: Hadoop YARN > Issue Type: Bug >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.9 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The current jquery-ui version used(1.12.1) in the trunk has the following > vulnerabilities CVE-2021-41182, CVE-2021-41183, CVE-2021-41184, so we need to > upgrade to at least 1.13.0. > > Also currently for the UI2 we are using the shims repo which is not being > maintained as per the discussion > [https://github.com/components/jqueryui/issues/70] , so if possible we should > move to the main jquery repo [https://github.com/jquery/jquery-ui] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11092) Upgrade jquery ui to 1.13.1
[ https://issues.apache.org/jira/browse/YARN-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557524#comment-17557524 ] Ashutosh Gupta commented on YARN-11092: --- This commit [https://github.com/apache/hadoop/pull/3791/commits/88552f2f2175956a25a728dd6b1c69b0c718b116] which upgraded jquery-3.5.1.min.js to jquery-3.6.0.min.js is missing in branch-3.3.3 and eventually in branch-3.3.4 as well > Upgrade jquery ui to 1.13.1 > --- > > Key: YARN-11092 > URL: https://issues.apache.org/jira/browse/YARN-11092 > Project: Hadoop YARN > Issue Type: Bug >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.9 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The current jquery-ui version used(1.12.1) in the trunk has the following > vulnerabilities CVE-2021-41182, CVE-2021-41183, CVE-2021-41184, so we need to > upgrade to at least 1.13.0. > > Also currently for the UI2 we are using the shims repo which is not being > maintained as per the discussion > [https://github.com/components/jqueryui/issues/70] , so if possible we should > move to the main jquery repo [https://github.com/jquery/jquery-ui] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11092) Upgrade jquery ui to 1.13.1
[ https://issues.apache.org/jira/browse/YARN-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557511#comment-17557511 ] Ashutosh Gupta commented on YARN-11092: --- Hi [~ste...@apache.org] - Seems like jquery was upgraded from jquery-3.5.1.min.js to jquery-3.6.0.min.js as part of this Jira - https://issues.apache.org/jira/browse/HADOOP-18044 > Upgrade jquery ui to 1.13.1 > --- > > Key: YARN-11092 > URL: https://issues.apache.org/jira/browse/YARN-11092 > Project: Hadoop YARN > Issue Type: Bug >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.2.4, 3.3.9 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The current jquery-ui version used(1.12.1) in the trunk has the following > vulnerabilities CVE-2021-41182, CVE-2021-41183, CVE-2021-41184, so we need to > upgrade to at least 1.13.0. > > Also currently for the UI2 we are using the shims repo which is not being > maintained as per the discussion > [https://github.com/components/jqueryui/issues/70] , so if possible we should > move to the main jquery repo [https://github.com/jquery/jquery-ui] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7463) Using getLocalPathForWrite for Container related debug information
[ https://issues.apache.org/jira/browse/YARN-7463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557214#comment-17557214 ] Ashutosh Gupta commented on YARN-7463: -- This issue seems to be already fixed in trunk > Using getLocalPathForWrite for Container related debug information > -- > > Key: YARN-7463 > URL: https://issues.apache.org/jira/browse/YARN-7463 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Minor > Attachments: YARN-7463.1.patch > > > Containers debug information launch_container.sh and directory.info are > always logged into first directory of NM_LOG_DIRS instead of using the log > directory returned from getLogPathForWrite. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9373) HBaseTimelineSchemaCreator has to allow user to configure pre-splits
[ https://issues.apache.org/jira/browse/YARN-9373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557206#comment-17557206 ] Ashutosh Gupta commented on YARN-9373: -- The issue seems to be pending for a while. Taking it up. > HBaseTimelineSchemaCreator has to allow user to configure pre-splits > > > Key: YARN-9373 > URL: https://issues.apache.org/jira/browse/YARN-9373 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Attachments: Configurable_PreSplits.png, YARN-9373-001.patch, > YARN-9373-002.patch, YARN-9373-003.patch > > > Most of the TimelineService HBase tables is set with username splits which is > based on lowercase alphabet (a,ad,an,b,ca). This won't help if the rowkey > starts with either number or uppercase alphabet. We need to allow user to > configure based upon their data. For example, say a user has configured the > yarn.resourcemanager.cluster-id to be ATS or 123, then the splits can be > configured as A,B,C,,, or 100,200,300,,, -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9373) HBaseTimelineSchemaCreator has to allow user to configure pre-splits
[ https://issues.apache.org/jira/browse/YARN-9373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta reassigned YARN-9373: Assignee: Ashutosh Gupta (was: Prabhu Joseph) > HBaseTimelineSchemaCreator has to allow user to configure pre-splits > > > Key: YARN-9373 > URL: https://issues.apache.org/jira/browse/YARN-9373 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Affects Versions: 3.2.0 >Reporter: Prabhu Joseph >Assignee: Ashutosh Gupta >Priority: Major > Attachments: Configurable_PreSplits.png, YARN-9373-001.patch, > YARN-9373-002.patch, YARN-9373-003.patch > > > Most of the TimelineService HBase tables is set with username splits which is > based on lowercase alphabet (a,ad,an,b,ca). This won't help if the rowkey > starts with either number or uppercase alphabet. We need to allow user to > configure based upon their data. For example, say a user has configured the > yarn.resourcemanager.cluster-id to be ATS or 123, then the splits can be > configured as A,B,C,,, or 100,200,300,,, -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9408) @Path("/apps/{appid}/appattempts") error message misleads
[
https://issues.apache.org/jira/browse/YARN-9408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557205#comment-17557205
]
Ashutosh Gupta commented on YARN-9408:
--
The issue seems to be pending for a while. Taking it up.
> @Path("/apps/{appid}/appattempts") error message misleads
> -
>
> Key: YARN-9408
> URL: https://issues.apache.org/jira/browse/YARN-9408
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9408-001.patch, YARN-9408-002.patch
>
>
> {code} @Path("/apps/{appid}/appattempts") {code} error message is misleading.
> NotFoundException "Unable to find the context flow name, and flow run id, and
> user id" is displayed while app attempts is looked.
> {code}
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0004/appattempts?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats,
> appId=application_1553258815132_0004",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/clusters/ats/apps/application_1553258815132_0005/appattempts?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats,
> appId=application_1553258815132_0005",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/clusters/ats1/apps/application_1553258815132_0001/containers/container_e14_1553258815132_0001_01_01?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats1,
> appId=application_1553258815132_0001",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/clusters/ats1/apps/application_1553258815132_0001/appattempts/appattempt_1553258815132_0001_01/containers?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats1,
> appId=application_1553258815132_0001",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9408) @Path("/apps/{appid}/appattempts") error message misleads
[
https://issues.apache.org/jira/browse/YARN-9408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta reassigned YARN-9408:
Assignee: Ashutosh Gupta (was: Prabhu Joseph)
> @Path("/apps/{appid}/appattempts") error message misleads
> -
>
> Key: YARN-9408
> URL: https://issues.apache.org/jira/browse/YARN-9408
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Minor
> Attachments: YARN-9408-001.patch, YARN-9408-002.patch
>
>
> {code} @Path("/apps/{appid}/appattempts") {code} error message is misleading.
> NotFoundException "Unable to find the context flow name, and flow run id, and
> user id" is displayed while app attempts is looked.
> {code}
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/apps/application_1553258815132_0004/appattempts?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats,
> appId=application_1553258815132_0004",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/clusters/ats/apps/application_1553258815132_0005/appattempts?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats,
> appId=application_1553258815132_0005",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/clusters/ats1/apps/application_1553258815132_0001/containers/container_e14_1553258815132_0001_01_01?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats1,
> appId=application_1553258815132_0001",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> [hbase@yarn-ats-3 ~]$ curl -s
> "http://yarn-ats-3:8198/ws/v2/timeline/clusters/ats1/apps/application_1553258815132_0001/appattempts/appattempt_1553258815132_0001_01/containers?user.name=hbase;
> | jq .
> {
> "exception": "NotFoundException",
> "message": "java.lang.Exception: Unable to find the context flow name, and
> flow run id, and user id for clusterId=ats1,
> appId=application_1553258815132_0001",
> "javaClassName": "org.apache.hadoop.yarn.webapp.NotFoundException"
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9822) TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down.
[
https://issues.apache.org/jira/browse/YARN-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557198#comment-17557198
]
Ashutosh Gupta commented on YARN-9822:
--
This is an important issue. As its pending for a while. Taking it up - making
the required changes and raising the PR
> TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down.
> -
>
> Key: YARN-9822
> URL: https://issues.apache.org/jira/browse/YARN-9822
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: YARN-9822-001.patch, YARN-9822-002.patch
>
>
> TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down.
> YARN-9374 prevents the threads getting blocked when it has already identified
> that Hbase down before accessing Hbase. TimelineCollector can check if the
> Writer Backend is up or down before locking the writer.
> {code}
> synchronized (writer) {
> response = writeTimelineEntities(entities, callerUgi);
> flushBufferedTimelineEntities();
> }
> {code}
> {code}
> "qtp183259297-80" #80 daemon prio=5 os_prio=0 tid=0x7f5f567fd000
> nid=0x5fbb waiting for monitor entry [0x7f5f236d4000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollector.putEntities(TimelineCollector.java:164)
> - waiting to lock <0x0006c7c05770> (a
> org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl)
> at
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorWebService.putEntities(TimelineCollectorWebService.java:186)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:304)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1624)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
[jira] [Assigned] (YARN-9822) TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down.
[
https://issues.apache.org/jira/browse/YARN-9822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta reassigned YARN-9822:
Assignee: Ashutosh Gupta (was: Prabhu Joseph)
> TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down.
> -
>
> Key: YARN-9822
> URL: https://issues.apache.org/jira/browse/YARN-9822
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Major
> Attachments: YARN-9822-001.patch, YARN-9822-002.patch
>
>
> TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down.
> YARN-9374 prevents the threads getting blocked when it has already identified
> that Hbase down before accessing Hbase. TimelineCollector can check if the
> Writer Backend is up or down before locking the writer.
> {code}
> synchronized (writer) {
> response = writeTimelineEntities(entities, callerUgi);
> flushBufferedTimelineEntities();
> }
> {code}
> {code}
> "qtp183259297-80" #80 daemon prio=5 os_prio=0 tid=0x7f5f567fd000
> nid=0x5fbb waiting for monitor entry [0x7f5f236d4000]
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollector.putEntities(TimelineCollector.java:164)
> - waiting to lock <0x0006c7c05770> (a
> org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl)
> at
> org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollectorWebService.putEntities(TimelineCollectorWebService.java:186)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:304)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1624)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
>
[jira] [Assigned] (YARN-9874) Remove unnecessary LevelDb write call in LeveldbConfigurationStore#confirmMutation
[
https://issues.apache.org/jira/browse/YARN-9874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta reassigned YARN-9874:
Assignee: Ashutosh Gupta (was: Prabhu Joseph)
> Remove unnecessary LevelDb write call in
> LeveldbConfigurationStore#confirmMutation
> --
>
> Key: YARN-9874
> URL: https://issues.apache.org/jira/browse/YARN-9874
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Minor
> Attachments: YARN-9874-001.patch, YARN-9874-002.patch
>
>
> Remove unnecessary LevelDb write call in
> LeveldbConfigurationStore#confirmMutation.
> {code}
> public void confirmMutation(boolean isValid) throws IOException {
> WriteBatch updateBatch = db.createWriteBatch();
> if (isValid) {
> ...
> }
> db.write(updateBatch);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9874) Remove unnecessary LevelDb write call in LeveldbConfigurationStore#confirmMutation
[
https://issues.apache.org/jira/browse/YARN-9874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557196#comment-17557196
]
Ashutosh Gupta commented on YARN-9874:
--
The issue seems to be pending for a while. Taking it up.
> Remove unnecessary LevelDb write call in
> LeveldbConfigurationStore#confirmMutation
> --
>
> Key: YARN-9874
> URL: https://issues.apache.org/jira/browse/YARN-9874
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments: YARN-9874-001.patch, YARN-9874-002.patch
>
>
> Remove unnecessary LevelDb write call in
> LeveldbConfigurationStore#confirmMutation.
> {code}
> public void confirmMutation(boolean isValid) throws IOException {
> WriteBatch updateBatch = db.createWriteBatch();
> if (isValid) {
> ...
> }
> db.write(updateBatch);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10320) Replace FSDataInputStream#read with readFully in Log Aggregation
[
https://issues.apache.org/jira/browse/YARN-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557173#comment-17557173
]
Ashutosh Gupta commented on YARN-10320:
---
This issue seems to be pending for a while. Taking it up.
> Replace FSDataInputStream#read with readFully in Log Aggregation
>
>
> Key: YARN-10320
> URL: https://issues.apache.org/jira/browse/YARN-10320
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Tanu Ajmera
>Priority: Major
> Attachments: YARN-10320-001.patch, YARN-10320-002.patch
>
>
> Have observed Log Aggregation code has used FSDataInputStream#read instead of
> readFully in multiple places like below. One of the place is fixed by
> YARN-8106.
> This Jira targets to fix at all other places.
> LogAggregationIndexedFileController#loadUUIDFromLogFile
> {code}
> byte[] b = new byte[uuid.length];
> int actual = fsDataInputStream.read(b);
> if (actual != uuid.length || Arrays.equals(b, uuid)) {
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10320) Replace FSDataInputStream#read with readFully in Log Aggregation
[
https://issues.apache.org/jira/browse/YARN-10320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta reassigned YARN-10320:
-
Assignee: Ashutosh Gupta (was: Tanu Ajmera)
> Replace FSDataInputStream#read with readFully in Log Aggregation
>
>
> Key: YARN-10320
> URL: https://issues.apache.org/jira/browse/YARN-10320
> Project: Hadoop YARN
> Issue Type: Bug
> Components: log-aggregation
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Major
> Attachments: YARN-10320-001.patch, YARN-10320-002.patch
>
>
> Have observed Log Aggregation code has used FSDataInputStream#read instead of
> readFully in multiple places like below. One of the place is fixed by
> YARN-8106.
> This Jira targets to fix at all other places.
> LogAggregationIndexedFileController#loadUUIDFromLogFile
> {code}
> byte[] b = new byte[uuid.length];
> int actual = fsDataInputStream.read(b);
> if (actual != uuid.length || Arrays.equals(b, uuid)) {
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9971) YARN Native Service HttpProbe logs THIS_HOST in error messages
[
https://issues.apache.org/jira/browse/YARN-9971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17553565#comment-17553565
]
Ashutosh Gupta commented on YARN-9971:
--
Issue seems to be pending for a while. Taking it up.
> YARN Native Service HttpProbe logs THIS_HOST in error messages
> --
>
> Key: YARN-9971
> URL: https://issues.apache.org/jira/browse/YARN-9971
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Tarun Parimi
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> YARN Native Service HttpProbe logs THIS_HOST in error messages. While
> logging, missed to use the replaced url string.
> {code:java}
> 2019-11-12 19:25:47,317 [pool-7-thread-1] INFO probe.HttpProbe - Probe
> http://${THIS_HOST}:18010/master-status failed for IP 172.27.75.198:
> java.net.ConnectException: Connection refused (Connection refused)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9971) YARN Native Service HttpProbe logs THIS_HOST in error messages
[
https://issues.apache.org/jira/browse/YARN-9971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta reassigned YARN-9971:
Assignee: Ashutosh Gupta (was: Tarun Parimi)
> YARN Native Service HttpProbe logs THIS_HOST in error messages
> --
>
> Key: YARN-9971
> URL: https://issues.apache.org/jira/browse/YARN-9971
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Ashutosh Gupta
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> YARN Native Service HttpProbe logs THIS_HOST in error messages. While
> logging, missed to use the replaced url string.
> {code:java}
> 2019-11-12 19:25:47,317 [pool-7-thread-1] INFO probe.HttpProbe - Probe
> http://${THIS_HOST}:18010/master-status failed for IP 172.27.75.198:
> java.net.ConnectException: Connection refused (Connection refused)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9827) Fix Http Response code in GenericExceptionHandler.
[ https://issues.apache.org/jira/browse/YARN-9827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta reassigned YARN-9827: Assignee: Ashutosh Gupta (was: Minni Mittal) > Fix Http Response code in GenericExceptionHandler. > -- > > Key: YARN-9827 > URL: https://issues.apache.org/jira/browse/YARN-9827 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Abhishek Modi >Assignee: Ashutosh Gupta >Priority: Major > > GenericExceptionHandler should respond with SERVICE_UNAVAILABLE in case of > connection and service unavailable exception instead of > INTERNAL_SERVICE_ERROR. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9827) Fix Http Response code in GenericExceptionHandler.
[ https://issues.apache.org/jira/browse/YARN-9827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544520#comment-17544520 ] Ashutosh Gupta commented on YARN-9827: -- Taking it up. > Fix Http Response code in GenericExceptionHandler. > -- > > Key: YARN-9827 > URL: https://issues.apache.org/jira/browse/YARN-9827 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Abhishek Modi >Assignee: Minni Mittal >Priority: Major > > GenericExceptionHandler should respond with SERVICE_UNAVAILABLE in case of > connection and service unavailable exception instead of > INTERNAL_SERVICE_ERROR. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11115) Add configuration to disable AM preemption for capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-5?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta reassigned YARN-5: - Assignee: Ashutosh Gupta (was: Junfan Zhang) > Add configuration to disable AM preemption for capacity scheduler > - > > Key: YARN-5 > URL: https://issues.apache.org/jira/browse/YARN-5 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Yuan Luo >Assignee: Ashutosh Gupta >Priority: Major > > I think it's necessary to add configuration to disable AM preemption for > capacity-scheduler, like fair-scheduler feature: YARN-9537. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11115) Add configuration to disable AM preemption for capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541500#comment-17541500 ] Ashutosh Gupta commented on YARN-5: --- Taking it up. > Add configuration to disable AM preemption for capacity scheduler > - > > Key: YARN-5 > URL: https://issues.apache.org/jira/browse/YARN-5 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Yuan Luo >Assignee: Junfan Zhang >Priority: Major > > I think it's necessary to add configuration to disable AM preemption for > capacity-scheduler, like fair-scheduler feature: YARN-9537. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11115) Add configuration to disable AM preemption for capacity scheduler
[ https://issues.apache.org/jira/browse/YARN-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17540324#comment-17540324 ] Ashutosh Gupta commented on YARN-5: --- [~zuston] - Are you working on it or I can take it over? > Add configuration to disable AM preemption for capacity scheduler > - > > Key: YARN-5 > URL: https://issues.apache.org/jira/browse/YARN-5 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Yuan Luo >Assignee: Junfan Zhang >Priority: Major > > I think it's necessary to add configuration to disable AM preemption for > capacity-scheduler, like fair-scheduler feature: YARN-9537. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11142) Remove unused Imports in Hadoop YARN project
Ashutosh Gupta created YARN-11142: - Summary: Remove unused Imports in Hadoop YARN project Key: YARN-11142 URL: https://issues.apache.org/jira/browse/YARN-11142 Project: Hadoop YARN Issue Type: Improvement Reporter: Ashutosh Gupta Assignee: Ashutosh Gupta h3. Optimize Imports to keep code clean # Remove any unused imports -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11125) Backport YARN-6483 to branch-2.10
[ https://issues.apache.org/jira/browse/YARN-11125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated YARN-11125: -- Component/s: (was: yarn) > Backport YARN-6483 to branch-2.10 > - > > Key: YARN-11125 > URL: https://issues.apache.org/jira/browse/YARN-11125 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Backport YARN-6483 to branch-2.10 -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11135) [Umbrella] Remove unused Imports in hadoop project
[ https://issues.apache.org/jira/browse/YARN-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated YARN-11135: -- Summary: [Umbrella] Remove unused Imports in hadoop project (was: Remove unused Imports in hadoop project) > [Umbrella] Remove unused Imports in hadoop project > -- > > Key: YARN-11135 > URL: https://issues.apache.org/jira/browse/YARN-11135 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > h3. Optimize Imports to keep code clean > # Remove any unused imports > # -Sort the import statements.- > # -Remove .* imports- -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11135) Optimize Imports in hadoop project
[ https://issues.apache.org/jira/browse/YARN-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated YARN-11135: -- Description: h3. Optimize Imports to keep code clean # Remove any unused imports # -Sort the import statements.- # -Remove .* imports- was: h3. Optimize Imports to keep code clean # Remove any unused imports # Sort the import statements. # Remove .* imports > Optimize Imports in hadoop project > -- > > Key: YARN-11135 > URL: https://issues.apache.org/jira/browse/YARN-11135 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > h3. Optimize Imports to keep code clean > # Remove any unused imports > # -Sort the import statements.- > # -Remove .* imports- -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11135) Remove unused Imports in hadoop project
[ https://issues.apache.org/jira/browse/YARN-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated YARN-11135: -- Summary: Remove unused Imports in hadoop project (was: Optimize Imports in hadoop project) > Remove unused Imports in hadoop project > --- > > Key: YARN-11135 > URL: https://issues.apache.org/jira/browse/YARN-11135 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > h3. Optimize Imports to keep code clean > # Remove any unused imports > # -Sort the import statements.- > # -Remove .* imports- -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11135) Optimize Imports in hadoop project
[ https://issues.apache.org/jira/browse/YARN-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated YARN-11135: -- Description: h3. Optimize Imports to keep code clean # Remove any unused imports # Sort the import statements. # Remove .* imports was: h3. Optimize Imports to keep code clean # Remove any unused imports, # Sort the import statements. > Optimize Imports in hadoop project > -- > > Key: YARN-11135 > URL: https://issues.apache.org/jira/browse/YARN-11135 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > > h3. Optimize Imports to keep code clean > # Remove any unused imports > # Sort the import statements. > # Remove .* imports -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11135) Optimize Imports in hadoop project
Ashutosh Gupta created YARN-11135: - Summary: Optimize Imports in hadoop project Key: YARN-11135 URL: https://issues.apache.org/jira/browse/YARN-11135 Project: Hadoop YARN Issue Type: New Feature Reporter: Ashutosh Gupta Assignee: Ashutosh Gupta h3. Optimize Imports to keep code clean # Remove any unused imports, # Sort the import statements. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10080) Support show app id on localizer thread pool
[ https://issues.apache.org/jira/browse/YARN-10080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta reassigned YARN-10080: - Assignee: Ashutosh Gupta (was: zhoukang) > Support show app id on localizer thread pool > > > Key: YARN-10080 > URL: https://issues.apache.org/jira/browse/YARN-10080 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: zhoukang >Assignee: Ashutosh Gupta >Priority: Major > Attachments: YARN-10080-001.patch, YARN-10080.002.patch > > > Currently when we are troubleshooting a container localizer issue, if we want > to analyze the jstack with thread detail, we can not figure out which thread > is processing the given container. So i want to add app id on the thread name -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10080) Support show app id on localizer thread pool
[ https://issues.apache.org/jira/browse/YARN-10080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533382#comment-17533382 ] Ashutosh Gupta commented on YARN-10080: --- Taking it up. > Support show app id on localizer thread pool > > > Key: YARN-10080 > URL: https://issues.apache.org/jira/browse/YARN-10080 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: zhoukang >Assignee: zhoukang >Priority: Major > Attachments: YARN-10080-001.patch, YARN-10080.002.patch > > > Currently when we are troubleshooting a container localizer issue, if we want > to analyze the jstack with thread detail, we can not figure out which thread > is processing the given container. So i want to add app id on the thread name -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9355) RMContainerRequestor#makeRemoteRequest has confusing log message
[
https://issues.apache.org/jira/browse/YARN-9355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533380#comment-17533380
]
Ashutosh Gupta commented on YARN-9355:
--
Taking it up.
> RMContainerRequestor#makeRemoteRequest has confusing log message
>
>
> Key: YARN-9355
> URL: https://issues.apache.org/jira/browse/YARN-9355
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Umesh Mittal
>Priority: Trivial
> Labels: newbie, newbie++
> Attachments: YARN-9355.001.patch, YARN-9355.002.patch
>
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor#makeRemoteRequest
> has this log:
> {code:java}
> if (ask.size() > 0 || release.size() > 0) {
> LOG.info("getResources() for " + applicationId + ":" + " ask="
> + ask.size() + " release= " + release.size() + " newContainers="
> + allocateResponse.getAllocatedContainers().size()
> + " finishedContainers=" + numCompletedContainers
> + " resourcelimit=" + availableResources + " knownNMs="
> + clusterNmCount);
> }
> {code}
> The reason why "getResources()" is printed because
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator#getResources
> invokes makeRemoteRequest. This is not too informative and error-prone as
> name of getResources could change over time and the log will be outdated.
> Moreover, it's not a good idea to print a method name from a method below the
> current one in the stack.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9355) RMContainerRequestor#makeRemoteRequest has confusing log message
[
https://issues.apache.org/jira/browse/YARN-9355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta reassigned YARN-9355:
Assignee: Ashutosh Gupta (was: Umesh Mittal)
> RMContainerRequestor#makeRemoteRequest has confusing log message
>
>
> Key: YARN-9355
> URL: https://issues.apache.org/jira/browse/YARN-9355
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Ashutosh Gupta
>Priority: Trivial
> Labels: newbie, newbie++
> Attachments: YARN-9355.001.patch, YARN-9355.002.patch
>
>
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor#makeRemoteRequest
> has this log:
> {code:java}
> if (ask.size() > 0 || release.size() > 0) {
> LOG.info("getResources() for " + applicationId + ":" + " ask="
> + ask.size() + " release= " + release.size() + " newContainers="
> + allocateResponse.getAllocatedContainers().size()
> + " finishedContainers=" + numCompletedContainers
> + " resourcelimit=" + availableResources + " knownNMs="
> + clusterNmCount);
> }
> {code}
> The reason why "getResources()" is printed because
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator#getResources
> invokes makeRemoteRequest. This is not too informative and error-prone as
> name of getResources could change over time and the log will be outdated.
> Moreover, it's not a good idea to print a method name from a method below the
> current one in the stack.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11128) Fix comments in TestProportionalCapacityPreemptionPolicy*
Ashutosh Gupta created YARN-11128: - Summary: Fix comments in TestProportionalCapacityPreemptionPolicy* Key: YARN-11128 URL: https://issues.apache.org/jira/browse/YARN-11128 Project: Hadoop YARN Issue Type: New Feature Components: capacityscheduler Reporter: Ashutosh Gupta At various places, comment for appsConfig is `// queueName\t(priority,resource,host,expression,#repeat,reserved,pending)` but should be `// queueName\t(priority,resource,host,expression,#repeat,reserved,pending)` -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11128) Fix comments in TestProportionalCapacityPreemptionPolicy*
[ https://issues.apache.org/jira/browse/YARN-11128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta reassigned YARN-11128: - Assignee: Ashutosh Gupta > Fix comments in TestProportionalCapacityPreemptionPolicy* > - > > Key: YARN-11128 > URL: https://issues.apache.org/jira/browse/YARN-11128 > Project: Hadoop YARN > Issue Type: New Feature > Components: capacityscheduler >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Minor > > At various places, comment for appsConfig is > `// queueName\t(priority,resource,host,expression,#repeat,reserved,pending)` > but should be > `// queueName\t(priority,resource,host,expression,#repeat,reserved,pending)` > -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11124) Backport YARN-6483 to branch-2.10
[ https://issues.apache.org/jira/browse/YARN-11124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531007#comment-17531007 ] Ashutosh Gupta commented on YARN-11124: --- Closed this as this was created duplicate of https://issues.apache.org/jira/browse/YARN-11125 > Backport YARN-6483 to branch-2.10 > - > > Key: YARN-11124 > URL: https://issues.apache.org/jira/browse/YARN-11124 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Ashutosh Gupta >Priority: Major > > Backport YARN-6483 to branch-2.10 -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11124) Backport YARN-6483 to branch-2.10
[ https://issues.apache.org/jira/browse/YARN-11124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta resolved YARN-11124. --- Resolution: Duplicate > Backport YARN-6483 to branch-2.10 > - > > Key: YARN-11124 > URL: https://issues.apache.org/jira/browse/YARN-11124 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Ashutosh Gupta >Priority: Major > > Backport YARN-6483 to branch-2.10 -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11125) Backport YARN-6483 to branch-2.10
[ https://issues.apache.org/jira/browse/YARN-11125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531004#comment-17531004 ] Ashutosh Gupta commented on YARN-11125: --- As part of this PR: [https://github.com/apache/hadoop/pull/4244] > Backport YARN-6483 to branch-2.10 > - > > Key: YARN-11125 > URL: https://issues.apache.org/jira/browse/YARN-11125 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Backport YARN-6483 to branch-2.10 -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11125) Backport YARN-6483 to branch-2.10
[ https://issues.apache.org/jira/browse/YARN-11125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta reassigned YARN-11125: - Assignee: Ashutosh Gupta > Backport YARN-6483 to branch-2.10 > - > > Key: YARN-11125 > URL: https://issues.apache.org/jira/browse/YARN-11125 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > > Backport YARN-6483 to branch-2.10 -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11125) Backport YARN-6483 to branch-2.10
Ashutosh Gupta created YARN-11125: - Summary: Backport YARN-6483 to branch-2.10 Key: YARN-11125 URL: https://issues.apache.org/jira/browse/YARN-11125 Project: Hadoop YARN Issue Type: Bug Components: yarn Reporter: Ashutosh Gupta Backport YARN-6483 to branch-2.10 -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11124) Backport YARN-6483 to branch-2.10
Ashutosh Gupta created YARN-11124: - Summary: Backport YARN-6483 to branch-2.10 Key: YARN-11124 URL: https://issues.apache.org/jira/browse/YARN-11124 Project: Hadoop YARN Issue Type: Bug Components: yarn Reporter: Ashutosh Gupta Backport YARN-6483 to branch-2.10 -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11092) Upgrade jquery ui to 1.13.1
[ https://issues.apache.org/jira/browse/YARN-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta reassigned YARN-11092: - Assignee: Ashutosh Gupta (was: D M Murali Krishna Reddy) > Upgrade jquery ui to 1.13.1 > --- > > Key: YARN-11092 > URL: https://issues.apache.org/jira/browse/YARN-11092 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Assignee: Ashutosh Gupta >Priority: Major > > The current jquery-ui version used(1.12.1) in the trunk has the following > vulnerabilities CVE-2021-41182, CVE-2021-41183, CVE-2021-41184, so we need to > upgrade to at least 1.13.0. > > Also currently for the UI2 we are using the shims repo which is not being > maintained as per the discussion > [https://github.com/components/jqueryui/issues/70] , so if possible we should > move to the main jquery repo [https://github.com/jquery/jquery-ui] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10538) Add recommissioning nodes to the list of updated nodes returned to the AM
[ https://issues.apache.org/jira/browse/YARN-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529499#comment-17529499 ] Ashutosh Gupta edited comment on YARN-10538 at 4/28/22 3:32 PM: Hi [~aajisaka] - I have raise the PR - https://github.com/apache/hadoop/pull/4244 was (Author: groot): Hi [~aajisaka] - I have raise the PR - [https://github.com/apache/hadoop/pull/4244|https://github.com/apache/hadoop/pull/4244.] > Add recommissioning nodes to the list of updated nodes returned to the AM > - > > Key: YARN-10538 > URL: https://issues.apache.org/jira/browse/YARN-10538 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.9.1, 3.1.1 >Reporter: Srinivas S T >Assignee: Srinivas S T >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.1, 3.2.3 > > Time Spent: 2h > Remaining Estimate: 0h > > YARN-6483 introduced nodes that transitioned to DECOMMISSIONING state to the > list of updated nodes returned to the AM. This allows the Spark application > master to gracefully decommission its containers on the decommissioning node. > But if the node were to be recommissioned, the Spark application master would > not be aware of this. We propose to add recommissioned node to the list of > updated nodes sent to the AM when a recommission node transition occurs. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10538) Add recommissioning nodes to the list of updated nodes returned to the AM
[ https://issues.apache.org/jira/browse/YARN-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17529499#comment-17529499 ] Ashutosh Gupta commented on YARN-10538: --- Hi [~aajisaka] - I have raise the PR - [https://github.com/apache/hadoop/pull/4244|https://github.com/apache/hadoop/pull/4244.] > Add recommissioning nodes to the list of updated nodes returned to the AM > - > > Key: YARN-10538 > URL: https://issues.apache.org/jira/browse/YARN-10538 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.9.1, 3.1.1 >Reporter: Srinivas S T >Assignee: Srinivas S T >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.1, 3.2.3 > > Time Spent: 2h > Remaining Estimate: 0h > > YARN-6483 introduced nodes that transitioned to DECOMMISSIONING state to the > list of updated nodes returned to the AM. This allows the Spark application > master to gracefully decommission its containers on the decommissioning node. > But if the node were to be recommissioned, the Spark application master would > not be aware of this. We propose to add recommissioned node to the list of > updated nodes sent to the AM when a recommission node transition occurs. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11119) Backport YARN-10538 to branch-2.10
[ https://issues.apache.org/jira/browse/YARN-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Gupta updated YARN-9: -- Description: Backport YARN-10538 to branch-2.10 Add recommissioning nodes to the list of updated nodes returned to the AM was:Backport YARN-10538 to branch-2.10 > Backport YARN-10538 to branch-2.10 > -- > > Key: YARN-9 > URL: https://issues.apache.org/jira/browse/YARN-9 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Ashutosh Gupta >Assignee: Ashutosh Gupta >Priority: Major > > Backport YARN-10538 to branch-2.10 > > Add recommissioning nodes to the list of updated nodes returned to the AM -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11119) Backport YARN-10538 to branch-2.10
Ashutosh Gupta created YARN-9: - Summary: Backport YARN-10538 to branch-2.10 Key: YARN-9 URL: https://issues.apache.org/jira/browse/YARN-9 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.10.0 Reporter: Ashutosh Gupta Assignee: Ashutosh Gupta Backport YARN-10538 to branch-2.10 -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10538) Add recommissioning nodes to the list of updated nodes returned to the AM
[ https://issues.apache.org/jira/browse/YARN-10538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528753#comment-17528753 ] Ashutosh Gupta commented on YARN-10538: --- [~aajisaka] - Shall we backport it to branch-2.10 as well? This seems like a really useful fix. > Add recommissioning nodes to the list of updated nodes returned to the AM > - > > Key: YARN-10538 > URL: https://issues.apache.org/jira/browse/YARN-10538 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.9.1, 3.1.1 >Reporter: Srinivas S T >Assignee: Srinivas S T >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.1, 3.2.3 > > Time Spent: 2h > Remaining Estimate: 0h > > YARN-6483 introduced nodes that transitioned to DECOMMISSIONING state to the > list of updated nodes returned to the AM. This allows the Spark application > master to gracefully decommission its containers on the decommissioning node. > But if the node were to be recommissioned, the Spark application master would > not be aware of this. We propose to add recommissioned node to the list of > updated nodes sent to the AM when a recommission node transition occurs. -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10080) Support show app id on localizer thread pool
[ https://issues.apache.org/jira/browse/YARN-10080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17528458#comment-17528458 ] Ashutosh Gupta commented on YARN-10080: --- [~cane] - This seems to be useful while debugging. Can you raise a PR or I can do that? > Support show app id on localizer thread pool > > > Key: YARN-10080 > URL: https://issues.apache.org/jira/browse/YARN-10080 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: zhoukang >Assignee: zhoukang >Priority: Major > Attachments: YARN-10080-001.patch, YARN-10080.002.patch > > > Currently when we are troubleshooting a container localizer issue, if we want > to analyze the jstack with thread detail, we can not figure out which thread > is processing the given container. So i want to add app id on the thread name -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11092) Upgrade jquery ui to 1.13.1
[ https://issues.apache.org/jira/browse/YARN-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17527262#comment-17527262 ] Ashutosh Gupta commented on YARN-11092: --- Hi [~dmmkr] - Thanks for reporting. Are you planning to raise a CR to upgrade it or shall I do it? > Upgrade jquery ui to 1.13.1 > --- > > Key: YARN-11092 > URL: https://issues.apache.org/jira/browse/YARN-11092 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > > The current jquery-ui version used(1.12.1) in the trunk has the following > vulnerabilities CVE-2021-41182, CVE-2021-41183, CVE-2021-41184, so we need to > upgrade to at least 1.13.0. > > Also currently for the UI2 we are using the shims repo which is not being > maintained as per the discussion > [https://github.com/components/jqueryui/issues/70] , so if possible we should > move to the main jquery repo [https://github.com/jquery/jquery-ui] -- This message was sent by Atlassian Jira (v8.20.7#820007) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9429) A status code error in ResourceManager REST api doc
[
https://issues.apache.org/jira/browse/YARN-9429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17526848#comment-17526848
]
Ashutosh Gupta commented on YARN-9429:
--
Taking it up.
> A status code error in ResourceManager REST api doc
> ---
>
> Key: YARN-9429
> URL: https://issues.apache.org/jira/browse/YARN-9429
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Reporter: Jinjiang Ling
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Attachments: YARN-9429.001.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> A status code error in ResourceManager api docs.
> In section "Cluster Application State API",the unauthorized error response
> header is described below.
> {code:java}
> Response Header:
> HTTP/1.1 403 Unauthorized
> Server: Jetty(6.1.26)
> {code}
> As commonly known, the unauthorized status code should be *401*.
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9429) A status code error in ResourceManager REST api doc
[
https://issues.apache.org/jira/browse/YARN-9429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Gupta reassigned YARN-9429:
Assignee: Ashutosh Gupta (was: Jinjiang Ling)
> A status code error in ResourceManager REST api doc
> ---
>
> Key: YARN-9429
> URL: https://issues.apache.org/jira/browse/YARN-9429
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Reporter: Jinjiang Ling
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
> Attachments: YARN-9429.001.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> A status code error in ResourceManager api docs.
> In section "Cluster Application State API",the unauthorized error response
> header is described below.
> {code:java}
> Response Header:
> HTTP/1.1 403 Unauthorized
> Server: Jetty(6.1.26)
> {code}
> As commonly known, the unauthorized status code should be *401*.
>
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
