[jira] [Commented] (YARN-10383) YarnCommands.md is inconsistent with the source code
[ https://issues.apache.org/jira/browse/YARN-10383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17170526#comment-17170526 ] Hudson commented on YARN-10383: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18491 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18491/]) YARN-10383. YarnCommands.md is inconsistent with the source code (#2177) (github: rev ab2b3df2de1d7c490a4aa83460328bcd910c39d4) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md > YarnCommands.md is inconsistent with the source code > > > Key: YARN-10383 > URL: https://issues.apache.org/jira/browse/YARN-10383 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Reporter: zhaoshengjie >Assignee: zhaoshengjie >Priority: Minor > Fix For: 3.4.0 > > Attachments: 01.png, 02.png > > > The markdown document (YarnCommands.md) is inconsistent with the source code > !01.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10229) [Federation] Client should be able to submit application to RM directly using normal client conf
[ https://issues.apache.org/jira/browse/YARN-10229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17169793#comment-17169793 ] Hudson commented on YARN-10229: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18489 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18489/]) YARN-10229. [Federation] Client should be able to submit application to (brahma: rev eac558380fd7d3c2e78b8956e2080688bb1dd8bb) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java > [Federation] Client should be able to submit application to RM directly using > normal client conf > > > Key: YARN-10229 > URL: https://issues.apache.org/jira/browse/YARN-10229 > Project: Hadoop YARN > Issue Type: Bug > Components: amrmproxy, federation >Affects Versions: 3.1.1 >Reporter: JohnsonGuo >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10229.001.patch, YARN-10229.002.patch, > YARN-10229.003.patch, YARN-10229.004.patch, YARN-10229.005.patch, > YARN-10229.006.patch, YARN-10229.007.patch, YARN-10229.008.patch > > > Scenario： When enable the yarn federation feature with multi yarn clusters, > one can submit their job to yarn-router by *modified* their client > configuration with yarn router address. > But if one still wants to submit their jobs via the original client (before > enable federation) to RM directly, it will encounter the AMRMToken exception. > That means once enable federation ,if some one want to submit job, they have > to modify the client conf. > > one possible solution for this Scenario is: > In NodeManger, when the client ApplicationMaster request comes: > * get the client job.xml from HDFS "". > * parse the "yarn.resourcemanager.scheduler.address" parameter in job.xml > * if the value of the parameter is "localhost:8049"(AMRM address),then do > the AMRMToken valid process > * if the value of the parameter is "rm:port"(rm address),then skip the > AMRMToken valid process > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10359) Log container report only if list is not empty
[
https://issues.apache.org/jira/browse/YARN-10359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17169247#comment-17169247
]
Hudson commented on YARN-10359:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18485 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18485/])
YARN-10359. Log container report only if list is not empty. Contributed
(bibinchundatt: rev 5323e83edfe63355ec38ffdaacc0c27d14cad31c)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
> Log container report only if list is not empty
> --
>
> Key: YARN-10359
> URL: https://issues.apache.org/jira/browse/YARN-10359
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: YARN-10359.001.patch, YARN-10359.002.patch
>
>
> In NodeStatusUpdaterImpl print log only if containerReports list is not empty
> {code:java}
> if (containerReports != null) {
> LOG.info("Registering with RM using containers :" + containerReports);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1529) Add Localization overhead metrics to NM
[ https://issues.apache.org/jira/browse/YARN-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168064#comment-17168064 ] Hudson commented on YARN-1529: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18481 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18481/]) YARN-1529: Add Localization overhead metrics to NM. Contributed by (ericp: rev e0c9653166df48a47267dbc81d124ab78267e039) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/Container.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerResourceLocalizedEvent.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalizedResource.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/MockContainer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java > Add Localization overhead metrics to NM > --- > > Key: YARN-1529 > URL: https://issues.apache.org/jira/browse/YARN-1529 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Gera Shegalov >Assignee: Jim Brennan >Priority: Major > Attachments: YARN-1529.005.patch, YARN-1529.006.patch, > YARN-1529.v01.patch, YARN-1529.v02.patch, YARN-1529.v03.patch, > YARN-1529.v04.patch > > > Users are often unaware of localization cost that their jobs incur. To > measure effectiveness of localization caches it is necessary to expose the > overhead in the form of metrics. > We propose addition of the following metrics to NodeManagerMetrics. > When a container is about to launch, its set of LocalResources has to be > fetched from a central location, typically on HDFS, that results in a number > of download requests for the files missing in caches. > LocalizedFilesMissed: total files (requests) downloaded from DFS. Cache > misses. > LocalizedFilesCached: total localization requests that were served from local > caches. Cache hits. > LocalizedBytesMissed: total bytes downloaded from DFS due to cache misses. > LocalizedBytesCached: total bytes satisfied from local caches. > Localized(Files|Bytes)CachedRatio: percentage of localized (files|bytes) that > were served out of cache: ratio = 100 * caches / (caches + misses) > LocalizationDownloadNanos: total elapsed time in nanoseconds for a container > to go from ResourceRequestTransition to LocalizedTransition -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10369) Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG
[
https://issues.apache.org/jira/browse/YARN-10369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17167426#comment-17167426
]
Hudson commented on YARN-10369:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18479 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18479/])
YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId
(bibinchundatt: rev 5d8600e80ad7864b332b60d5a01585fdf00848ee)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/NMTokenSecretManagerInRM.java
> Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG
> --
>
> Key: YARN-10369
> URL: https://issues.apache.org/jira/browse/YARN-10369
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: YARN-10369.001.patch, YARN-10369.002.patch
>
>
> This message is logged at the info level, but it doesn't really add much
> information.
> We changed this to DEBUG internally years ago and haven't missed it.
> {noformat}
> 2020-07-27 21:51:29,027 INFO [RM Event dispatcher]
> security.NMTokenSecretManagerInRM
> (NMTokenSecretManagerInRM.java:createAndGetNMToken(200)) - Sending NMToken
> for nodeId : localhost.localdomain:45454 for container :
> container_1595886659189_0001_01_01
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10343) Legacy RM UI should include labeled metrics for allocated, total, and reserved resources.
[ https://issues.apache.org/jira/browse/YARN-10343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166693#comment-17166693 ] Hudson commented on YARN-10343: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18477 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18477/]) YARN-10343. Legacy RM UI should include labeled metrics for allocated, (jhung: rev 3eaf62726ffe90b3b096798fe501abd1ed0c5f15) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceUsage.java > Legacy RM UI should include labeled metrics for allocated, total, and > reserved resources. > - > > Key: YARN-10343 > URL: https://issues.apache.org/jira/browse/YARN-10343 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.10.0, 3.2.1, 3.1.3 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > Fix For: 3.2.2, 2.10.1, 3.4.0, 3.3.1, 3.1.5 > > Attachments: Screen Shot 2020-07-07 at 1.00.22 PM.png, Screen Shot > 2020-07-07 at 1.03.26 PM.png, YARN-10343.000.patch, YARN-10343.001.patch, > YARN-10343.branch-2.10.001.patch, YARN-10343.branch-3.2.001.patch > > > The current legacy RM UI only includes resources metrics for the default > partition. If a cluster has labeled nodes, those are not included in the > resource metrics for allocated, total, and reserved resources. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10208) Add capacityScheduler metric for NODE_UPDATE interval
[ https://issues.apache.org/jira/browse/YARN-10208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166337#comment-17166337 ] Hudson commented on YARN-10208: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18476 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18476/]) YARN-10208. Add capacityScheduler metric for NODE_UPDATE interval. (bibinchundatt: rev 5dadf963d3639cc6d37902d9c7beaacdafac0e9c) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerMetrics.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestCapacitySchedulerMetrics.java > Add capacityScheduler metric for NODE_UPDATE interval > - > > Key: YARN-10208 > URL: https://issues.apache.org/jira/browse/YARN-10208 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Pranjal Protim Borah >Assignee: Pranjal Protim Borah >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10208.001.patch, YARN-10208.002.patch, > YARN-10208.003.patch, YARN-10208.004.patch, YARN-10208.005.patch, > YARN-10208.006.patch, YARN-10208.007.patch > > > Metric measuring average time interval between node heartbeats in capacity > scheduler on node update event. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10366) Yarn rmadmin help message shows two labels for one node for --replaceLabelsOnNode
[ https://issues.apache.org/jira/browse/YARN-10366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165488#comment-17165488 ] Hudson commented on YARN-10366: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18474 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18474/]) YARN-10366. Fix Yarn rmadmin help message shows two labels for one node (pjoseph: rev 60a254621a3d07a6c3c4611ea86cffd9625e8fb2) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java > Yarn rmadmin help message shows two labels for one node for > --replaceLabelsOnNode > - > > Key: YARN-10366 > URL: https://issues.apache.org/jira/browse/YARN-10366 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Tanu Ajmera >Assignee: Tanu Ajmera >Priority: Major > Fix For: 3.4.0 > > Attachments: Screenshot 2020-07-24 at 4.07.10 PM.png, > YARN-10366-001.patch > > > In the help message of “yarn rmadmin” , looks like one node can be assign > with two labels, which is not consistent with the “Each node can have only > one node label” -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10362) Javadoc for TimelineReaderAuthenticationFilterInitializer is broken
[
https://issues.apache.org/jira/browse/YARN-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165291#comment-17165291
]
Hudson commented on YARN-10362:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18473 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18473/])
YARN-10362. Javadoc for TimelineReaderAuthenticationFilterInitializer is
(aajisaka: rev d02be17a269cb7f5ac51910802683a1729510250)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/security/TimelineReaderAuthenticationFilterInitializer.java
> Javadoc for TimelineReaderAuthenticationFilterInitializer is broken
> ---
>
> Key: YARN-10362
> URL: https://issues.apache.org/jira/browse/YARN-10362
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: HADOOP-17148.000.patch
>
>
> mvn javadoc:javadoc fails for
> TimelineReaderAuthenticationFilterInitializer.java
> {code:java}
> [ERROR]
> /Users/sri/projects/hadoop-mirror/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/security/TimelineReaderAuthenticationFilterInitializer.java:39:
> error: value does not refer to a constant
> [ERROR]* {@value TimelineAuthenticationFilterInitializer#PREFIX}.
> [ERROR] ^
> [ERROR]
> /Users/sri/projects/hadoop-mirror/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/security/TimelineReaderAuthenticationFilterInitializer.java:39:
> error: reference not found
> [ERROR]* {@value TimelineAuthenticationFilterInitializer#PREFIX}.
> {code}
> This issue seems to be caused by changes in YARN-10339
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10367) Failed to get nodejs 10.21.0 when building docker image
[
https://issues.apache.org/jira/browse/YARN-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165255#comment-17165255
]
Hudson commented on YARN-10367:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18472 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18472/])
YARN-10367. Failed to get nodejs 10.21.0 when building docker image (github:
rev e277d338da2675262b4d6985e08c78e408b927a3)
* (edit) dev-support/docker/Dockerfile_aarch64
* (edit) dev-support/docker/Dockerfile
> Failed to get nodejs 10.21.0 when building docker image
> ---
>
> Key: YARN-10367
> URL: https://issues.apache.org/jira/browse/YARN-10367
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build, webapp
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Blocker
> Fix For: 3.4.0, 3.3.1
>
>
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/213/console
> {noformat}
> Version '10.21.0-1nodesource1' for 'nodejs' was not found
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4771) Some containers can be skipped during log aggregation after NM restart
[ https://issues.apache.org/jira/browse/YARN-4771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164670#comment-17164670 ] Hudson commented on YARN-4771: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18470 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18470/]) YARN-4771. Some containers can be skipped during log aggregation after (ebadger: rev ac5f21dbef0f0ad4210e4027f53877760fa606a5) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java > Some containers can be skipped during log aggregation after NM restart > -- > > Key: YARN-4771 > URL: https://issues.apache.org/jira/browse/YARN-4771 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.10.0, 3.2.1, 3.1.3 >Reporter: Jason Darrell Lowe >Assignee: Jim Brennan >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-4771.001.patch, YARN-4771.002.patch, > YARN-4771.003.patch > > > A container can be skipped during log aggregation after a work-preserving > nodemanager restart if the following events occur: > # Container completes more than > yarn.nodemanager.duration-to-track-stopped-containers milliseconds before the > restart > # At least one other container completes after the above container and before > the restart -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10319) Record Last N Scheduler Activities from ActivitiesManager
[ https://issues.apache.org/jira/browse/YARN-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17164184#comment-17164184 ] Hudson commented on YARN-10319: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18468 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18468/]) YARN-10319. Record Last N Scheduler Activities from ActivitiesManager (pjoseph: rev 247eb0979b6a3a723ea9a249ba4db1ee079eb909) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/RouterWebServices.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/FederationInterceptorREST.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/DefaultRequestInterceptorREST.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWSConsts.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesSchedulerActivities.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/JAXBContextResolver.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/ActivitiesTestUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServiceProtocol.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/MockRESTRequestInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/PassThroughRESTRequestInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/activities/ActivitiesManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/BulkActivitiesInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/BaseRouterWebServicesTest.java > Record Last N Scheduler Activities from ActivitiesManager > - > > Key: YARN-10319 > URL: https://issues.apache.org/jira/browse/YARN-10319 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: activitiesmanager > Fix For: 3.4.0 > > Attachments: Screen Shot 2020-06-18 at 1.26.31 PM.png, > YARN-10319-001-WIP.patch, YARN-10319-002.patch, YARN-10319-003.patch, > YARN-10319-004.patch, YARN-10319-005.patch, YARN-10319-006.patch > > > ActivitiesManager records a call flow for a given nodeId or a last call flow. > This is useful when debugging the issue live where the user queries with > right nodeId. But capturing last N scheduler activities during the issue > period can help to debug the issue offline. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10315) Avoid sending RMNodeResourceupdate event if resource is same
[ https://issues.apache.org/jira/browse/YARN-10315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163460#comment-17163460 ] Hudson commented on YARN-10315: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18467 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18467/]) YARN-10315. Avoid sending RMNodeResourceupdate event if resource is (bibinchundatt: rev bfcd775381f1e0b94b17ce3cfca7eade95df1ea8) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java > Avoid sending RMNodeResourceupdate event if resource is same > > > Key: YARN-10315 > URL: https://issues.apache.org/jira/browse/YARN-10315 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin Chundatt >Assignee: Sushil Ks >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10315.001.patch, YARN-10315.002.patch > > > When the node is in DECOMMISSIONING state the RMNodeResourceUpdateEvent is > send for every heartbeat . Which will result in scheduler resource update. > Avoid sending the same. > Scheduler node resource update iterates through all the queues for resource > update which is costly.. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10353) Log vcores used and cumulative cpu in containers monitor
[
https://issues.apache.org/jira/browse/YARN-10353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17161450#comment-17161450
]
Hudson commented on YARN-10353:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18456 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18456/])
[YARN-10353] Log vcores used and cumulative cpu in containers monitor.
(ebadger: rev 736bed6d6d20a17b522a0686ca3fd2d97e7e6838)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
> Log vcores used and cumulative cpu in containers monitor
>
>
> Key: YARN-10353
> URL: https://issues.apache.org/jira/browse/YARN-10353
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.4.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: YARN-10353.001.patch, YARN-10353.002.patch
>
>
> We currently log the percentage/cpu and percentage/cpus-used-by-yarn in the
> Containers Monitor log. It would be useful to also log vcores used vs vcores
> assigned, and total accumulated CPU time.
> For example, currently we have an audit log that looks like this:
> {noformat}
> 2020-07-16 20:33:51,550 DEBUG [Container Monitor] ContainersMonitorImpl.audit
> (ContainersMonitorImpl.java:recordUsage(651)) - Resource usage of ProcessTree
> 809 for container-id container_1594931466123_0002_01_07: 309.5 MB of 2 GB
> physical memory used; 2.8 GB of 4.2 GB virtual memory used CPU:143.0905
> CPU/core:35.772625
> {noformat}
> The proposal is to add two more fields to show vCores and Cumulative CPU ms:
> {noformat}
> 2020-07-16 20:33:51,550 DEBUG [Container Monitor] ContainersMonitorImpl.audit
> (ContainersMonitorImpl.java:recordUsage(651)) - Resource usage of ProcessTree
> 809 for container-id container_1594931466123_0002_01_07: 309.5 MB of 2 GB
> physical memory used; 2.8 GB of 4.2 GB virtual memory used CPU:143.0905
> CPU/core:35.772625 vCores:2/1 CPU-ms:4180
> {noformat}
> This is a snippet of a log from one of our clusters running branch-2.8 with a
> similar change.
> {noformat}
> 2020-07-16 21:00:02,240 [Container Monitor] DEBUG
> ContainersMonitorImpl.audit: Memory usage of ProcessTree 5267 for
> container-id container_e04_1594079801456_1397450_01_001992: 1.6 GB of 2.5 GB
> physical memory used; 3.8 GB of 5.3 GB virtual memory used. CPU usage: 18 of
> 10 CPU vCores used. Cumulative CPU time: 157410
> 2020-07-16 21:00:02,269 [Container Monitor] DEBUG
> ContainersMonitorImpl.audit: Memory usage of ProcessTree 18801 for
> container-id container_e04_1594079801456_1390375_01_19: 413.2 MB of 2.5
> GB physical memory used; 3.8 GB of 5.3 GB virtual memory used. CPU usage: 0
> of 10 CPU vCores used. Cumulative CPU time: 113830
> 2020-07-16 21:00:02,298 [Container Monitor] DEBUG
> ContainersMonitorImpl.audit: Memory usage of ProcessTree 5279 for
> container-id container_e04_1594079801456_1397450_01_001991: 2.2 GB of 2.5 GB
> physical memory used; 3.8 GB of 5.3 GB virtual memory used. CPU usage: 17 of
> 10 CPU vCores used. Cumulative CPU time: 128630
> 2020-07-16 21:00:02,339 [Container Monitor] DEBUG
> ContainersMonitorImpl.audit: Memory usage of ProcessTree 24189 for
> container-id container_e04_1594079801456_1390430_01_000415: 392.7 MB of 2.5
> GB physical memory used; 3.8 GB of 5.3 GB virtual memory used. CPU usage: 0
> of 10 CPU vCores used. Cumulative CPU time: 96060
> 2020-07-16 21:00:02,367 [Container Monitor] DEBUG
> ContainersMonitorImpl.audit: Memory usage of ProcessTree 6751 for
> container-id container_e04_1594079801456_1397923_01_003248: 1.3 GB of 3 GB
> physical memory used; 4.3 GB of 6.3 GB virtual memory used. CPU usage: 12 of
> 10 CPU vCores used. Cumulative CPU time: 116820
> 2020-07-16 21:00:02,396 [Container Monitor] DEBUG
> ContainersMonitorImpl.audit: Memory usage of ProcessTree 12138 for
> container-id container_e04_1594079801456_1397760_01_44: 4.4 GB of 6 GB
> physical memory used; 6.9 GB of 12.6 GB virtual memory used. CPU usage: 15 of
> 10 CPU vCores used. Cumulative CPU time: 45900
> 2020-07-16 21:00:02,424 [Container Monitor] DEBUG
> ContainersMonitorImpl.audit: Memory usage of ProcessTree 101918 for
> container-id container_e04_1594079801456_1391130_01_002378: 2.4 GB of 4 GB
> physical memory used; 5.8 GB of 8.4 GB virtual memory used. CPU usage: 13 of
> 10 CPU vCores used. Cumulative CPU time: 2572390
> 2020-07-16 21:00:02,456 [Container Monitor] DEBUG
> ContainersMonitorImpl.audit: Memory usage of ProcessTree 26596 for
> container-id container_e04_1594079801456_1390446_01_000665: 418.6 MB of 2.5
> GB physical
[jira] [Commented] (YARN-10339) Timeline Client in Nodemanager gets 403 errors when simple auth is used in kerberos environments
[
https://issues.apache.org/jira/browse/YARN-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159398#comment-17159398
]
Hudson commented on YARN-10339:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18445 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18445/])
YARN-10339. Fix TimelineClient in NodeManager failing when Simple Http
(pjoseph: rev cc71d50b219c1cc682b4185ea739b485e519501f)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineConnector.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/security/TestTimelineAuthenticationFilterForV1.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/timeline/security/TestTimelineAuthenticationFilterInitializer.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/security/TestTimelineAuthFilterForV2.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClientImpl.java
> Timeline Client in Nodemanager gets 403 errors when simple auth is used in
> kerberos environments
>
>
> Key: YARN-10339
> URL: https://issues.apache.org/jira/browse/YARN-10339
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineclient
>Affects Versions: 3.1.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10339.001.patch, YARN-10339.002.patch
>
>
> We get below errors in NodeManager logs whenever we set
> yarn.timeline-service.http-authentication.type=simple in a cluster which has
> kerberos enabled. There are use cases where simple auth is used only in
> timeline server for convenience although kerberos is enabled.
> {code:java}
> 2020-05-20 20:06:30,181 ERROR impl.TimelineV2ClientImpl
> (TimelineV2ClientImpl.java:putObjects(321)) - Response from the timeline
> server is not successful, HTTP error code: 403, Server response:
> {"exception":"ForbiddenException","message":"java.lang.Exception: The owner
> of the posted timeline entities is not
> set","javaClassName":"org.apache.hadoop.yarn.webapp.ForbiddenException"}
> {code}
> This seems to affect the NM timeline publisher which uses
> TimelineV2ClientImpl. Doing a simple auth directly to timeline service via
> curl works fine. So this issue is in the authenticator configuration in
> timeline client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10350) TestUserGroupMappingPlacementRule fails
[
https://issues.apache.org/jira/browse/YARN-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157910#comment-17157910
]
Hudson commented on YARN-10350:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18439 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18439/])
YARN-10350. TestUserGroupMappingPlacementRule fails (aajisaka: rev
8980c23fc2cdc3e30eb2066fc4a16dca4db819ec)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SimpleGroupsMapping.java
> TestUserGroupMappingPlacementRule fails
> ---
>
> Key: YARN-10350
> URL: https://issues.apache.org/jira/browse/YARN-10350
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-10350.001.patch, YARN-10350.002.patch
>
>
> TestUserGroupMappingPlacementRule fails on trunk:
> {noformat}
> [INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule
> [ERROR] Tests run: 31, Failures: 1, Errors: 2, Skipped: 0, Time elapsed:
> 2.662 s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule
> [ERROR]
> testResolvedQueueIsNotManaged(org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule)
> Time elapsed: 0.03 s <<< ERROR!
> java.lang.Exception: Unexpected exception,
> expected but
> was
> at
> org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:28)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by: java.lang.AssertionError: Queue expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule.verifyQueueMapping(TestUserGroupMappingPlacementRule.java:236)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule.testResolvedQueueIsNotManaged(TestUserGroupMappingPlacementRule.java:516)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19)
> ... 18 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
[jira] [Commented] (YARN-10348) Allow RM to always cancel tokens after app completes
[ https://issues.apache.org/jira/browse/YARN-10348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157032#comment-17157032 ] Hudson commented on YARN-10348: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18431 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18431/]) YARN-10348. Allow RM to always cancel tokens after app completes. (ebadger: rev 48f90115b5ecb37f814af281f09bb404361b2bba) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java > Allow RM to always cancel tokens after app completes > > > Key: YARN-10348 > URL: https://issues.apache.org/jira/browse/YARN-10348 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.10.0, 3.1.3 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10348.001.patch, YARN-10348.002.patch > > > (Note: this change was originally done on our internal branch by [~daryn]). > The RM currently has an option for a client to specify disabling token > cancellation when a job completes. This feature was an initial attempt to > address the use case of a job launching sub-jobs (ie. oozie launcher) and the > original job finishing prior to the sub-job(s) completion - ex. original job > completion triggered premature cancellation of tokens needed by the sub-jobs. > Many years ago, [~daryn] added a more robust implementation to ref count > tokens ([YARN-3055]). This prevented premature cancellation of the token > until all apps using the token complete, and invalidated the need for a > client to specify cancel=false. Unfortunately the config option was not > removed. > We have seen cases where oozie "java actions" and some users were explicitly > disabling token cancellation. This can lead to a buildup of defunct tokens > that may overwhelm the ZK buffer used by the KDC's backing store. At which > point the KMS fails to connect to ZK and is unable to issue/validate new > tokens - rendering the KDC only able to authenticate pre-existing tokens. > Production incidents have occurred due to the buffer size issue. > To avoid these issues, the RM should have the option to ignore/override the > client's request to not cancel tokens. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10297) TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently
[
https://issues.apache.org/jira/browse/YARN-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156926#comment-17156926
]
Hudson commented on YARN-10297:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18429 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18429/])
YARN-10297. (ericp: rev 0427100b7543d412f4fafe631b7ace289662d28c)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestContinuousScheduling.java
> TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails
> intermittently
> ---
>
> Key: YARN-10297
> URL: https://issues.apache.org/jira/browse/YARN-10297
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jonathan Hung
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-10297.001.patch, YARN-10297.002.patch,
> YARN-10297.003.patch
>
>
> After YARN-6492, testFairSchedulerContinuousSchedulingInitTime fails
> intermittently when running {{mvn test -Dtest=TestContinuousScheduling}}
> {noformat}[INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling
> [ERROR] Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.682
> s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling
> [ERROR]
> testFairSchedulerContinuousSchedulingInitTime(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling)
> Time elapsed: 0.194 s <<< ERROR!
> org.apache.hadoop.metrics2.MetricsException: Metrics source
> PartitionQueueMetrics,partition= already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.getPartitionMetrics(QueueMetrics.java:362)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.incrPendingResources(QueueMetrics.java:601)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updatePendingResources(AppSchedulingInfo.java:388)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.internalAddResourceRequests(AppSchedulingInfo.java:320)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.internalAddResourceRequests(AppSchedulingInfo.java:347)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateResourceRequests(AppSchedulingInfo.java:183)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.updateResourceRequests(SchedulerApplicationAttempt.java:456)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.allocate(FairScheduler.java:898)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling.testFairSchedulerContinuousSchedulingInitTime(TestContinuousScheduling.java:375)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10333) YarnClient obtain Delegation Token for Log Aggregation Path
[ https://issues.apache.org/jira/browse/YARN-10333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154290#comment-17154290 ] Hudson commented on YARN-10333: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18422 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18422/]) YARN-10333. YarnClient obtain Delegation Token for Log Aggregation Path. (sunilg: rev 5dd270e2085c8e8c3428287ed6f0c541a5548a31) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml > YarnClient obtain Delegation Token for Log Aggregation Path > --- > > Key: YARN-10333 > URL: https://issues.apache.org/jira/browse/YARN-10333 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10333-001.patch, YARN-10333-002.patch, > YARN-10333-003.patch > > > There are use cases where Yarn Log Aggregation Path is configured to a > FileSystem like S3 or ABFS different from what is configured in fs.defaultFS > (HDFS). Log Aggregation fails as the client has token only for fs.defaultFS > and not for log aggregation path. > This Jira is to improve YarnClient by obtaining delegation token for log > aggregation path and add it to the Credential of Container Launch Context > similar to how it does for Timeline Delegation Token. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10341) Yarn Service Container Completed event doesn't get processed
[
https://issues.apache.org/jira/browse/YARN-10341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154281#comment-17154281
]
Hudson commented on YARN-10341:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18421 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18421/])
YARN-10341. Yarn Service Container Completed event doesn't get (brahma: rev
dfe60392c91be21f574c1659af22f5c381b2675a)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestServiceAM.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ServiceScheduler.java
> Yarn Service Container Completed event doesn't get processed
> -
>
> Key: YARN-10341
> URL: https://issues.apache.org/jira/browse/YARN-10341
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Critical
> Fix For: 3.4.0, 3.3.1
>
> Attachments: YARN-10341.001.patch, YARN-10341.002.patch,
> YARN-10341.003.patch, YARN-10341.004.patch
>
>
> If there 10 workers running and if containers get killed , after a while we
> see that there are just 9 workers runnning. This is due to CONTAINER
> COMPLETED Event is not processed on AM side.
> Issue is in below code:
> {code:java}
> public void onContainersCompleted(List statuses) {
> for (ContainerStatus status : statuses) {
> ContainerId containerId = status.getContainerId();
> ComponentInstance instance =
> liveInstances.get(status.getContainerId());
> if (instance == null) {
> LOG.warn(
> "Container {} Completed. No component instance exists.
> exitStatus={}. diagnostics={} ",
> containerId, status.getExitStatus(), status.getDiagnostics());
> return;
> }
> ComponentEvent event =
> new ComponentEvent(instance.getCompName(), CONTAINER_COMPLETED)
> .setStatus(status).setInstance(instance)
> .setContainerId(containerId);
> dispatcher.getEventHandler().handle(event);
> }
> {code}
> If component instance doesnt exist for a container, it doesnt iterate over
> other containers as its returning from method
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10344) Sync netty versions in hadoop-yarn-csi
[
https://issues.apache.org/jira/browse/YARN-10344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154190#comment-17154190
]
Hudson commented on YARN-10344:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18420 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18420/])
YARN-10344. Sync netty versions in hadoop-yarn-csi. (#2126) (github: rev
10d218934c9bc143bf8578c92cdbd6df6a4d3b98)
* (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml
> Sync netty versions in hadoop-yarn-csi
> --
>
> Key: YARN-10344
> URL: https://issues.apache.org/jira/browse/YARN-10344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: 3.4.0, 3.3.1
>
>
> netty-all is now 4.1.50.Final but the other netty libraries are 4.1.42.Final:
> {noformat}
> [INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ hadoop-yarn-csi
> ---
> [INFO] org.apache.hadoop:hadoop-yarn-csi:jar:3.3.0
> [INFO] +- com.google.guava:guava:jar:20.0:compile
> [INFO] +- com.google.protobuf:protobuf-java:jar:3.6.1:compile
> [INFO] +- io.netty:netty-all:jar:4.1.50.Final:compile
> [INFO] +- io.grpc:grpc-core:jar:1.26.0:compile
> [INFO] | +- io.grpc:grpc-api:jar:1.26.0:compile (version selected from
> constraint [1.26.0,1.26.0])
> [INFO] | | +- io.grpc:grpc-context:jar:1.26.0:compile
> [INFO] | | +-
> com.google.errorprone:error_prone_annotations:jar:2.3.3:compile
> [INFO] | | \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
> [INFO] | +- com.google.code.gson:gson:jar:2.2.4:compile
> [INFO] | +- com.google.android:annotations:jar:4.1.1.4:compile
> [INFO] | +- io.perfmark:perfmark-api:jar:0.19.0:compile
> [INFO] | +- io.opencensus:opencensus-api:jar:0.24.0:compile
> [INFO] | \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.24.0:compile
> [INFO] +- io.grpc:grpc-protobuf:jar:1.26.0:compile
> [INFO] | +- com.google.api.grpc:proto-google-common-protos:jar:1.12.0:compile
> [INFO] | \- io.grpc:grpc-protobuf-lite:jar:1.26.0:compile
> [INFO] +- io.grpc:grpc-stub:jar:1.26.0:compile
> [INFO] +- io.grpc:grpc-netty:jar:1.26.0:compile
> [INFO] | +- io.netty:netty-codec-http2:jar:4.1.42.Final:compile (version
> selected from constraint [4.1.42.Final,4.1.42.Final])
> [INFO] | | +- io.netty:netty-common:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-buffer:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-transport:jar:4.1.42.Final:compile
> [INFO] | | | \- io.netty:netty-resolver:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-codec:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-handler:jar:4.1.42.Final:compile
> [INFO] | | \- io.netty:netty-codec-http:jar:4.1.42.Final:compile
> [INFO] | \- io.netty:netty-handler-proxy:jar:4.1.42.Final:compile
> [INFO] | \- io.netty:netty-codec-socks:jar:4.1.42.Final:compile
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8047) RMWebApp make external class pluggable
[ https://issues.apache.org/jira/browse/YARN-8047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153315#comment-17153315 ] Hudson commented on YARN-8047: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18418 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18418/]) YARN-8047. RMWebApp make external class pluggable. (pjoseph: rev 3a4d05b850449c51a13f3a15fe0d756fdf50b4b2) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml > RMWebApp make external class pluggable > -- > > Key: YARN-8047 > URL: https://issues.apache.org/jira/browse/YARN-8047 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bibin Chundatt >Assignee: Bilwa S T >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-8047-001.patch, YARN-8047-002.patch, > YARN-8047-003.patch, YARN-8047.004.patch, YARN-8047.005.patch, > YARN-8047.006.patch > > > JIra should make sure we should be able to plugin webservices and web pages > of scheduler in Resourcemanager > * RMWebApp allow to bind external classes > * RMController allow to plugin scheduler classes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10337) TestRMHATimelineCollectors fails on hadoop trunk
[
https://issues.apache.org/jira/browse/YARN-10337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152693#comment-17152693
]
Hudson commented on YARN-10337:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18416 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18416/])
YARN-10337. Fix failing testcase TestRMHATimelineCollectors. (pjoseph: rev
2bbd00dff498027241a5d84713f4e3f13ac45e65)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMHATimelineCollectors.java
> TestRMHATimelineCollectors fails on hadoop trunk
>
>
> Key: YARN-10337
> URL: https://issues.apache.org/jira/browse/YARN-10337
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test, yarn
>Reporter: Ahmed Hussein
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10337.001.patch
>
>
> {{TestRMHATimelineCollectors}} has been failing on trunk. I see it frequently
> in the qbt reports and the yetus reprts
> {code:bash}
> [INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.95
> s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR]
> testRebuildCollectorDataOnFailover(org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors)
> Time elapsed: 5.615 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover(TestRMHATimelineCollectors.java:105)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:80)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover:105
> NullPointer
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [ERROR] There are test failures.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
[jira] [Commented] (YARN-10330) Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule
[
https://issues.apache.org/jira/browse/YARN-10330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149394#comment-17149394
]
Hudson commented on YARN-10330:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18402 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18402/])
YARN-10330. Add missing test scenarios to (snemeth: rev
04abd0eb17b58e321893e8651ec596e9f7ac786f)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SimpleGroupsMapping.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestAppNameMappingPlacementRule.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
> Add missing test scenarios to TestUserGroupMappingPlacementRule and
> TestAppNameMappingPlacementRule
> ---
>
> Key: YARN-10330
> URL: https://issues.apache.org/jira/browse/YARN-10330
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler, capacityscheduler, test
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10330-001.patch, YARN-10330-002.patch,
> YARN-10330-003.patch, YARN-10330-004.patch, YARN-10330-branch-3.3-001.patch
>
>
> After running {{TestUserGroupMappingPlacementRule}} with EclEmma, it turned
> out that there are at least 8-10 missing test scenarios that are not covered.
> Since we're planning to enhance mapping rule logic with extra features, it is
> crucial to have good coverage so that we can verify backward compatibility.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10325) Document max-parallel-apps for Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17149364#comment-17149364 ] Hudson commented on YARN-10325: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18401 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18401/]) YARN-10325. Document max-parallel-apps for Capacity Scheduler. (snemeth: rev 9b5557a9e811f04b964aa3a31ba8846a907d26f9) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md > Document max-parallel-apps for Capacity Scheduler > - > > Key: YARN-10325 > URL: https://issues.apache.org/jira/browse/YARN-10325 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10325-001.patch, YARN-10325-branch-3.3.001.patch > > > New feature introduced by YARN-9930 should be reflected in the upstream > documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9809) NMs should supply a health status when registering with RM
[ https://issues.apache.org/jira/browse/YARN-9809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148914#comment-17148914 ] Hudson commented on YARN-9809: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18394 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18394/]) YARN-9809. Added node manager health status to resource manager (eyang: rev e8dc862d3856e9eaea124c625dade36f1dd53fe2) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/NodeManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/health/TimedHealthReporterService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/resourcetracker/TestNMReconnect.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestEventFlow.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/logaggregationstatus/TestRMAppLogAggregationStatus.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestNMProxy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerHealth.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/health/NodeHealthScriptRunner.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeStartedEvent.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodes.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/BaseContainerManagerTest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java * (edit)
[jira] [Commented] (YARN-10331) Upgrade node.js to 10.21.0
[ https://issues.apache.org/jira/browse/YARN-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148417#comment-17148417 ] Hudson commented on YARN-10331: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18391 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18391/]) YARN-10331. Upgrade node.js to 10.21.0. (#2106) (github: rev cd188ea9f0e807df1e2cc13f62be3e4c956b1e69) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/pom.xml * (edit) dev-support/docker/Dockerfile * (edit) dev-support/docker/Dockerfile_aarch64 > Upgrade node.js to 10.21.0 > -- > > Key: YARN-10331 > URL: https://issues.apache.org/jira/browse/YARN-10331 > Project: Hadoop YARN > Issue Type: Bug > Components: build, yarn-ui-v2 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Fix For: 3.4.0, 3.3.1 > > > YARN-10036 upgraded Node.js to 8.17.0, but Node.js 8.x is already EoL. > https://nodejs.org/en/about/releases/ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9903) Support reservations continue looking for Node Labels
[
https://issues.apache.org/jira/browse/YARN-9903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17148084#comment-17148084
]
Hudson commented on YARN-9903:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18389 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18389/])
YARN-9903: Support reservations continue looking for Node Labels. (ericp: rev
74fc13cf91818a70f434401244f7560c4db3a676)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java
> Support reservations continue looking for Node Labels
> -
>
> Key: YARN-9903
> URL: https://issues.apache.org/jira/browse/YARN-9903
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Tarun Parimi
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-9903-branch-3.2.001.patch, YARN-9903.001.patch,
> YARN-9903.002.patch, YARN-9903.003.patch, YARN-9903.004.patch
>
>
> YARN-1769 brought in reservations continue looking feature which improves the
> several resource reservation scenarios. However, it is not handled currently
> when nodes have a label assigned to them. This is useful and in many cases
> necessary even for Node Labels. So we should look to support this for node
> labels also.
> For example, in AbstractCSQueue.java, we have the below TODO.
> {code:java}
> // TODO, now only consider reservation cases when the node has no label
> if (this.reservationsContinueLooking && nodePartition.equals(
> RMNodeLabelsManager.NO_LABEL) && Resources.greaterThan( resourceCalculator,
> clusterResource, resourceCouldBeUnreserved, Resources.none())) {
> {code}
> cc [~sunilg]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10328) Too many ZK Curator NodeExists exception logs in YARN Service AM logs
[
https://issues.apache.org/jira/browse/YARN-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17147938#comment-17147938
]
Hudson commented on YARN-10328:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18388 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18388/])
YARN-10328. Fixed ZK Curator NodeExists exception in YARN service AM (eyang:
rev 0be26811f3db49abb62d12e6a051a31553495da8)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/registry/YarnRegistryViewForProviders.java
> Too many ZK Curator NodeExists exception logs in YARN Service AM logs
> -
>
> Key: YARN-10328
> URL: https://issues.apache.org/jira/browse/YARN-10328
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10328.001.patch
>
>
> Following debug logs are printed everytime when component is started.
> {code:java}
> [pool-6-thread-3] DEBUG zk.CuratorService - path already present:
> /registry/users/server/services/yarn-service/default-worker/components
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists for
> /registry/users/hetuserver/services/yarn-service/default-worker/components
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1480)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
> at
> org.apache.hadoop.registry.client.impl.zk.CuratorService.zkMkPath(CuratorService.java:587)
> at
> org.apache.hadoop.registry.client.impl.zk.RegistryOperationsService.mknode(RegistryOperationsService.java:99)
> at
> org.apache.hadoop.yarn.service.registry.YarnRegistryViewForProviders.putComponent(YarnRegistryViewForProviders.java:146)
> at
> org.apache.hadoop.yarn.service.registry.YarnRegistryViewForProviders.putComponent(YarnRegistryViewForProviders.java:128)
> at
> org.apache.hadoop.yarn.service.component.instance.ComponentInstance.updateServiceRecord(ComponentInstance.java:511)
> at
> org.apache.hadoop.yarn.service.component.instance.ComponentInstance.updateContainerStatus(ComponentInstance.java:449)
> at
> org.apache.hadoop.yarn.service.component.instance.ComponentInstance$ContainerStatusRetriever.run(ComponentInstance.java:620)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6526) Refactoring SQLFederationStateStore by avoiding to recreate a connection at every call
[ https://issues.apache.org/jira/browse/YARN-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146407#comment-17146407 ] Hudson commented on YARN-6526: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18384 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18384/]) YARN-6526. Refactoring SQLFederationStateStore by avoiding to recreate a (brahma: rev 2c03524fa4be754aa95889d4ac0f5d57dca8cda8) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/utils/FederationStateStoreUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/SQLFederationStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/TestSQLFederationStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/TestZookeeperFederationStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/metrics/FederationStateStoreClientMetrics.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/FederationStateStoreBaseTest.java > Refactoring SQLFederationStateStore by avoiding to recreate a connection at > every call > -- > > Key: YARN-6526 > URL: https://issues.apache.org/jira/browse/YARN-6526 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Giovanni Matteo Fumarola >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-6526.001.patch, YARN-6526.002.patch, > YARN-6526.003.patch, YARN-6526.004.patch, YARN-6526.005.patch, > YARN-6526.006.patch, YARN-6526.007.patch, YARN-6526.008.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10327) Remove duplication of checking for invalid application ID in TestLogsCLI
[ https://issues.apache.org/jira/browse/YARN-10327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146251#comment-17146251 ] Hudson commented on YARN-10327: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18383 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18383/]) YARN-10327. Remove duplication of checking for invalid application ID in (snemeth: rev e0c1d8a96905bfbedbddde9000fc08ce2af1d277) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java > Remove duplication of checking for invalid application ID in TestLogsCLI > > > Key: YARN-10327 > URL: https://issues.apache.org/jira/browse/YARN-10327 > Project: Hadoop YARN > Issue Type: Test >Reporter: Hudáky Márton Gyula >Assignee: Hudáky Márton Gyula >Priority: Trivial > Fix For: 3.4.0 > > Attachments: YARN-10327.001.patch > > > TestLogsCLI has a separate function to test for invalid application ID > (#testInvalidApplicationId) and another (#testLogsCLIWithInvalidArgs) to test > multiple invalid arguments (including application ID). One of them should be > eliminated. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10318) ApplicationHistory Web UI incorrect column indexing
[ https://issues.apache.org/jira/browse/YARN-10318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146240#comment-17146240 ] Hudson commented on YARN-10318: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18382 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18382/]) YARN-10318. ApplicationHistory Web UI incorrect column indexing. (snemeth: rev 6459cc73e6c0878be51c0f2f70ec9868325dec34) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppsBlock.java > ApplicationHistory Web UI incorrect column indexing > --- > > Key: YARN-10318 > URL: https://issues.apache.org/jira/browse/YARN-10318 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Minor > Fix For: 3.4.0 > > Attachments: Screenshot 2020-06-25 at 10.15.32.png, > YARN-10318.001.patch, image-2020-06-16-17-14-55-921.png > > > The ApplicationHistory UI is broken due to an incorrect column indexing. This > bug was probably introduced in YARN-10038, which presumes, that the table > contains the application tag column (which is true for RM Web UI, but not for > AH Web UI). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10277) CapacityScheduler test TestUserGroupMappingPlacementRule should build proper hierarchy
[ https://issues.apache.org/jira/browse/YARN-10277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17146157#comment-17146157 ] Hudson commented on YARN-10277: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18381 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18381/]) YARN-10277. CapacityScheduler test TestUserGroupMappingPlacementRule (snemeth: rev 33d3df7ee2b3942560e9affe1409854ac3e8bb96) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java > CapacityScheduler test TestUserGroupMappingPlacementRule should build proper > hierarchy > -- > > Key: YARN-10277 > URL: https://issues.apache.org/jira/browse/YARN-10277 > Project: Hadoop YARN > Issue Type: Task >Reporter: Gergely Pollak >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10277.001.patch, YARN-10277.002.patch, > YARN-10277.003.patch, YARN-10277.branch-3.3.001.patch > > > Since the CapacityScheduler internal implementation depends more and more on > queue being hierarchical, the test gets really hard to maintain. A lot of > test cases were failing because they used non existing queues, but the older > placement rule solution ignored missing parents, but since the leaf queue > change in CS, we must be able to get a full path for any queue, since all > queues are referenced by their full path. > This test should reflect this and instead of creating and expecting the > existance of fictional queues, it should create a proper queue hierarchy, > with a way to describe it better. > Currently we set up a bunch of mockito "when" statements to simulate the > queue behavior, but this is a hassle to maintain, and easy to miss a few > method. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10279) Avoid unnecessary QueueMappingEntity creations
[ https://issues.apache.org/jira/browse/YARN-10279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17145042#comment-17145042 ] Hudson commented on YARN-10279: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18379 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18379/]) YARN-10279. Avoid unnecessary QueueMappingEntity creations. Contributed (snemeth: rev 6a8fd73b273629d0c7c071cf4d090f67d9b96fe4) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/QueuePlacementRuleUtils.java > Avoid unnecessary QueueMappingEntity creations > -- > > Key: YARN-10279 > URL: https://issues.apache.org/jira/browse/YARN-10279 > Project: Hadoop YARN > Issue Type: Task >Reporter: Gergely Pollak >Assignee: Hudáky Márton Gyula >Priority: Minor > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10279.001.patch, YARN-10279.003.patch, > YARN-10279.004.patch, YARN-10279.005.patch, YARN-10279.006.patch > > > In CS UserGroupMappingPlacementRule and AppNameMappingPlacementRule classes > we create new instances of QueueMappingEntity class. In some cases we simply > copy the already received class, so we just duplicate it, which is > unnecessary since the class is immutable. > This is just a minor improvement, probably doesn't have much impact, but > still puts some unnecessary load on GC. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10316) FS-CS converter: convert maxAppsDefault, maxRunningApps settings
[
https://issues.apache.org/jira/browse/YARN-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17142824#comment-17142824
]
Hudson commented on YARN-10316:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18376 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18376/])
YARN-10316. FS-CS converter: convert maxAppsDefault, maxRunningApps (snemeth:
rev 03f855e3e7a4505362e221c8a07ae9317af773d0)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigRuleHandler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigRuleHandler.java
> FS-CS converter: convert maxAppsDefault, maxRunningApps settings
>
>
> Key: YARN-10316
> URL: https://issues.apache.org/jira/browse/YARN-10316
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Fix For: 3.4.0
>
> Attachments: YARN-10136-001.patch, YARN-10316-002.patch,
> YARN-10316-003.patch, YARN-10316-branch-3.3.001.patch
>
>
> In YARN-9930, support for maximum running applications (called "max parallel
> apps") has been introduced.
> The converter now can handle the following settings in {{fair-scheduler.xml}}:
> * {{}} per user
> * {{}} per queue
> * {{}}
> * {{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10321) Break down TestUserGroupMappingPlacementRule#testMapping into test scenarios
[ https://issues.apache.org/jira/browse/YARN-10321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17141769#comment-17141769 ] Hudson commented on YARN-10321: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18372 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18372/]) YARN-10321. Break down TestUserGroupMappingPlacementRule#testMapping (pjoseph: rev ce1008fe61a70d23845c05fad0380ae4906e74f5) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java > Break down TestUserGroupMappingPlacementRule#testMapping into test scenarios > > > Key: YARN-10321 > URL: https://issues.apache.org/jira/browse/YARN-10321 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10321.001.patch > > > org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule#testMapping > is very large and hard to read/maintain and moreover, error-prone. > We should break this testcase down into several separate testcases. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9460) QueueACLsManager and ReservationsACLManager should not use instanceof checks
[ https://issues.apache.org/jira/browse/YARN-9460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17141103#comment-17141103 ] Hudson commented on YARN-9460: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18370 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18370/]) YARN-9460. QueueACLsManager and ReservationsACLManager should not use (surendralilhore: rev b2facc84a1b48b9dcbe0816e120778d2100b320e) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/CapacityReservationsACLsManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/GenericQueueACLsManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/QueueACLsManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/AbstractReservationSystem.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/package-info.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ReservationsACLsManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/FairReservationsACLsManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/CapacityQueueACLsManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/FairQueueACLsManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMTokens.java > QueueACLsManager and ReservationsACLManager should not use instanceof checks > > > Key: YARN-9460 > URL: https://issues.apache.org/jira/browse/YARN-9460 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Szilard Nemeth >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-9460.001.patch, YARN-9460.002.patch, > YARN-9460.003.patch, YARN-9460.004.patch, YARN-9460.005.patch > > > QueueACLsManager and ReservationsACLManager should not use instanceof checks > for the scheduler type. > Rather, we should abstract this into two classes: Capacity and Fair variants > of these ACL classes. > QueueACLsManager and ReservationsACLManager could be abstract classes, but > the implementation is the decision of one who will work on this jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140526#comment-17140526 ] Hudson commented on YARN-9930: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18365 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18365/]) YARN-9930. Support max running app logic for CapacityScheduler. (snemeth: rev 469841446f921f3da5bbd96cf83b3a808dde8084) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueStateManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSMaxRunningAppsEnforcer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCSMaxRunningAppsEnforcer.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerMaxParallelApps.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/TestReservationSystem.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueState.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-005.patch, > YARN-9930-006.patch, YARN-9930-POC01.patch, YARN-9930-POC02.patch, > YARN-9930-POC03.patch, YARN-9930-POC04.patch, YARN-9930-POC05.patch, > screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10308) Update javadoc and variable names for keytab in yarn services as it supports filesystems other than hdfs and local file system
[
https://issues.apache.org/jira/browse/YARN-10308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17138619#comment-17138619
]
Hudson commented on YARN-10308:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18359 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18359/])
YARN-10308. Update javadoc and variable names for YARN service.
(eyang: rev 89689c52c39cdcc498d04508dbd235c6036ec17c)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/KerberosPrincipal.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/client/ServiceClient.java
> Update javadoc and variable names for keytab in yarn services as it supports
> filesystems other than hdfs and local file system
> --
>
> Key: YARN-10308
> URL: https://issues.apache.org/jira/browse/YARN-10308
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: YARN-10308.001.patch, YARN-10308.002.patch
>
>
> 1. Below description should be updated
> {code:java}
> @ApiModelProperty(value = "The URI of the kerberos keytab. It supports two " +
> "schemes \"hdfs\" and \"file\". If the URI starts with \"hdfs://\" " +
> "scheme, it indicates the path on hdfs where the keytab is stored. The
> " +
> "keytab will be localized by YARN and made available to AM in its
> local" +
> " directory. If the URI starts with \"file://\" scheme, it indicates a
> " +
> "path on the local host where the keytab is presumbaly installed by " +
> "admins upfront. ")
> public String getKeytab() {
> return keytab;
> }
> {code}
> 2. Variables below are still named on hdfs which is confusing
> {code:java}
> if ("file".equals(keytabURI.getScheme())) {
> LOG.info("Using a keytab from localhost: " + keytabURI);
> } else {
> Path keytabOnhdfs = new Path(keytabURI);
> if (!fileSystem.getFileSystem().exists(keytabOnhdfs)) {
> LOG.warn(service.getName() + "'s keytab (principalName = "
> + principalName + ") doesn't exist at: " + keytabOnhdfs);
> return;
> }
> LocalResource keytabRes = fileSystem.createAmResource(keytabOnhdfs,
> LocalResourceType.FILE);
> localResource.put(String.format(YarnServiceConstants.KEYTAB_LOCATION,
> service.getName()), keytabRes);
> LOG.info("Adding " + service.getName() + "'s keytab for "
> + "localization, uri = " + keytabOnhdfs);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10281) Redundant QueuePath usage in UserGroupMappingPlacementRule and AppNameMappingPlacementRule
[ https://issues.apache.org/jira/browse/YARN-10281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17138412#comment-17138412 ] Hudson commented on YARN-10281: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18357 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18357/]) YARN-10281. Redundant QueuePath usage in UserGroupMappingPlacementRule (snemeth: rev 5b1a56f9f1aec7d75b14a60d0c42192b04407356) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/AppNameMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/QueuePlacementRuleUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/QueueMapping.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/QueuePath.java > Redundant QueuePath usage in UserGroupMappingPlacementRule and > AppNameMappingPlacementRule > -- > > Key: YARN-10281 > URL: https://issues.apache.org/jira/browse/YARN-10281 > Project: Hadoop YARN > Issue Type: Task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10281.001.patch, YARN-10281.002.patch, > YARN-10281.003.patch, YARN-10281.004.patch, YARN-10281.branch-3.3.001.patch > > > We use the QueuePath and QueueMapping (or QueueMappingEntity) objects in the > aforementioned classes, but these technically store the same kind of > information, yet we keep converting between them, let's examine if we can use > only the QueueMapping(Entity) instead, since that holds more information. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10314) YarnClient throws NoClassDefFoundError for WebSocketException with only shaded client jars
[
https://issues.apache.org/jira/browse/YARN-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17138090#comment-17138090
]
Hudson commented on YARN-10314:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18355 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18355/])
YARN-10314. YarnClient throws NoClassDefFoundError for (github: rev
fc4ebb0499fe1095b87ff782c265e9afce154266)
* (edit) hadoop-client-modules/hadoop-client-minicluster/pom.xml
* (edit) hadoop-client-modules/hadoop-client-runtime/pom.xml
> YarnClient throws NoClassDefFoundError for WebSocketException with only
> shaded client jars
> --
>
> Key: YARN-10314
> URL: https://issues.apache.org/jira/browse/YARN-10314
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.0
>Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Blocker
> Fix For: 3.3.0
>
>
> After YARN-8778, with only shaded hadoop client jars in classpath Unable to
> submit job.
> CC: [~ayushtkn] confirmed the same. Hive 4.0 doesnot work due to this, shaded
> client is necessary there to avoid guava jar's conflicts.
> {noformat}
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/hadoop/shaded/org/eclipse/jetty/websocket/api/WebSocketException
> at
> org.apache.hadoop.yarn.client.api.YarnClient.createYarnClient(YarnClient.java:92)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:109)
> at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:153)
> at
> org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34)
> at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:130)
> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:109)
> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:102)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1545)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1541)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.mapreduce.Job.connect(Job.java:1541)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1570)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1594)
> at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.shaded.org.eclipse.jetty.websocket.api.WebSocketException
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10293) Reserved Containers not allocated from available space of other nodes in CandidateNodeSet in MultiNodePlacement (YARN-10259)
[
https://issues.apache.org/jira/browse/YARN-10293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17134262#comment-17134262
]
Hudson commented on YARN-10293:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18347 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18347/])
YARN-10293. Reserved Containers not allocated from available space of
(taoyang1: rev 7c4de59fc10953170bbef9a320ce70bcddae8bba)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerMultiNodes.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerMultiNodesWithPreemption.java
> Reserved Containers not allocated from available space of other nodes in
> CandidateNodeSet in MultiNodePlacement (YARN-10259)
>
>
> Key: YARN-10293
> URL: https://issues.apache.org/jira/browse/YARN-10293
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10293-001.patch, YARN-10293-002.patch,
> YARN-10293-003-WIP.patch, YARN-10293-004.patch, YARN-10293-005.patch
>
>
> Reserved Containers not allocated from available space of other nodes in
> CandidateNodeSet in MultiNodePlacement. YARN-10259 has fixed two issues
> related to it
> https://issues.apache.org/jira/browse/YARN-10259?focusedCommentId=17105987=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17105987
> Have found one more bug in the CapacityScheduler.java code which causes the
> same issue with slight difference in the repro.
> *Repro:*
> *Nodes : Available : Used*
> Node1 - 8GB, 8vcores - 8GB. 8cores
> Node2 - 8GB, 8vcores - 8GB. 8cores
> Node3 - 8GB, 8vcores - 8GB. 8cores
> Queues -> A and B both 50% capacity, 100% max capacity
> MultiNode enabled + Preemption enabled
> 1. JobA submitted to A queue and which used full cluster 24GB and 24 vcores
> 2. JobB Submitted to B queue with AM size of 1GB
> {code}
> 2020-05-21 12:12:27,313 INFO
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=systest
> IP=172.27.160.139 OPERATION=Submit Application Request
> TARGET=ClientRMService RESULT=SUCCESS APPID=application_1590046667304_0005
> CALLERCONTEXT=CLI QUEUENAME=dummy
> {code}
> 3. Preemption happens and used capacity is lesser than 1.0f
> {code}
> 2020-05-21 12:12:48,222 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics:
> Non-AM container preempted, current
> appAttemptId=appattempt_1590046667304_0004_01,
> containerId=container_e09_1590046667304_0004_01_24,
> resource=
> {code}
> 4. JobB gets a Reserved Container as part of
> CapacityScheduler#allocateOrReserveNewContainer
> {code}
> 2020-05-21 12:12:48,226 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
> container_e09_1590046667304_0005_01_01 Container Transitioned from NEW to
> RESERVED
> 2020-05-21 12:12:48,226 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp:
> Reserved container=container_e09_1590046667304_0005_01_01, on node=host:
> tajmera-fullnodes-3.tajmera-fullnodes.root.hwx.site:8041 #containers=8
> available= used= with
> resource=
> {code}
> *Why RegularContainerAllocator reserved the container when the used capacity
> is <= 1.0f ?*
> {code}
> The reason is even though the container is preempted - nodemanager has to
> stop the container and heartbeat and update the available and unallocated
> resources to ResourceManager.
> {code}
> 5. Now, no new allocation happens and reserved container stays at reserved.
> After reservation the used capacity becomes 1.0f, below will be in a loop and
> no new allocate or reserve happens. The reserved container cannot be
> allocated as reserved node does not have space. node2 has space for 1GB,
> 1vcore but CapacityScheduler#allocateOrReserveNewContainers not getting
> called causing the Hang.
> *[INFINITE LOOP] CapacityScheduler#allocateContainersOnMultiNodes ->
> CapacityScheduler#allocateFromReservedContainer -> Re-reserve the container
> on node*
> {code}
> 2020-05-21 12:13:33,242 INFO
>
[jira] [Commented] (YARN-10312) Add support for yarn logs -logFile to retain backward compatibility
[
https://issues.apache.org/jira/browse/YARN-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17133701#comment-17133701
]
Hudson commented on YARN-10312:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18346 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18346/])
YARN-10312. Add support for yarn logs -logFile to retain backward (ebadger: rev
fed6fecd3a9e24efc20f9221505da35a7e1949c7)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
> Add support for yarn logs -logFile to retain backward compatibility
> ---
>
> Key: YARN-10312
> URL: https://issues.apache.org/jira/browse/YARN-10312
> Project: Hadoop YARN
> Issue Type: Bug
> Components: client
>Affects Versions: 2.10.0, 3.4.1
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Labels: compatibility
> Attachments: YARN-10312.001.patch
>
>
> The YARN CLI logs command line option {{-logFiles}} was changed to
> {{-log_files}} in 2.9 and later releases. This change was made as part of
> YARN-5363.
> Verizon Media is in the process of moving from Hadoop-2.8 to Hadoop-2.10, and
> while testing integration with Spark, we ran into this issue. We are
> concerned that we will run into more cases of this as we roll out to
> production, and rather than break user scripts, we'd prefer to add
> {{-logFiles}} as an alias of {{-log_files}}. If both are provided,
> {{-logFiles}} will be ignored.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10300) appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat
[
https://issues.apache.org/jira/browse/YARN-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129725#comment-17129725
]
Hudson commented on YARN-10300:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18342 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18342/])
YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails
(ericp: rev 56247db3022705635580c4d2f8b0abde109f954f)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
> appMasterHost not set in RM ApplicationSummary when AM fails before first
> heartbeat
> ---
>
> Key: YARN-10300
> URL: https://issues.apache.org/jira/browse/YARN-10300
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-10300.001.patch, YARN-10300.002.patch,
> YARN-10300.003.patch
>
>
> {noformat}
> 2020-05-23 14:09:10,086 INFO resourcemanager.RMAppManager$ApplicationSummary:
> appId=application_1586003420099_12444961,name=job_name,user=username,queue=queuename,state=FAILED,trackingUrl=https
>
> ://cluster:port/applicationhistory/app/application_1586003420099_12444961,appMasterHost=N/A,startTime=1590241207309,finishTime=1590242950085,finalStatus=FAILED,memorySeconds=13750,vcoreSeconds=67,preemptedMemorySeconds=0,preemptedVcoreSeconds=0,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources= vCores:0>,applicationType=MAPREDUCE
> {noformat}
> {{appMasterHost=N/A}} should have the AM hostname instead of N/A
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10292) FS-CS converter: add an option to enable asynchronous scheduling in CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126761#comment-17126761 ] Hudson commented on YARN-10292: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18330 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18330/]) YARN-10292. FS-CS converter: add an option to enable asynchronous (snemeth: rev 8abff5151a7c157355930475dbea2dc535d51c19) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSYarnSiteConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/ConversionOptions.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSYarnSiteConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigArgumentHandler.java > FS-CS converter: add an option to enable asynchronous scheduling in > CapacityScheduler > - > > Key: YARN-10292 > URL: https://issues.apache.org/jira/browse/YARN-10292 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10292.001.patch, YARN-10292.002.patch > > > FS doesn't have an equivalent setting to the CapacityScheduler's > yarn.scheduler.capacity.schedule-asynchronously.enable option so the FS to CS > converter won't add this to the yarn-site.xml. An optional command line > switch should be added to support this option during migration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10274) Merge QueueMapping and QueueMappingEntity
[ https://issues.apache.org/jira/browse/YARN-10274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126601#comment-17126601 ] Hudson commented on YARN-10274: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18328 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18328/]) YARN-10274. Merge QueueMapping and QueueMappingEntity. Contributed by (snemeth: rev 8b146c17b35532af7b9685aa1896d2b4f4fadc05) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/AppNameMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestPlacementManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestAppNameMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/QueueMapping.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerQueueMappingFactory.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/QueueMappingEntity.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/QueuePlacementRuleUtils.java > Merge QueueMapping and QueueMappingEntity > - > > Key: YARN-10274 > URL: https://issues.apache.org/jira/browse/YARN-10274 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10274.001.patch, YARN-10274.002.patch, > YARN-10274.003.patch > > > The role, usage and internal behaviour of these classes are almost identical, > but it makes no sense to keep both of them. One is used by UserGroup > placement rule definitions the other is used by Application placement rules. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10296) Make ContainerPBImpl#getId/setId synchronized
[ https://issues.apache.org/jira/browse/YARN-10296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126547#comment-17126547 ] Hudson commented on YARN-10296: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18327 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18327/]) YARN-10296. Make ContainerPBImpl#getId/setId synchronized. Contributed (snemeth: rev a4835db95a2cca255bf0e40fa5d016526ef03857) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerPBImpl.java > Make ContainerPBImpl#getId/setId synchronized > - > > Key: YARN-10296 > URL: https://issues.apache.org/jira/browse/YARN-10296 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10296.001.patch, YARN-10296.002.patch > > > ContainerPBImpl getId and setId methods can be accessed from multiple > threads. In order to avoid any simultaneous accesses and race conditions > these methods should be synchronized. > The idea came from the issue described in YARN-10295, however that patch is > only applicable to branch-3.2 and 3.1. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10254) CapacityScheduler incorrect User Group Mapping after leaf queue change
[ https://issues.apache.org/jira/browse/YARN-10254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124044#comment-17124044 ] Hudson commented on YARN-10254: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18315 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18315/]) YARN-10254. CapacityScheduler incorrect User Group Mapping after leaf (snemeth: rev b5efdea4fd385453fd9f9da7106e908d2a9a3812) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerQueueMappingFactory.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java > CapacityScheduler incorrect User Group Mapping after leaf queue change > -- > > Key: YARN-10254 > URL: https://issues.apache.org/jira/browse/YARN-10254 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10254.001.patch, YARN-10254.002.patch, > YARN-10254.003.patch, YARN-10254.004.patch, YARN-10254.005.patch, > YARN-10254.branch-3.3.001.patch > > > YARN-9879 and YARN-10198 introduced some major changes to user group mapping, > and some of them unfortunately had some negative impact on the way mapping > works. > In some cases incorrect PlacementContexts were created, where full queue path > was passed as leaf queue name. This affects how the yarn cli app list > displays the queues. > u:%user:%primary_group.%user mapping fails with an incorrect validation error > when the %primary_group parent queue was a managed parent. > Group based rules in certain cases are mapped to root.[primary_group] rules, > loosing the ability to create deeper structures. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10284) Add lazy initialization of LogAggregationFileControllerFactory in LogServlet
[
https://issues.apache.org/jira/browse/YARN-10284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123851#comment-17123851
]
Hudson commented on YARN-10284:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18314 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18314/])
YARN-10284. Add lazy initialization of (snemeth: rev
aa6d13455b9435fb6c6d8f942c2b278dfada8f0c)
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/webapp/TestLogServlet.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogServlet.java
> Add lazy initialization of LogAggregationFileControllerFactory in LogServlet
>
>
> Key: YARN-10284
> URL: https://issues.apache.org/jira/browse/YARN-10284
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: log-aggregation, yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10284.001.patch, YARN-10284.002.patch,
> YARN-10284.003.patch, YARN-10284.004.patch, YARN-10284.branch-3.3.001.patch
>
>
> Suppose the {{mapred}} user has no access to the remote folder. Pinging the
> JHS if it's online in every few seconds will produce the following entry in
> the log:
> {noformat}
> 2020-05-19 00:17:20,331 WARN
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController:
> Unable to determine if the filesystem supports append operation
> java.nio.file.AccessDeniedException: test-bucket:
> org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: There is no mapped role
> for the group(s) associated with the authenticated user. (user: mapred)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:204)
> [...]
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:513)
> at
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController.getRollOverLogMaxSize(LogAggregationIndexedFileController.java:1157)
> at
> org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController.initInternal(LogAggregationIndexedFileController.java:149)
> at
> org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController.initialize(LogAggregationFileController.java:135)
> at
> org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileControllerFactory.(LogAggregationFileControllerFactory.java:139)
> at
> org.apache.hadoop.yarn.server.webapp.LogServlet.(LogServlet.java:66)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.(HsWebServices.java:99)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices$$FastClassByGuice$$1eb8d5d6.newInstance()
> at
> com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40)
> [...]
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> We should only create the {{LogAggregationFactory}} instance when we actually
> need it, not every time the {{LogServlet}} object is instantiated (so
> definitely not in the constructor). In this way we prevent pressure on the
> S3A auth side, especially if the authentication request is a costly operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10286) PendingContainers bugs in the scheduler outputs
[
https://issues.apache.org/jira/browse/YARN-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123827#comment-17123827
]
Hudson commented on YARN-10286:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18313 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18313/])
YARN-10286. PendingContainers bugs in the scheduler outputs. Contributed
(snemeth: rev e0a0741ac86dcc4f98ec2f9739b70b3697a4d0c0)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerQueueInfo.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md
> PendingContainers bugs in the scheduler outputs
> ---
>
> Key: YARN-10286
> URL: https://issues.apache.org/jira/browse/YARN-10286
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Critical
> Fix For: 3.4.0
>
> Attachments: YARN-10286.001.patch
>
>
> There are some problems with the {{ws/v1/cluster/scheduler}} output of the
> ResourceManager:
> Even though the pendingContainers field is
> [documented|https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API]
> in the FairScheduler output, it is
> [missing|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerQueueInfo.java]
> from the actual output. In case of the CapacityScheduler this field
> [exists|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java],
> but it is missing from the documentation. Let's fix both cases!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10108) FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping
[
https://issues.apache.org/jira/browse/YARN-10108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117685#comment-17117685
]
Hudson commented on YARN-10108:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18298 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18298/])
YARN-10108. FS-CS converter: nestedUserQueue with default rule results
(snemeth: rev 593af878c006953a2df5da504e59aa65505db438)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoQueueCreation.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
> FS-CS converter: nestedUserQueue with default rule results in invalid queue
> mapping
> ---
>
> Key: YARN-10108
> URL: https://issues.apache.org/jira/browse/YARN-10108
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Gergely Pollak
>Priority: Major
> Labels: fs2cs
> Fix For: 3.4.0, 3.3.1
>
> Attachments: YARN-10108.001.patch, YARN-10108.002.patch,
> YARN-10108.003.patch, YARN-10108.004.patch, YARN-10108.branch-3.3.001.patch,
> YARN-10108.branch-3.3.002.patch
>
>
> FS Queue Placement Policy
> {code:java}
>
>
>
>
>
> {code}
> gets mapped to an invalid CS queue mapping "u:%user:root.users.%user"
> RM fails to start with above queue mapping in CS
> {code:java}
> 2020-01-28 00:19:12,889 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
> ResourceManager
> org.apache.hadoop.service.ServiceStateException: java.io.IOException: mapping
> contains invalid or non-leaf queue [%user] and invalid parent queue
> [root.users]
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:173)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534)
> Caused by: java.io.IOException: mapping contains invalid or non-leaf queue
> [%user] and invalid parent queue [root.users]
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.QueuePlacementRuleUtils.validateQueueMappingUnderParentQueue(QueuePlacementRuleUtils.java:48)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.validateAndGetAutoCreatedQueueMapping(UserGroupMappingPlacementRule.java:363)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.initialize(UserGroupMappingPlacementRule.java:300)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getUserGroupMappingPlacementRule(CapacityScheduler.java:671)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updatePlacementRules(CapacityScheduler.java:712)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:753)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> ... 7 more
> {code}
> QueuePlacementConverter#handleNestedRule has to be fixed.
> {code:java}
> else if (pr instanceof DefaultPlacementRule) {
> DefaultPlacementRule defaultRule = (DefaultPlacementRule) pr;
> mapping.append("u:" + USER + ":")
> .append(defaultRule.defaultQueueName)
> .append("." + USER);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (YARN-6492) Generate queue metrics for each partition
[ https://issues.apache.org/jira/browse/YARN-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117179#comment-17117179 ] Hudson commented on YARN-6492: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18297 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18297/]) YARN-6492. Generate queue metrics for each partition. Contributed by (jhung: rev c30c23cb665761e997bcfc1dc00908f70b069fa2) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/PartitionQueueMetrics.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoCreatedQueueBase.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestPartitionQueueMetrics.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueMetrics.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoAppAttempt.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ContainerUpdateContext.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoQueueCreation.java > Generate queue metrics for each partition > - > > Key: YARN-6492 > URL: https://issues.apache.org/jira/browse/YARN-6492 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Jonathan Hung >Assignee: Manikandan R >Priority: Major > Fix For: 3.4.0 > > Attachments: PartitionQueueMetrics_default_partition.txt, > PartitionQueueMetrics_x_partition.txt, PartitionQueueMetrics_y_partition.txt, > YARN-6492-junits.patch, YARN-6492.001.patch, YARN-6492.002.patch, > YARN-6492.003.patch, YARN-6492.004.patch, YARN-6492.005.WIP.patch, > YARN-6492.006.WIP.patch, YARN-6492.007.WIP.patch, YARN-6492.008.WIP.patch, > YARN-6492.009.WIP.patch, YARN-6492.010.WIP.patch, YARN-6492.011.WIP.patch, > YARN-6492.012.WIP.patch, YARN-6492.013.patch, partition_metrics.txt > > > We are interested in having queue metrics for all partitions. Right now each > queue has one QueueMetrics object which captures metrics either in default > partition or across all partitions. (After YARN-6467 it will be in default > partition) > But having the partition metrics would be very useful. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10228) Yarn Service fails if am java opts contains ZK authentication file path
[
https://issues.apache.org/jira/browse/YARN-10228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112429#comment-17112429
]
Hudson commented on YARN-10228:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18281 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18281/])
YARN-10228. Relax restriction of file path character in (eyang: rev
726b8e324b6fc99aac5a26fbbc7edd26a3a25479)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/ServiceApiUtil.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/utils/TestServiceApiUtil.java
> Yarn Service fails if am java opts contains ZK authentication file path
> ---
>
> Key: YARN-10228
> URL: https://issues.apache.org/jira/browse/YARN-10228
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.3.0, 3.4.0, 3.3.1
>
> Attachments: YARN-10228.001.patch
>
>
> If i configure
> {code:java}
> yarn.service.am.java.opts=-Xmx768m
> -Djava.security.auth.login.config=/opt/hadoop/etc/jaas-zk.conf
> {code}
> Invalid character error is getting printed .
> This is due to jvm opts validation added in YARN-9718
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9606) Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient
[
https://issues.apache.org/jira/browse/YARN-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111817#comment-17111817
]
Hudson commented on YARN-9606:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18279 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18279/])
YARN-9606. Set sslfactory for AuthenticatedURL() while creating (pjoseph: rev
cef0756929458f65c4f6731fe21d659100bcc8ea)
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util/WebServiceClient.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
* (delete)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/webapp/TestWebServiceClient.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/util/TestWebServiceClient.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/Router.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
* (delete)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServiceClient.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/RouterWebServiceUtil.java
> Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient
> --
>
> Key: YARN-9606
> URL: https://issues.apache.org/jira/browse/YARN-9606
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-9606-001.patch, YARN-9606-002.patch,
> YARN-9606.003.patch, YARN-9606.004.patch, YARN-9606.005.patch,
> YARN-9606.006.patch, YARN-9606.007.patch, YARN-9606.008.patch
>
>
> Yarn logs fails for running containers
>
>
> {quote}
>
>
>
> Unable to fetch log files list
> Exception in thread "main" java.io.IOException:
> com.sun.jersey.api.client.ClientHandlerException:
> javax.net.ssl.SSLHandshakeException: Error while authenticating with
> endpoint:
> [https://vm2:65321/ws/v1/node/containers/container_e05_1559802125016_0001_01_08/logs]
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.getContainerLogFiles(LogsCLI.java:543)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.getMatchedContainerLogFiles(LogsCLI.java:1338)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.getMatchedOptionForRunningApp(LogsCLI.java:1514)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.fetchContainerLogs(LogsCLI.java:1052)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:367)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:152)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:399)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9606) Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient
[
https://issues.apache.org/jira/browse/YARN-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110026#comment-17110026
]
Hudson commented on YARN-9606:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18266 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18266/])
Revert "YARN-9606. Set sslfactory for AuthenticatedURL() while creating
(aajisaka: rev b65815d6914996fed25bd9fef4d37d00828bc664)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
* (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml
> Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient
> --
>
> Key: YARN-9606
> URL: https://issues.apache.org/jira/browse/YARN-9606
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-9606-001.patch, YARN-9606-002.patch,
> YARN-9606.003.patch
>
>
> Yarn logs fails for running containers
>
>
> {quote}
>
>
>
> Unable to fetch log files list
> Exception in thread "main" java.io.IOException:
> com.sun.jersey.api.client.ClientHandlerException:
> javax.net.ssl.SSLHandshakeException: Error while authenticating with
> endpoint:
> [https://vm2:65321/ws/v1/node/containers/container_e05_1559802125016_0001_01_08/logs]
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.getContainerLogFiles(LogsCLI.java:543)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.getMatchedContainerLogFiles(LogsCLI.java:1338)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.getMatchedOptionForRunningApp(LogsCLI.java:1514)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.fetchContainerLogs(LogsCLI.java:1052)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:367)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:152)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:399)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9606) Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient
[
https://issues.apache.org/jira/browse/YARN-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107350#comment-17107350
]
Hudson commented on YARN-9606:
--
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #18257 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18257/])
YARN-9606. Set sslfactory for AuthenticatedURL() while creating (pjoseph: rev
7836bc4c3533e93e7adc0c7da0659bc04bdf2494)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
* (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml
> Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient
> --
>
> Key: YARN-9606
> URL: https://issues.apache.org/jira/browse/YARN-9606
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-9606-001.patch, YARN-9606-002.patch,
> YARN-9606.003.patch
>
>
> Yarn logs fails for running containers
>
>
> {quote}
>
>
>
> Unable to fetch log files list
> Exception in thread "main" java.io.IOException:
> com.sun.jersey.api.client.ClientHandlerException:
> javax.net.ssl.SSLHandshakeException: Error while authenticating with
> endpoint:
> [https://vm2:65321/ws/v1/node/containers/container_e05_1559802125016_0001_01_08/logs]
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.getContainerLogFiles(LogsCLI.java:543)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.getMatchedContainerLogFiles(LogsCLI.java:1338)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.getMatchedOptionForRunningApp(LogsCLI.java:1514)
> at
> org.apache.hadoop.yarn.client.cli.LogsCLI.fetchContainerLogs(LogsCLI.java:1052)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:367)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:152)
> at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:399)
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10259) Reserved Containers not allocated from available space of other nodes in CandidateNodeSet in MultiNodePlacement
[
https://issues.apache.org/jira/browse/YARN-10259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107239#comment-17107239
]
Hudson commented on YARN-10259:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18256 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18256/])
YARN-10259. Fix reservation logic in Multi Node Placement. (pjoseph: rev
6ce295b78737aca8103912121d54f318cb5d36ef)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerMultiNodes.java
> Reserved Containers not allocated from available space of other nodes in
> CandidateNodeSet in MultiNodePlacement
> ---
>
> Key: YARN-10259
> URL: https://issues.apache.org/jira/browse/YARN-10259
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.2.0, 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10259-001.patch, YARN-10259-002.patch,
> YARN-10259-003.patch
>
>
> Reserved Containers are not allocated from the available space of other nodes
> in CandidateNodeSet in MultiNodePlacement.
> *Repro:*
> 1. MultiNode Placement Enabled.
> 2. Two nodes h1 and h2 with 8GB
> 3. Submit app1 AM (5GB) which gets placed in h1 and app2 AM (5GB) which gets
> placed in h2.
> 4. Submit app3 AM which is reserved in h1
> 5. Kill app2 which frees space in h2.
> 6. app3 AM never gets ALLOCATED
> RM logs shows YARN-8127 fix rejecting the allocation proposal for app3 AM on
> h2 as it expects the assignment to be on same node where reservation has
> happened.
> {code}
> 2020-05-05 18:49:37,264 DEBUG [AsyncDispatcher event handler]
> scheduler.SchedulerApplicationAttempt
> (SchedulerApplicationAttempt.java:commonReserve(573)) - Application attempt
> appattempt_1588684773609_0003_01 reserved container
> container_1588684773609_0003_01_01 on node host: h1:1234 #containers=1
> available= used=. This attempt
> currently has 1 reserved containers at priority 0; currentReservation
>
> 2020-05-05 18:49:37,264 INFO [AsyncDispatcher event handler]
> fica.FiCaSchedulerApp (FiCaSchedulerApp.java:apply(670)) - Reserved
> container=container_1588684773609_0003_01_01, on node=host: h1:1234
> #containers=1 available= used=
> with resource=
>RESERVED=[(Application=appattempt_1588684773609_0003_01;
> Node=h1:1234; Resource=)]
>
> 2020-05-05 18:49:38,283 DEBUG [Time-limited test]
> allocator.RegularContainerAllocator
> (RegularContainerAllocator.java:assignContainer(514)) - assignContainers:
> node=h2 application=application_1588684773609_0003 priority=0
> pendingAsk=,repeat=1>
> type=OFF_SWITCH
> 2020-05-05 18:49:38,285 DEBUG [Time-limited test] fica.FiCaSchedulerApp
> (FiCaSchedulerApp.java:commonCheckContainerAllocation(371)) - Try to allocate
> from reserved container container_1588684773609_0003_01_01, but node is
> not reserved
>ALLOCATED=[(Application=appattempt_1588684773609_0003_01;
> Node=h2:1234; Resource=)]
> {code}
> Attached testcase which reproduces the issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10265) Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue
[ https://issues.apache.org/jira/browse/YARN-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107179#comment-17107179 ] Hudson commented on YARN-10265: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18255 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18255/]) YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix (surendralilhore: rev 1958cb7c2bed08bd295a41fb4400b869a547b667) * (edit) hadoop-hdfs-project/hadoop-hdfs-client/pom.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml * (edit) hadoop-project/pom.xml * (edit) hadoop-hdfs-project/hadoop-hdfs/pom.xml > Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue > -- > > Key: YARN-10265 > URL: https://issues.apache.org/jira/browse/YARN-10265 > Project: Hadoop YARN > Issue Type: Bug >Reporter: liusheng >Assignee: liusheng >Priority: Major > Fix For: 3.3.0, 3.4.0, 3.3.1 > > Attachments: YARN-10265.001.patch > > > In the issue: https://issues.apache.org/jira/browse/YARN-9898 we have added a > workaround using an non-official released netty-4.1.48 to fix the ARM support > issue. but just a few hours agon, Netty has release the 4.1.50 version which > is officially support ARM platform, please see: > [https://github.com/netty/netty/pull/9804] > > netty-4.1.50.Final release: > [https://github.com/netty/netty/releases/tag/netty-4.1.50.Final] > commits from netty-4.1.48 to netty-4.1.50: > [https://github.com/netty/netty/compare/netty-4.1.48.Final...netty-4.1.50.Final] > So, now it is better to upgrade the netty-dependency version of Hadoop to > 4.1.50 version. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9898) Dependency netty-all-4.1.27.Final doesn't support ARM platform
[
https://issues.apache.org/jira/browse/YARN-9898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106614#comment-17106614
]
Hudson commented on YARN-9898:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18254 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18254/])
YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM (ayushsaxena:
rev 0918433b4da1affbe380988b8f63fca39bc0850b)
* (edit) hadoop-project/pom.xml
* (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml
* (edit) hadoop-hdfs-project/hadoop-hdfs-client/pom.xml
* (edit) hadoop-hdfs-project/hadoop-hdfs/pom.xml
> Dependency netty-all-4.1.27.Final doesn't support ARM platform
> --
>
> Key: YARN-9898
> URL: https://issues.apache.org/jira/browse/YARN-9898
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: liusheng
>Assignee: liusheng
>Priority: Major
> Fix For: 3.3.0, 3.4.0
>
> Attachments: YARN-9898.001.patch, YARN-9898.002.patch,
> YARN-9898.003.patch, YARN-9898.004.patch
>
>
> Hadoop dependent the Netty package, but the *netty-all-4.1.27.Final* of
> io.netty maven repo, cannot support ARM platform.
> When run the test *TestCsiClient.testIdentityService* on ARM server, it will
> raise error like following:
> {code:java}
> Caused by: java.io.FileNotFoundException:
> META-INF/native/libnetty_transport_native_epoll_aarch_64.so
> at
> io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:161)
> ... 45 more
> Suppressed: java.lang.UnsatisfiedLinkError: no
> netty_transport_native_epoll_aarch_64 in java.library.path
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
> at java.lang.Runtime.loadLibrary0(Runtime.java:870)
> at java.lang.System.loadLibrary(System.java:1122)
> at
> io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
> at
> io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243)
> at
> io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:124)
> ... 45 more
> Suppressed: java.lang.UnsatisfiedLinkError: no
> netty_transport_native_epoll_aarch_64 in java.library.path
> at
> java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
> at java.lang.Runtime.loadLibrary0(Runtime.java:870)
> at java.lang.System.loadLibrary(System.java:1122)
> at
> io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at
> io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)
> at
> io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
> ... 46 more
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8942) PriorityBasedRouterPolicy throws exception if all sub-cluster weights have negative value
[
https://issues.apache.org/jira/browse/YARN-8942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106498#comment-17106498
]
Hudson commented on YARN-8942:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18251 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18251/])
YARN-8942. PriorityBasedRouterPolicy throws exception if all sub-cluster
(inigoiri: rev 108ecf992f0004dd64a7143d1c400de1361b13f3)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/TestPriorityRouterPolicy.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/PriorityRouterPolicy.java
> PriorityBasedRouterPolicy throws exception if all sub-cluster weights have
> negative value
> -
>
> Key: YARN-8942
> URL: https://issues.apache.org/jira/browse/YARN-8942
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Akshay Agarwal
>Assignee: Bilwa S T
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: YARN-8942.001.patch, YARN-8942.002.patch
>
>
> In *PriorityBasedRouterPolicy* if all sub-cluster weights are *set to
> negative values* it is throwing exception while running a job.
> Ideally it should handle the negative priority as well according to the home
> sub cluster selection process of the policy.
> *Exception Details:*
> {code:java}
> java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Unable
> to insert the ApplicationId application_1540356760422_0015 into the
> FederationStateStore
> at
> org.apache.hadoop.yarn.server.router.RouterServerUtil.logAndThrowException(RouterServerUtil.java:56)
> at
> org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor.submitApplication(FederationClientInterceptor.java:418)
> at
> org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService.submitApplication(RouterClientRMService.java:218)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:282)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:579)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678)
> Caused by:
> org.apache.hadoop.yarn.server.federation.store.exception.FederationStateStoreInvalidInputException:
> Missing SubCluster Id information. Please try again by specifying Subcluster
> Id information.
> at
> org.apache.hadoop.yarn.server.federation.store.utils.FederationMembershipStateStoreInputValidator.checkSubClusterId(FederationMembershipStateStoreInputValidator.java:247)
> at
> org.apache.hadoop.yarn.server.federation.store.utils.FederationApplicationHomeSubClusterStoreInputValidator.checkApplicationHomeSubCluster(FederationApplicationHomeSubClusterStoreInputValidator.java:160)
> at
> org.apache.hadoop.yarn.server.federation.store.utils.FederationApplicationHomeSubClusterStoreInputValidator.validate(FederationApplicationHomeSubClusterStoreInputValidator.java:65)
> at
> org.apache.hadoop.yarn.server.federation.store.impl.ZookeeperFederationStateStore.addApplicationHomeSubCluster(ZookeeperFederationStateStore.java:159)
> at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
> at
>
[jira] [Commented] (YARN-10154) CS Dynamic Queues cannot be configured with absolute resources
[ https://issues.apache.org/jira/browse/YARN-10154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106159#comment-17106159 ] Hudson commented on YARN-10154: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18249 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18249/]) YARN-10154. Addendum Patch which fixes below bugs (pjoseph: rev 450e5aa9dd49eae46a0e05151bbddc56083eafd5) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceWithAutoQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ManagedParentQueue.java > CS Dynamic Queues cannot be configured with absolute resources > -- > > Key: YARN-10154 > URL: https://issues.apache.org/jira/browse/YARN-10154 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.3 >Reporter: Sunil G >Assignee: Manikandan R >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10154.001.patch, YARN-10154.002.patch, > YARN-10154.003.patch, YARN-10154.addendum-001.patch, > YARN-10154.addendum-002.patch, YARN-10154.addendum-003.patch, > YARN-10154.addendum-004.patch > > > In CS, ManagedParent Queue and its template cannot take absolute resource > value like > [memory=8192,vcores=8] > Thsi Jira is to track and improve the configuration reading module of > DynamicQueue to support absolute resource values. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10260) Allow transitioning queue from DRAINING to RUNNING state
[ https://issues.apache.org/jira/browse/YARN-10260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105648#comment-17105648 ] Hudson commented on YARN-10260: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18243 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18243/]) YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. (jhung: rev fff1d2c1226ec23841b04dd478e8b97f31abbeba) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueState.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java > Allow transitioning queue from DRAINING to RUNNING state > > > Key: YARN-10260 > URL: https://issues.apache.org/jira/browse/YARN-10260 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Bilwa S T >Priority: Major > Fix For: 3.2.2, 2.10.1, 3.4.0, 3.3.1, 3.1.5 > > Attachments: YARN-10260.001.patch > > > We found in our cluster, a queue was erroneously stopped. Then queue is > internally in DRAINING state. It cannot be moved back to RUNNING state until > the queue is finished draining. For queues with large workloads, this can > block other apps from submitting to this queue for a long time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9301) Too many InvalidStateTransitionException with SLS
[
https://issues.apache.org/jira/browse/YARN-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105546#comment-17105546
]
Hudson commented on YARN-9301:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18240 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18240/])
YARN-9301. Too many InvalidStateTransitionException with SLS. (inigoiri: rev
96bbc3bc972619bd830b2f935c06a1585a5470c6)
* (edit)
hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/resourcemanager/MockAMLauncher.java
> Too many InvalidStateTransitionException with SLS
> -
>
> Key: YARN-9301
> URL: https://issues.apache.org/jira/browse/YARN-9301
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin Chundatt
>Assignee: Bilwa S T
>Priority: Major
> Labels: simulator
> Fix For: 3.4.0
>
> Attachments: YARN-9301-001.patch, YARN-9301.002.patch
>
>
> Too many InvalidStateTransistionExcetion
> {noformat}
> 19/02/13 17:44:43 ERROR rmcontainer.RMContainerImpl: Can't handle this event
> at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> LAUNCHED at RUNNING
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:483)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:65)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.containerLaunchedOnNode(SchedulerApplicationAttempt.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.containerLaunchedOnNode(AbstractYarnScheduler.java:359)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.updateNewContainerInfo(AbstractYarnScheduler.java:1010)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.nodeUpdate(AbstractYarnScheduler.java:1112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:1295)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1752)
> at
> org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.handle(SLSCapacityScheduler.java:205)
> at
> org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.handle(SLSCapacityScheduler.java:60)
> at
> org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
> at java.lang.Thread.run(Thread.java:745)
> 19/02/13 17:44:43 ERROR rmcontainer.RMContainerImpl: Invalid event LAUNCHED
> on container container_1550059705491_0067_01_01
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9301) Too many InvalidStateTransitionException with SLS
[
https://issues.apache.org/jira/browse/YARN-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105533#comment-17105533
]
Hudson commented on YARN-9301:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18239 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18239/])
YARN-9301. Too many InvalidStateTransitionException with SLS. (inigoiri: rev
9cbd0cd2a9268ff2e8fed0af335e9c4f91c5f601)
* (edit)
hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/resourcemanager/MockAMLauncher.java
> Too many InvalidStateTransitionException with SLS
> -
>
> Key: YARN-9301
> URL: https://issues.apache.org/jira/browse/YARN-9301
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin Chundatt
>Assignee: Bilwa S T
>Priority: Major
> Labels: simulator
> Fix For: 3.4.0
>
> Attachments: YARN-9301-001.patch, YARN-9301.002.patch
>
>
> Too many InvalidStateTransistionExcetion
> {noformat}
> 19/02/13 17:44:43 ERROR rmcontainer.RMContainerImpl: Can't handle this event
> at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event:
> LAUNCHED at RUNNING
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:483)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:65)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.containerLaunchedOnNode(SchedulerApplicationAttempt.java:655)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.containerLaunchedOnNode(AbstractYarnScheduler.java:359)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.updateNewContainerInfo(AbstractYarnScheduler.java:1010)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.nodeUpdate(AbstractYarnScheduler.java:1112)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:1295)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1752)
> at
> org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.handle(SLSCapacityScheduler.java:205)
> at
> org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.handle(SLSCapacityScheduler.java:60)
> at
> org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
> at java.lang.Thread.run(Thread.java:745)
> 19/02/13 17:44:43 ERROR rmcontainer.RMContainerImpl: Invalid event LAUNCHED
> on container container_1550059705491_0067_01_01
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9017) PlacementRule order is not maintained in CS
[
https://issues.apache.org/jira/browse/YARN-9017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101178#comment-17101178
]
Hudson commented on YARN-9017:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18224 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18224/])
YARN-9017. PlacementRule order is not maintained in CS. Contributed by
(inigoiri: rev 35010120fbbcad8618f99abf7130e53f98879a33)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfigValidator.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestPlacementManager.java
> PlacementRule order is not maintained in CS
> ---
>
> Key: YARN-9017
> URL: https://issues.apache.org/jira/browse/YARN-9017
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Bibin Chundatt
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-9017.001.patch, YARN-9017.002.patch,
> YARN-9017.003.patch
>
>
> {{yarn.scheduler.queue-placement-rules}} doesn't work as expected in Capacity
> Scheduler
> {quote}
> * **Queue Mapping Interface based on Default or User Defined Placement
> Rules** - This feature allows users to map a job to a specific queue based on
> some default placement rule. For instance based on user & group, or
> application name. User can also define their own placement rule.
> {quote}
> As per current UserGroupMapping is always added in placementRule.
> {{CapacityScheduler#updatePlacementRules}}
> {code}
> // Initialize placement rules
> Collection placementRuleStrs = conf.getStringCollection(
> YarnConfiguration.QUEUE_PLACEMENT_RULES);
> List placementRules = new ArrayList<>();
> ...
> // add UserGroupMappingPlacementRule if absent
> distingushRuleSet.add(YarnConfiguration.USER_GROUP_PLACEMENT_RULE);
> {code}
> PlacementRule configuration order is not maintained
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8959) TestContainerResizing fails randomly
[
https://issues.apache.org/jira/browse/YARN-8959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101054#comment-17101054
]
Hudson commented on YARN-8959:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18222 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18222/])
YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via (jeagles:
rev 92e3ebb40199aec0890b868b8d6bf2d7fe90abbf)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java
> TestContainerResizing fails randomly
>
>
> Key: YARN-8959
> URL: https://issues.apache.org/jira/browse/YARN-8959
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin Chundatt
>Assignee: Ahmed Hussein
>Priority: Minor
> Fix For: 3.2.2, 2.10.1, 3.4.0, 3.3.1, 3.1.5
>
> Attachments: YARN-8959-branch-2.10.002.patch,
> YARN-8959-branch-2.10.003.patch, YARN-8959-branch-2.10.004.patch,
> YARN-8959.001.patch, YARN-8959.002.patch, YARN-8959.003.patch
>
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.testSimpleDecreaseContainer
> {code}
> testSimpleDecreaseContainer(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing)
> Time elapsed: 0.348 s <<< FAILURE!
> java.lang.AssertionError: expected:<1024> but was:<3072>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.checkUsedResource(TestContainerResizing.java:1011)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.testSimpleDecreaseContainer(TestContainerResizing.java:210)
> {code}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.testIncreaseContainerUnreservedWhenContainerCompleted
> {code}
> testIncreaseContainerUnreservedWhenContainerCompleted(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing)
> Time elapsed: 0.445 s <<< FAILURE!
> java.lang.AssertionError: expected:<1024> but was:<7168>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.checkUsedResource(TestContainerResizing.java:1011)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.testIncreaseContainerUnreservedWhenContainerCompleted(TestContainerResizing.java:729)
> {code}
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.testExcessiveReservationWhenDecreaseSameContainer
> {code}
> testExcessiveReservationWhenDecreaseSameContainer(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing)
> Time elapsed: 0.321 s <<< FAILURE!
> java.lang.AssertionError: expected:<1024> but was:<2048>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.checkUsedResource(TestContainerResizing.java:1015)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.testExcessiveReservationWhenDecreaseSameContainer(TestContainerResizing.java:623)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10257) FS-CS converter: skip increment properties for mem/vcores and fix DRF check
[
https://issues.apache.org/jira/browse/YARN-10257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099929#comment-17099929
]
Hudson commented on YARN-10257:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18218 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18218/])
YARN-10257. FS-CS converter: skip increment properties for mem/vcores (snemeth:
rev cb6399c1095af52112cbf4356572d99923d694ae)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSYarnSiteConverter.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverterMain.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSYarnSiteConverter.java
> FS-CS converter: skip increment properties for mem/vcores and fix DRF check
> ---
>
> Key: YARN-10257
> URL: https://issues.apache.org/jira/browse/YARN-10257
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.4.0, 3.3.1
>
> Attachments: YARN-10257-001.patch, YARN-10257-002.patch
>
>
> Two issues have been discovered during fs2cs testing:
> 1. The conversion of allocation increment properties are not needed:
> {{yarn.scheduler.increment-allocation-mb}}
> {{yarn.scheduler.increment-allocation-vcores}}
> {{yarn.resource-types.memory-mb.increment-allocation}}
> {{yarn.resource-types.vcores.increment-allocation}}
> 2. The following piece of code is incorrect - the default scheduling policy
> can be different from DRF, which is a problem if DRF is used everywhere else:
> {code}
> private boolean isDrfUsed(FairScheduler fs) {
> FSQueue rootQueue = fs.getQueueManager().getRootQueue();
> AllocationConfiguration allocConf = fs.getAllocationConfiguration();
> String defaultPolicy = allocConf.getDefaultSchedulingPolicy().getName();
> if (DominantResourceFairnessPolicy.NAME.equals(defaultPolicy)) {
> return true;
> } else {
> return isDrfUsedOnQueueLevel(rootQueue);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10160) Add auto queue creation related configs to RMWebService#CapacitySchedulerQueueInfo
[
https://issues.apache.org/jira/browse/YARN-10160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099803#comment-17099803
]
Hudson commented on YARN-10160:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18217 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18217/])
YARN-10160. Add auto queue creation related configs to (snemeth: rev
0debe55d6cf36b358c86c27d43991aa44baef4f2)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/WebServicesTestUtils.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/LeafQueueTemplateInfo.java
> Add auto queue creation related configs to
> RMWebService#CapacitySchedulerQueueInfo
> --
>
> Key: YARN-10160
> URL: https://issues.apache.org/jira/browse/YARN-10160
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: Screen Shot 2020-02-25 at 9.06.52 PM.png,
> YARN-10160-001.patch, YARN-10160-002.patch, YARN-10160-003.patch,
> YARN-10160-004.patch, YARN-10160-005.patch, YARN-10160-006.patch,
> YARN-10160-007.patch, YARN-10160-008.patch, YARN-10160-009.patch
>
>
> Add auto queue creation related configs to
> RMWebService#CapacitySchedulerQueueInfo.
> {code}
> yarn.scheduler.capacity..auto-create-child-queue.enabled
> yarn.scheduler.capacity..leaf-queue-template.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10256) Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic
[
https://issues.apache.org/jira/browse/YARN-10256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17099049#comment-17099049
]
Hudson commented on YARN-10256:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18213 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18213/])
YARN-10256. Refactor (jeagles: rev f5081a9a5d51fe781da97f07ac4dca8dd49c98e2)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java
> Refactor
> TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic
> ---
>
> Key: YARN-10256
> URL: https://issues.apache.org/jira/browse/YARN-10256
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: refactoring, unit-test
> Attachments: YARN-10256.001.patch
>
>
> In 3.x,
> {{TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic}}
> has redundant assertions. Since the UT throws timeout exception,
> {{GenericTestsUtils.waitFor()}} guarantees that the predicate is met
> successfully. Otherwise, the UT would throw a timeout exception.
> The redundant loop causes confusion in understanding the test unit and may
> increase the possibility of failure in case the container terminates
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6553) Replace MockResourceManagerFacade with MockRM for AMRMProxy/Router tests
[
https://issues.apache.org/jira/browse/YARN-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097861#comment-17097861
]
Hudson commented on YARN-6553:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18206 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18206/])
YARN-6553. Replace MockResourceManagerFacade with MockRM for (surendralilhore:
rev 257b04ce4e153cb88dfc59eab63625909efdd56b)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestFederationClientInterceptorRetry.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestRouterClientRMService.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestFederationClientInterceptor.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestableFederationClientInterceptor.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/rmadmin/MockRMAdminRequestInterceptor.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/BaseRouterClientRMTest.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/pom.xml
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/MockClientRequestInterceptor.java
> Replace MockResourceManagerFacade with MockRM for AMRMProxy/Router tests
>
>
> Key: YARN-6553
> URL: https://issues.apache.org/jira/browse/YARN-6553
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Giovanni Matteo Fumarola
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-6553.001.patch, YARN-6553.002.patch,
> YARN-6553.003.patch, YARN-6553.004.patch
>
>
> Currently the AMRMProxy and Router tests use the
> {{MockResourceManagerFacade}}. This jira proposes replacing it with
> {{MockRM}} as is done in majority of the tests.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6973) Adding RM Cluster Id in ApplicationReport
[ https://issues.apache.org/jira/browse/YARN-6973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095785#comment-17095785 ] Hudson commented on YARN-6973: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18200 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18200/]) YARN-6973. Adding RM Cluster Id in ApplicationReport. Contributed by (inigoiri: rev d125d3910843eeaa25dd09fae493c6fd258757e5) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java > Adding RM Cluster Id in ApplicationReport > -- > > Key: YARN-6973 > URL: https://issues.apache.org/jira/browse/YARN-6973 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-6973.001.patch, YARN-6973.002.patch, > YARN-6973.003.patch > > > Adding RM Cluster Id in ApplicationReport. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10247) Application priority queue ACLs are not respected
[
https://issues.apache.org/jira/browse/YARN-10247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095490#comment-17095490
]
Hudson commented on YARN-10247:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18198 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18198/])
YARN-10247. Application priority queue ACLs are not respected. (snemeth: rev
410c605aec308a2ccd903f60aade3aaeefcaa610)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationPriorityACLs.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
> Application priority queue ACLs are not respected
> -
>
> Key: YARN-10247
> URL: https://issues.apache.org/jira/browse/YARN-10247
> Project: Hadoop YARN
> Issue Type: Task
> Components: capacity scheduler
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Blocker
> Fix For: 3.3.0, 3.4.0
>
> Attachments: YARN-10247.0001.patch
>
>
> This is a regression from queue path jira.
> App priority acls are not working correctly.
> {code:java}
> yarn.scheduler.capacity.root.B.acl_application_max_priority=[user=john
> group=users max_priority=4]
> {code}
> max_priority enforcement is not working. For user john, maximum supported
> priority is 4. However I can submit like priority 6 for this user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10215) Endpoint for obtaining direct URL for the logs
[ https://issues.apache.org/jira/browse/YARN-10215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094701#comment-17094701 ] Hudson commented on YARN-10215: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18194 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18194/]) YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed (snemeth: rev ab3642955971dec1ce285f39cf0f02e6cc64b4b2) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-jhs-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-app-jhs-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-jhs-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/helpers/log-adapter-helper.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-jhs-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/YarnWebServiceParams.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-jhs-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-redirect-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogServlet.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-jhs-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-jhs-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-app-redirect-log.js * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebServices.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-jhs-redirect-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogWebService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-app-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-app/logs.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-redirect-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-jhs-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-app-jhs-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-jhs-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-jhs-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-log.js > Endpoint for obtaining direct URL for the logs > -- > > Key: YARN-10215 > URL: https://issues.apache.org/jira/browse/YARN-10215 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Andras Gyori >Priority: Major > Fix For: 3.3.0, 3.4.0 > > Attachments: YARN-10025.001.patch, YARN-10025.002.patch, > YARN-10025.003.patch, YARN-10025.004.patch > > > If CORS protected UIs are set up, there is an issue when the browser tries to > access the logs of a running container in the RM web UIv2. > Assuming ATS is not up, the browser follows the following call chain: > - Tries to access ATS, it fails, falls back to JHS > - From RM the browser received basic app info, we know that the application > is running > - From the JHS we got the list of containers and their log files. >
[jira] [Commented] (YARN-9848) revert YARN-4946
[ https://issues.apache.org/jira/browse/YARN-9848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17093926#comment-17093926 ] Hudson commented on YARN-9848: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18191 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18191/]) YARN-9848. Revert YARN-4946. Contributed by Steven Rand. (aajisaka: rev 6ef01646ba9540248ffaf322f4678fbdac28fcd4) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/AppManagerTestBase.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java > revert YARN-4946 > > > Key: YARN-9848 > URL: https://issues.apache.org/jira/browse/YARN-9848 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, resourcemanager >Reporter: Steven Rand >Assignee: Steven Rand >Priority: Blocker > Fix For: 3.3.0 > > Attachments: YARN-9848-01.patch, YARN-9848.002.patch, > YARN-9848.003.patch > > > In YARN-4946, we've been discussing a revert due to the potential for keeping > more applications in the state store than desired, and the potential to > greatly increase RM recovery times. > > I'm in favor of reverting the patch, but other ideas along the lines of > YARN-9571 would work as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4946) RM should not consider an application as COMPLETED when log aggregation is not in a terminal state
[ https://issues.apache.org/jira/browse/YARN-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17093927#comment-17093927 ] Hudson commented on YARN-4946: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18191 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18191/]) YARN-9848. Revert YARN-4946. Contributed by Steven Rand. (aajisaka: rev 6ef01646ba9540248ffaf322f4678fbdac28fcd4) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/AppManagerTestBase.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java > RM should not consider an application as COMPLETED when log aggregation is > not in a terminal state > -- > > Key: YARN-4946 > URL: https://issues.apache.org/jira/browse/YARN-4946 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-4946.001.patch, YARN-4946.002.patch, > YARN-4946.003.patch, YARN-4946.004.patch > > > MAPREDUCE-6415 added a tool that combines the aggregated log files for each > Yarn App into a HAR file. When run, it seeds the list by looking at the > aggregated logs directory, and then filters out ineligible apps. One of the > criteria involves checking with the RM that an Application's log aggregation > status is not still running and has not failed. When the RM "forgets" about > an older completed Application (e.g. RM failover, enough time has passed, > etc), the tool won't find the Application in the RM and will just assume that > its log aggregation succeeded, even if it actually failed or is still running. > We can solve this problem by doing the following: > The RM should not consider an app to be fully completed (and thus removed > from its history) until the aggregation status has reached a terminal state > (e.g. SUCCEEDED, FAILED, TIME_OUT). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10156) Fix typo 'complaint' which means quite different in Federation.md
[
https://issues.apache.org/jira/browse/YARN-10156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17093526#comment-17093526
]
Hudson commented on YARN-10156:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18188 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18188/])
YARN-10156. Destroy Jersey Client in TimelineConnector. (pjoseph: rev
9224568b0ea9c97071d45e1c26241c64efdd2926)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineConnector.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java
> Fix typo 'complaint' which means quite different in Federation.md
> -
>
> Key: YARN-10156
> URL: https://issues.apache.org/jira/browse/YARN-10156
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation, federation
>Reporter: Sungpeo Kook
>Assignee: Sungpeo Kook
>Priority: Minor
> Fix For: 3.3.0, 2.9.3, 3.2.2, 3.1.4, 2.10.1
>
>
> [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/Federation.html#Running_Applications_across_Sub-Clusters]
> There is a typo in Fedration.md that could be confusing.
> {quote}{{`be complaint to`}} should be {{`be compliant with`}} on the context.
> {quote}
> I think that it need to be corrected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10194) YARN RMWebServices /scheduler-conf/validate leaks ZK Connections
[
https://issues.apache.org/jira/browse/YARN-10194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17093063#comment-17093063
]
Hudson commented on YARN-10194:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18186 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18186/])
YARN-10194. YARN RMWebServices /scheduler-conf/validate leaks ZK (snemeth: rev
f91e21ac109e753e76d19c5c872c59a767b7b837)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfigValidator.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/FSSchedulerConfigurationStore.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/InMemoryConfigurationStore.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/YarnConfigurationStore.java
> YARN RMWebServices /scheduler-conf/validate leaks ZK Connections
>
>
> Key: YARN-10194
> URL: https://issues.apache.org/jira/browse/YARN-10194
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.3.0
>Reporter: Akhil PB
>Assignee: Prabhu Joseph
>Priority: Blocker
> Fix For: 3.3.0, 3.4.0
>
> Attachments: YARN-10194-001.patch, YARN-10194-002.patch,
> YARN-10194-003.patch, YARN-10194-004.patch, YARN-10194-005.patch
>
>
> YARN RMWebServices /scheduler-conf/validate leaks ZK Connections. Validation
> API creates a new CapacityScheduler and missed to close after the validation.
> Every CapacityScheduler#init opens MutableCSConfigurationProvider which opens
> ZKConfigurationStore and creates a ZK Connection.
> *ZK LOGS*
> {code}
> -03-12 16:45:51,881 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: [2
> times] Error accepting new connection: Too many connections from
> /172.27.99.64 - max is 60
> 2020-03-12 16:45:52,449 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new
> connection: Too many connections from /172.27.99.64 - max is 60
> 2020-03-12 16:45:52,710 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new
> connection: Too many connections from /172.27.99.64 - max is 60
> 2020-03-12 16:45:52,876 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: [4 times] Error accepting
> new connection: Too many connections from /172.27.99.64 - max is 60
> 2020-03-12 16:45:53,068 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: [2 times] Error accepting
> new connection: Too many connections from /172.27.99.64 - max is 60
> 2020-03-12 16:45:53,391 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: [2 times] Error accepting
> new connection: Too many connections from /172.27.99.64 - max is 60
> 2020-03-12 16:45:54,008 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new
> connection: Too many connections from /172.27.99.64 - max is 60
> 2020-03-12 16:45:54,287 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new
> connection: Too many connections from /172.27.99.64 - max is 60
> 2020-03-12 16:45:54,483 WARN
> org.apache.zookeeper.server.NIOServerCnxnFactory: [4 times] Error accepting
> new connection: Too many connections from /172.27.99.64 - max is 60
> {code}
> And there is an another bug in ZKConfigurationStore which has not handled
> close() of ZKCuratorManager.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10237) Add isAbsoluteResource config for queue in scheduler response
[ https://issues.apache.org/jira/browse/YARN-10237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091662#comment-17091662 ] Hudson commented on YARN-10237: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18179 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18179/]) YARN-10237. Add isAbsoluteResource config for queue in scheduler (snemeth: rev d4874585f4eb8620b64855d466ae82148ded0038) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java > Add isAbsoluteResource config for queue in scheduler response > - > > Key: YARN-10237 > URL: https://issues.apache.org/jira/browse/YARN-10237 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 3.4.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10237-001.patch, YARN-10237-002.patch, > YARN-10237-003.patch > > > Internal Config Management tools have difficulty in managing the capacity > scheduler queue configs if user toggles between Absolute Resource to > Percentage or vice versa. > This jira is to expose if a queue is configured in absolute resource or not > as part of scheduler response. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10223) Duplicate jersey-test-framework-core dependency in yarn-server-common
[
https://issues.apache.org/jira/browse/YARN-10223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091059#comment-17091059
]
Hudson commented on YARN-10223:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18176 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18176/])
YARN-10223. Remove jersey-test-framework-core dependency from (github: rev
9827ff2961e249e8be9023f39d91e1c0fbd23d61)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
> Duplicate jersey-test-framework-core dependency in yarn-server-common
> -
>
> Key: YARN-10223
> URL: https://issues.apache.org/jira/browse/YARN-10223
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Minor
>
> The following warning appears in maven log.
> {noformat}
> [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must
> be unique:
> com.sun.jersey.jersey-test-framework:jersey-test-framework-core:jar ->
> version (?) vs 1.19 @ line 148, column 17
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10240) Prevent Fatal CancelledException in TimelineV2Client when stopping
[
https://issues.apache.org/jira/browse/YARN-10240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088351#comment-17088351
]
Hudson commented on YARN-10240:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18170 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18170/])
YARN-10240. Prevent Fatal CancelledException in TimelineV2Client when (pjoseph:
rev 60fa15366e8f2d59f4dc8e7beaa6edcbbcb9c18f)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineV2ClientImpl.java
> Prevent Fatal CancelledException in TimelineV2Client when stopping
> --
>
> Key: YARN-10240
> URL: https://issues.apache.org/jira/browse/YARN-10240
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: ATSv2
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10240.001.patch
>
>
> When the timeline client is stopped, it will cancel all sync EntityHolders
> after waiting for a drain timeout.
> {code:java}
> // if some entities were not drained then we need interrupt
> // the threads which had put sync EntityHolders to the
> queue.
> EntitiesHolder nextEntityInTheQueue = null;
> while ((nextEntityInTheQueue =
> timelineEntityQueue.poll()) != null) {
> nextEntityInTheQueue.cancel(true);
> }
> {code}
> We only handle interrupted exception here.
> {code:java}
> if (sync) {
> // In sync call we need to wait till its published and if any error
> then
> // throw it back
> try {
> entitiesHolder.get();
> } catch (ExecutionException e) {
> throw new YarnException("Failed while publishing entity",
> e.getCause());
> } catch (InterruptedException e) {
> Thread.currentThread().interrupt();
> throw new YarnException("Interrupted while publishing entity", e);
> }
> }
> {code}
> But calling nextEntityInTheQueue.cancel(true) will result in
> entitiesHolder.get() throwing a CancelledException which is not handled. This
> can result in FATAL error in NM. We need to prevent this.
> {code:java}
> FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(203)) - Error in
> dispatcher thread
> java.util.concurrent.CancellationException
> at java.util.concurrent.FutureTask.report(FutureTask.java:121)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher.dispatchEntities(TimelineV2ClientImpl.java:545)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.putEntities(TimelineV2ClientImpl.java:149)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.putEntity(NMTimelinePublisher.java:348)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9996) Code cleanup in QueueAdminConfigurationMutationACLPolicy
[ https://issues.apache.org/jira/browse/YARN-9996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086375#comment-17086375 ] Hudson commented on YARN-9996: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18159 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18159/]) YARN-9996. Code cleanup in QueueAdminConfigurationMutationACLPolicy. (snemeth: rev 92b28e38b7bcfb70a3e805ff40ef73d27ba24788) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/QueueAdminConfigurationMutationACLPolicy.java > Code cleanup in QueueAdminConfigurationMutationACLPolicy > > > Key: YARN-9996 > URL: https://issues.apache.org/jira/browse/YARN-9996 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Siddharth Ahuja >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-9996.001.patch > > > Method 'isMutationAllowed' contains many uses of substring and lastIndexOf. > These could be extracted and simplified. > Also, some logging could be added as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10189) Code cleanup in LeveldbRMStateStore
[ https://issues.apache.org/jira/browse/YARN-10189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086370#comment-17086370 ] Hudson commented on YARN-10189: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18158 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18158/]) YARN-10189. Code cleanup in LeveldbRMStateStore. Contributed by Benjamin (snemeth: rev 76900b4f5b7ad8d11baf505abc43f7b24d364071) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestLeveldbRMStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/LeveldbConfigurationStore.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/DBManager.java > Code cleanup in LeveldbRMStateStore > --- > > Key: YARN-10189 > URL: https://issues.apache.org/jira/browse/YARN-10189 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10189.001.patch, YARN-10189.POC001.patch, > YARN-10189.POC002.patch > > > Some things can be improved: > * throws Exception declaration can be removed from > LeveldbRMStateStore.initInternal method > * key variable is redundant in LeveldbRMStateStore.dbStoreVersion > * try can use automatic Resource management in > LeveldbRMStateStore.loadReservationState/loadRMDTSecretManagerKeys/loadRMDTSecretManagerTokens/loadRMApps/... > etc > * there were some methods which were copied to LeveldbConfigurationStore > (ie: openDatabase, storeVersion, loadVersion, CompactionTimerClass nested > class), a helper class could be created to reduce the duplicated code > * Any other cleanup -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9954) Configurable max application tags and max tag length
[
https://issues.apache.org/jira/browse/YARN-9954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085960#comment-17085960
]
Hudson commented on YARN-9954:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18156 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18156/])
YARN-9954. Configurable max application tags and max tag length. (jhung: rev
360105482258569e5e31c49d023d2bbdc3ef38cd)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java
> Configurable max application tags and max tag length
>
>
> Key: YARN-9954
> URL: https://issues.apache.org/jira/browse/YARN-9954
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jonathan Hung
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.3.0, 3.2.2, 3.1.4, 2.10.1, 3.4.0
>
> Attachments: YARN-9954-branch-3.3.patch,
> YARN-9954-branch-3.3.v2.patch, YARN-9954-branch-3.3.v3.patch,
> YARN-9954.001.patch, YARN-9954.002.patch, YARN-9954.003.patch
>
>
> Currently max tags and max tag length is hardcoded, it should be configurable
> {noformat}
> @Evolving
> public static final int APPLICATION_MAX_TAGS = 10;
> @Evolving
> public static final int APPLICATION_MAX_TAG_LENGTH = 100; {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10154) CS Dynamic Queues cannot be configured with absolute resources
[ https://issues.apache.org/jira/browse/YARN-10154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085758#comment-17085758 ] Hudson commented on YARN-10154: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18154 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18154/]) YARN-10154. CS Dynamic Queues cannot be configured with absolute (sunilg: rev 2fe122e322afd35dfae6e45b3e7fdaddd20a411f) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoCreatedQueueBase.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractAutoCreatedLeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedLeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/queuemanagement/GuaranteedOrZeroCapacityOverTimePolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceWithAutoQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ManagedParentQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java > CS Dynamic Queues cannot be configured with absolute resources > -- > > Key: YARN-10154 > URL: https://issues.apache.org/jira/browse/YARN-10154 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.3 >Reporter: Sunil G >Assignee: Manikandan R >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10154.001.patch, YARN-10154.002.patch, > YARN-10154.003.patch > > > In CS, ManagedParent Queue and its template cannot take absolute resource > value like > [memory=8192,vcores=8] > Thsi Jira is to track and improve the configuration reading module of > DynamicQueue to support absolute resource values. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9999) TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, general code cleanup
[ https://issues.apache.org/jira/browse/YARN-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084020#comment-17084020 ] Hudson commented on YARN-: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18147 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18147/]) YARN-. TestFSSchedulerConfigurationStore: Extend from (snemeth: rev 4db598e0e61bb13baf9ec7f854c0435cf5c71aa2) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestFSSchedulerConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ConfigurationStoreBaseTest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/PersistentConfigurationStoreBaseTest.java > TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, > general code cleanup > --- > > Key: YARN- > URL: https://issues.apache.org/jira/browse/YARN- > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Benjamin Teke >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-.001.patch, YARN-.002.patch, > YARN-.003.patch, YARN-.004.patch > > > All config store tests are extended from ConfigurationStoreBaseTest: > * TestInMemoryConfigurationStore > * TestLeveldbConfigurationStore > * TestZKConfigurationStore > TestFSSchedulerConfigurationStore should also extend from it. > Additionally, some general code cleanup can be applied as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10233) [YARN UI2] No Logs were found in "YARN Daemon Logs" page
[ https://issues.apache.org/jira/browse/YARN-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083918#comment-17083918 ] Hudson commented on YARN-10233: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18146 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18146/]) YARN-10233. Fix YARN UI2 Daemon Logs (pjoseph: rev 94f7470c71104af1871a22c34fbbce14a60c33ab) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/.gitignore * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-rm-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-tools/yarn-rm-log.hbs * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-tools.hbs * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-rm-log.js > [YARN UI2] No Logs were found in "YARN Daemon Logs" page > > > Key: YARN-10233 > URL: https://issues.apache.org/jira/browse/YARN-10233 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Blocker > Fix For: 3.3.0 > > Attachments: YARN-10233.001.patch, > YARN_UI2_Tools_Daemon_Logs_Page_Fixed.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10234) FS-CS converter: don't enable auto-create queue property for root
[
https://issues.apache.org/jira/browse/YARN-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083826#comment-17083826
]
Hudson commented on YARN-10234:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18145 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18145/])
YARN-10234. FS-CS converter: don't enable auto-create queue property for
(snemeth: rev 55fcbcb5c2a096f98f273fda52ae25ecaa1d8bb6)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java
> FS-CS converter: don't enable auto-create queue property for root
> -
>
> Key: YARN-10234
> URL: https://issues.apache.org/jira/browse/YARN-10234
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Fix For: 3.3.0, 3.4.0
>
> Attachments: YARN-10234-001.patch, YARN-10234-002.patch
>
>
> The auto-create-child-queue property should not be enabled for root,
> otherwise it creates an exception inside capacity scheduler.
> {noformat}
> 2020-04-14 09:48:54,117 INFO org.apache.hadoop.ha.ActiveStandbyElector:
> Trying to re-establish ZK session
> 2020-04-14 09:48:54,117 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received
> RMFatalEvent of type TRANSITION_TO_ACTIVE_FAILED, caused by failure to
> refresh configuration settings: org.apache.hadoop.ha.ServiceFailedException:
> RefreshAll operation failed
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:772)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:307)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896)
> at
> org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:476)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:636)
> at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
> Caused by: java.io.IOException: Failed to re-init queues : null
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:467)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:489)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:430)
> at
> org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:761)
> ... 6 more
> Caused by: java.lang.ClassCastException
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10219) YARN service placement constraints is broken
[
https://issues.apache.org/jira/browse/YARN-10219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082936#comment-17082936
]
Hudson commented on YARN-10219:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18140 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18140/])
YARN-10219. Fix YARN Native Service Placement Constraints with Node (pjoseph:
rev c791b0e90e0d9c7cb05d162d605e0679942bcbfb)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestYarnNativeServices.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/utils/TestServiceApiUtil.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/Component.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/ServiceApiUtil.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/YarnServiceAPI.md
> YARN service placement constraints is broken
>
>
> Key: YARN-10219
> URL: https://issues.apache.org/jira/browse/YARN-10219
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.0, 3.2.0, 3.1.1, 3.1.2, 3.3.0, 3.2.1, 3.1.3
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Blocker
> Attachments: YARN-10219.001.patch, YARN-10219.002.patch,
> YARN-10219.003.patch, YARN-10219.004.patch, YARN-10219.005.patch
>
>
> YARN service placement constraint does not work with node label nor node
> attributes. Example of placement constraints:
> {code}
> "placement_policy": {
> "constraints": [
> {
> "type": "AFFINITY",
> "scope": "NODE",
> "node_attributes": {
> "label":["genfile"]
> },
> "target_tags": [
> "ping"
> ]
> }
> ]
> },
> {code}
> Node attribute added:
> {code} ./bin/yarn nodeattributes -add "host-3.example.com:label=genfile"
> {code}
> Scheduling activities shows:
> {code} Node does not match partition or placement constraints,
> unsatisfied PC expression="in,node,ping", target-type=ALLOCATION_TAG
>
> 1
> host-3.example.com:45454{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10212) Create separate configuration for max global AM attempts
[
https://issues.apache.org/jira/browse/YARN-10212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079664#comment-17079664
]
Hudson commented on YARN-10212:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18135 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18135/])
YARN-10212. Create separate configuration for max global AM attempts. (jhung:
rev 23481ad378de7f8e95eabefbd102825f757714b8)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
> Create separate configuration for max global AM attempts
>
>
> Key: YARN-10212
> URL: https://issues.apache.org/jira/browse/YARN-10212
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jonathan Hung
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.3.0, 3.2.2, 3.1.4, 2.10.1, 3.4.0
>
> Attachments: YARN-10212.001.patch, YARN-10212.002.patch,
> YARN-10212.003.patch, YARN-10212.004.patch
>
>
> Right now user's default max AM attempts is set to the same as global max AM
> attempts:
> {noformat}
> int globalMaxAppAttempts = conf.getInt(YarnConfiguration.RM_AM_MAX_ATTEMPTS,
> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS); {noformat}
> If we want to increase global max AM attempts, it will also increase the
> default. So we should create a separate global AM max attempts config to
> separate the two.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9995) Code cleanup in TestSchedConfCLI
[ https://issues.apache.org/jira/browse/YARN-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079378#comment-17079378 ] Hudson commented on YARN-9995: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18132 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18132/]) YARN-9995. Code cleanup in TestSchedConfCLI. Contributed by Bilwa S T. (snemeth: rev c04a484aedb09bdac7484908be1b8fcd1add2e30) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestSchedConfCLI.java > Code cleanup in TestSchedConfCLI > > > Key: YARN-9995 > URL: https://issues.apache.org/jira/browse/YARN-9995 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Bilwa S T >Priority: Minor > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-9995.001.patch, YARN-9995.002.patch, > YARN-9995.003.patch, YARN-9995.004.patch > > > Some tests are too verbose: > - add / delete / remove queues testcases: Creating SchedConfUpdateInfo > instances could be simplified with a helper method or something like that. > - Some fields can be converted to local variables: sysOutStream, sysOut, > sysErr, csConf > - Any additional cleanup -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10226) NPE in Capacity Scheduler while using %primary_group queue mapping
[
https://issues.apache.org/jira/browse/YARN-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079350#comment-17079350
]
Hudson commented on YARN-10226:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18131 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18131/])
YARN-10226. NPE in Capacity Scheduler while using %primary_group queue (sunilg:
rev 6fb29d0f1710cbef021e58846eb5c67a8d0b1f87)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
> NPE in Capacity Scheduler while using %primary_group queue mapping
> --
>
> Key: YARN-10226
> URL: https://issues.apache.org/jira/browse/YARN-10226
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Fix For: 3.3.0, 3.4.0
>
> Attachments: YARN-10226-001.patch
>
>
> If we use the following queue mapping:
> {{u:%user:%primary_group}}
> then we get a NPE inside ResourceManager:
> {noformat}
> 2020-04-06 11:59:13,883 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:serviceStart(881)) - Failed to load/recover state
> java.lang.NullPointerException
> at
> java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.getQueue(CapacitySchedulerQueueManager.java:138)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.getContextForPrimaryGroup(UserGroupMappingPlacementRule.java:163)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.getPlacementForUser(UserGroupMappingPlacementRule.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.getPlacementForApp(UserGroupMappingPlacementRule.java:227)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.PlacementManager.placeApplication(PlacementManager.java:67)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.placeApplication(RMAppManager.java:827)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:378)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:367)
> at
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:594)
> ...
> {noformat}
> We to check if parent queue is null in
> {{UserGroupMappingPlacementRule.getContextForPrimaryGroup()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5277) When localizers fail due to resource timestamps being out, provide more diagnostics
[ https://issues.apache.org/jira/browse/YARN-5277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17079181#comment-17079181 ] Hudson commented on YARN-5277: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18129 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18129/]) YARN-5277. When localizers fail due to resource timestamps being out, (snemeth: rev 8e4517f251e9f12917f2f457831664905015612f) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java > When localizers fail due to resource timestamps being out, provide more > diagnostics > --- > > Key: YARN-5277 > URL: https://issues.apache.org/jira/browse/YARN-5277 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Siddharth Ahuja >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-5277.001.patch, YARN-5277.002.patch > > > When an NM fails a resource D/L as the timestamps are wrong, there's not much > info, just two long values. > It would be good to also include the local time values, *and the current wall > time*. These are the things people need to know when trying to work out what > went wrong -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10063) Usage output of container-executor binary needs to include --http/--https argument
[ https://issues.apache.org/jira/browse/YARN-10063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077739#comment-17077739 ] Hudson commented on YARN-10063: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18126 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18126/]) YARN-10063. Add container-executor arguments --http/--https to usage. (wilfreds: rev 2214005c0f11955b2c50c4d2d4bd14947dd797ba) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c > Usage output of container-executor binary needs to include --http/--https > argument > -- > > Key: YARN-10063 > URL: https://issues.apache.org/jira/browse/YARN-10063 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Siddharth Ahuja >Assignee: Siddharth Ahuja >Priority: Minor > Attachments: YARN-10063.001.patch, YARN-10063.002.patch, > YARN-10063.003.patch, YARN-10063.004.patch > > > YARN-8448/YARN-6586 seems to have introduced a new option - "\--http" > (default) and "\--https" that is possible to be passed in to the > container-executor binary, see : > https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c#L564 > and > https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c#L521 > however, the usage output seems to have missed this: > https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c#L74 > Raising this jira to improve this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10207) CLOSE_WAIT socket connection leaks during rendering of (corrupted) aggregated logs on the JobHistoryServer Web UI
[
https://issues.apache.org/jira/browse/YARN-10207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077332#comment-17077332
]
Hudson commented on YARN-10207:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18124 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18124/])
YARN-10207. CLOSE_WAIT socket connection leaks during rendering of (snemeth:
rev bffb43b00e14a23d96f08b5a5df01e7f760b11ed)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java
> CLOSE_WAIT socket connection leaks during rendering of (corrupted) aggregated
> logs on the JobHistoryServer Web UI
> -
>
> Key: YARN-10207
> URL: https://issues.apache.org/jira/browse/YARN-10207
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Siddharth Ahuja
>Assignee: Siddharth Ahuja
>Priority: Major
> Fix For: 3.3.0, 3.4.0
>
> Attachments: YARN-10207.001.patch, YARN-10207.002.patch,
> YARN-10207.003.patch, YARN-10207.004.patch, YARN-10207.branch-3.2.001.patch
>
>
> File descriptor leaks are observed coming from the JobHistoryServer process
> while it tries to render a "corrupted" aggregated log on the JHS Web UI.
> Issue reproduced using the following steps:
> # Ran a sample Hadoop MR Pi job, it had the id -
> application_1582676649923_0026.
> # Copied an aggregated log file from HDFS to local FS:
> {code}
> hdfs dfs -get
> /tmp/logs/systest/logs/application_1582676649923_0026/_8041
> {code}
> # Updated the TFile metadata at the bottom of this file with some junk to
> corrupt the file :
> *Before:*
> {code}
>
> ^@^GVERSION*(^@_1582676649923_0026_01_03^F^Dnone^A^Pª5²ª5²^C^Qdata:BCFile.index^Dnoneª5þ^M^M^Pdata:TFile.index^Dnoneª5È66^Odata:TFile.meta^Dnoneª5Â^F^F^@^@^@^@^@^B6^K^@^A^@^@Ñ^QÓh<91>µ×¶9ßA@<92>ºáP
> {code}
> *After:*
> {code}
>
> ^@^GVERSION*(^@_1582676649923_0026_01_03^F^Dnone^A^Pª5²ª5²^C^Qdata:BCFile.index^Dnoneª5þ^M^M^Pdata:TFile.index^Dnoneª5È66^Odata:TFile.meta^Dnoneª5Â^F^F^@^@^@^@^@^B6^K^@^A^@^@Ñ^QÓh<91>µ×¶9ßA@<92>ºáPblah
> {code}
> Notice "blah" (junk) added at the very end.
> # Remove the existing aggregated log file that will need to be replaced by
> our modified copy from step 3 (as otherwise HDFS will prevent it from placing
> the file with the same name as it already exists):
> {code}
> hdfs dfs -rm -r -f
> /tmp/logs/systest/logs/application_1582676649923_0026/_8041
> {code}
> # Upload the corrupted aggregated file back to HDFS:
> {code}
> hdfs dfs -put _8041
> /tmp/logs/systest/logs/application_1582676649923_0026
> {code}
> # Visit HistoryServer Web UI
> # Click on job_1582676649923_0026
> # Click on "logs" link against the AM (assuming the AM ran on nm_hostname)
> # Review the JHS logs, following exception will be seen:
> {code}
> 2020-03-24 20:03:48,484 ERROR org.apache.hadoop.yarn.webapp.View: Error
> getting logs for job_1582676649923_0026
> java.io.IOException: Not a valid BCFile.
> at
> org.apache.hadoop.io.file.tfile.BCFile$Magic.readAndVerify(BCFile.java:927)
> at
> org.apache.hadoop.io.file.tfile.BCFile$Reader.(BCFile.java:628)
> at
> org.apache.hadoop.io.file.tfile.TFile$Reader.(TFile.java:804)
> at
> org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogReader.(AggregatedLogFormat.java:588)
> at
> org.apache.hadoop.yarn.logaggregation.filecontroller.tfile.TFileAggregatedLogsBlock.render(TFileAggregatedLogsBlock.java:111)
> at
> org.apache.hadoop.yarn.logaggregation.filecontroller.tfile.LogAggregationTFileController.renderAggregatedLogsBlock(LogAggregationTFileController.java:341)
> at
> org.apache.hadoop.yarn.webapp.log.AggregatedLogsBlock.render(AggregatedLogsBlock.java:117)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:235)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49)
> at
> org.apache.hadoop.yarn.webapp.hamlet2.HamletImpl$EImp._v(HamletImpl.java:117)
> at
> org.apache.hadoop.yarn.webapp.hamlet2.Hamlet$TD.__(Hamlet.java:848)
> at
> org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:71)
> at
>
[jira] [Commented] (YARN-10001) Add explanation of unimplemented methods in InMemoryConfigurationStore
[ https://issues.apache.org/jira/browse/YARN-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077270#comment-17077270 ] Hudson commented on YARN-10001: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18123 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18123/]) YARN-10001. Add explanation of unimplemented methods in (snemeth: rev 45362a9f4cbe512ee4cd6b7f65aa47d59fee612e) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/InMemoryConfigurationStore.java > Add explanation of unimplemented methods in InMemoryConfigurationStore > -- > > Key: YARN-10001 > URL: https://issues.apache.org/jira/browse/YARN-10001 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Siddharth Ahuja >Priority: Major > Fix For: 3.3.0, 3.4.0 > > Attachments: YARN-10001.001.patch, YARN-10001.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10120) In Federation Router Nodes/Applications/About pages throws 500 exception when https is enabled
[
https://issues.apache.org/jira/browse/YARN-10120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074379#comment-17074379
]
Hudson commented on YARN-10120:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18114 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18114/])
YARN-10120. Amendment fix for Java Doc. (pjoseph: rev
7a39da3537f09b13246a27667199d629b5e62a5b)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServiceClient.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/DefaultRequestInterceptorREST.java
> In Federation Router Nodes/Applications/About pages throws 500 exception when
> https is enabled
> --
>
> Key: YARN-10120
> URL: https://issues.apache.org/jira/browse/YARN-10120
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Reporter: Sushanta Sen
>Assignee: Bilwa S T
>Priority: Critical
> Fix For: 3.4.0
>
> Attachments: YARN-10120-YARN-7402.patch,
> YARN-10120-addendum-01.patch, YARN-10120.001.patch, YARN-10120.002.patch
>
>
> In Federation Router Nodes/Applications/About pages throws 500 exception when
> https is enabled.
> yarn.router.webapp.https.address =router ip:8091
> {noformat}
> 2020-02-07 16:38:49,990 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
> handling URI: /cluster/apps
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:166)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
> at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
> at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
> at
> com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
> at
> org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644)
> at
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
> at
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1622)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
> at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
[jira] [Commented] (YARN-10120) In Federation Router Nodes/Applications/About pages throws 500 exception when https is enabled
[
https://issues.apache.org/jira/browse/YARN-10120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073710#comment-17073710
]
Hudson commented on YARN-10120:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18112 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18112/])
YARN-10120. Https Support in Router WebServiceClient. (pjoseph: rev
25361b077bd34606259dac5a00c41faddd2dfc7d)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/AboutBlock.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/DefaultRequestInterceptorREST.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/RouterWebServiceUtil.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/AppsBlock.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/NodesBlock.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServiceClient.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/webapp/TestWebServiceClient.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/Router.java
> In Federation Router Nodes/Applications/About pages throws 500 exception when
> https is enabled
> --
>
> Key: YARN-10120
> URL: https://issues.apache.org/jira/browse/YARN-10120
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Reporter: Sushanta Sen
>Assignee: Bilwa S T
>Priority: Critical
> Fix For: 3.3.0
>
> Attachments: YARN-10120.001.patch, YARN-10120.002.patch
>
>
> In Federation Router Nodes/Applications/About pages throws 500 exception when
> https is enabled.
> yarn.router.webapp.https.address =router ip:8091
> {noformat}
> 2020-02-07 16:38:49,990 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error
> handling URI: /cluster/apps
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:166)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> at
> com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277)
> at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182)
> at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133)
> at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130)
> at
> com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203)
> at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
> at
> org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767)
> at
>
[jira] [Commented] (YARN-10202) Fix documentation about NodeAttributes.
[
https://issues.apache.org/jira/browse/YARN-10202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17072469#comment-17072469
]
Hudson commented on YARN-10202:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18106 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18106/])
YARN-10202. Fix documentation about NodeAttributes. Contributed by Sen
(aajisaka: rev c162648aff68552d87db8a013b850c17fee762c0)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeAttributes.md
> Fix documentation about NodeAttributes.
> ---
>
> Key: YARN-10202
> URL: https://issues.apache.org/jira/browse/YARN-10202
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.2.1
>Reporter: Sen Zhao
>Assignee: Sen Zhao
>Priority: Minor
> Fix For: 3.3.0, 3.2.2
>
> Attachments: YARN-10202.001.patch
>
>
> {noformat:title=NodeAttributes.md}
> The above SchedulingRequest requests for 1 container on nodes that must
> satisfy following constraints:
> 1. Node attribute *`rm.yarn.io/python`* doesn't exist on the node or it exist
> but its value is not equal to 3
> 2. Node attribute *`rm.yarn.io/java`* must exist on the node and its value is
> equal to 1.8
> {noformat}
> should be
> {noformat}
> The above SchedulingRequest requests for 1 container on nodes that must
> satisfy following constraints:
> 1. Node attribute *`rm.yarn.io/python`* doesn't exist on the node or it exist
> but its value is not equal to 3
> 2. Node attribute *`rm.yarn.io/java`* must exist on the node and its value is
> equal to 1.8
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10210) Add a RMFailoverProxyProvider that does DNS resolution on failover
[ https://issues.apache.org/jira/browse/YARN-10210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067778#comment-17067778 ] Hudson commented on YARN-10210: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18095 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18095/]) YARN-10210. Add a RMFailoverProxyProvider that does DNS resolution on (github: rev 50f7f6dfd10031df142cb9105bddf2d0cd74f017) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestNoHaRMFailoverProxyProvider.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/AutoRefreshNoHARMFailoverProxyProvider.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/AutoRefreshRMFailoverProxyProvider.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailoverProxyProvider.java > Add a RMFailoverProxyProvider that does DNS resolution on failover > -- > > Key: YARN-10210 > URL: https://issues.apache.org/jira/browse/YARN-10210 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.1.2 >Reporter: Roger Liu >Assignee: Roger Liu >Priority: Major > Fix For: 3.3.0 > > > In Kubernetes, the a node may go down and then come back later with a > different IP address. YARN clients which are already running will be unable > to rediscover the node after it comes back up due to caching the original IP > address. This is problematic for cases such as Spark HA on Kubernetes, as the > node containing the resource manager may go down and come back up, meaning > existing node managers must then also be restarted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10043) FairOrderingPolicy Improvements
[ https://issues.apache.org/jira/browse/YARN-10043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067645#comment-17067645 ] Hudson commented on YARN-10043: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18094 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18094/]) YARN-10043. FairOrderingPolicy Improvements. Contributed by Manikandan R (snemeth: rev 3d5ade1839205db1c4a11f73bd02d847187f48ca) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/SchedulableEntity.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/MockSchedulableEntity.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FairOrderingPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java > FairOrderingPolicy Improvements > --- > > Key: YARN-10043 > URL: https://issues.apache.org/jira/browse/YARN-10043 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-10043.001.patch, YARN-10043.002.patch, > YARN-10043.003.patch, YARN-10043.004.patch > > > FairOrderingPolicy can be improved by using some of the approaches (only > relevant) implemented in FairSharePolicy of FS. This improvement has > significance in FS to CS migration context. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
