[jira] [Commented] (YARN-10383) YarnCommands.md is inconsistent with the source code
[ https://issues.apache.org/jira/browse/YARN-10383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17170526#comment-17170526 ] Hudson commented on YARN-10383: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18491 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18491/]) YARN-10383. YarnCommands.md is inconsistent with the source code (#2177) (github: rev ab2b3df2de1d7c490a4aa83460328bcd910c39d4) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/YarnCommands.md > YarnCommands.md is inconsistent with the source code > > > Key: YARN-10383 > URL: https://issues.apache.org/jira/browse/YARN-10383 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Reporter: zhaoshengjie >Assignee: zhaoshengjie >Priority: Minor > Fix For: 3.4.0 > > Attachments: 01.png, 02.png > > > The markdown document (YarnCommands.md) is inconsistent with the source code > !01.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10229) [Federation] Client should be able to submit application to RM directly using normal client conf
[ https://issues.apache.org/jira/browse/YARN-10229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17169793#comment-17169793 ] Hudson commented on YARN-10229: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18489 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18489/]) YARN-10229. [Federation] Client should be able to submit application to (brahma: rev eac558380fd7d3c2e78b8956e2080688bb1dd8bb) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestAMRMProxyService.java > [Federation] Client should be able to submit application to RM directly using > normal client conf > > > Key: YARN-10229 > URL: https://issues.apache.org/jira/browse/YARN-10229 > Project: Hadoop YARN > Issue Type: Bug > Components: amrmproxy, federation >Affects Versions: 3.1.1 >Reporter: JohnsonGuo >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10229.001.patch, YARN-10229.002.patch, > YARN-10229.003.patch, YARN-10229.004.patch, YARN-10229.005.patch, > YARN-10229.006.patch, YARN-10229.007.patch, YARN-10229.008.patch > > > Scenario: When enable the yarn federation feature with multi yarn clusters, > one can submit their job to yarn-router by *modified* their client > configuration with yarn router address. > But if one still wants to submit their jobs via the original client (before > enable federation) to RM directly, it will encounter the AMRMToken exception. > That means once enable federation ,if some one want to submit job, they have > to modify the client conf. > > one possible solution for this Scenario is: > In NodeManger, when the client ApplicationMaster request comes: > * get the client job.xml from HDFS "". > * parse the "yarn.resourcemanager.scheduler.address" parameter in job.xml > * if the value of the parameter is "localhost:8049"(AMRM address),then do > the AMRMToken valid process > * if the value of the parameter is "rm:port"(rm address),then skip the > AMRMToken valid process > > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10359) Log container report only if list is not empty
[ https://issues.apache.org/jira/browse/YARN-10359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17169247#comment-17169247 ] Hudson commented on YARN-10359: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18485 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18485/]) YARN-10359. Log container report only if list is not empty. Contributed (bibinchundatt: rev 5323e83edfe63355ec38ffdaacc0c27d14cad31c) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java > Log container report only if list is not empty > -- > > Key: YARN-10359 > URL: https://issues.apache.org/jira/browse/YARN-10359 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10359.001.patch, YARN-10359.002.patch > > > In NodeStatusUpdaterImpl print log only if containerReports list is not empty > {code:java} > if (containerReports != null) { > LOG.info("Registering with RM using containers :" + containerReports); > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1529) Add Localization overhead metrics to NM
[ https://issues.apache.org/jira/browse/YARN-1529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17168064#comment-17168064 ] Hudson commented on YARN-1529: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18481 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18481/]) YARN-1529: Add Localization overhead metrics to NM. Contributed by (ericp: rev e0c9653166df48a47267dbc81d124ab78267e039) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/Container.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerResourceLocalizedEvent.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/LocalizedResource.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/MockContainer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/metrics/NodeManagerMetrics.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationConstants.java > Add Localization overhead metrics to NM > --- > > Key: YARN-1529 > URL: https://issues.apache.org/jira/browse/YARN-1529 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Gera Shegalov >Assignee: Jim Brennan >Priority: Major > Attachments: YARN-1529.005.patch, YARN-1529.006.patch, > YARN-1529.v01.patch, YARN-1529.v02.patch, YARN-1529.v03.patch, > YARN-1529.v04.patch > > > Users are often unaware of localization cost that their jobs incur. To > measure effectiveness of localization caches it is necessary to expose the > overhead in the form of metrics. > We propose addition of the following metrics to NodeManagerMetrics. > When a container is about to launch, its set of LocalResources has to be > fetched from a central location, typically on HDFS, that results in a number > of download requests for the files missing in caches. > LocalizedFilesMissed: total files (requests) downloaded from DFS. Cache > misses. > LocalizedFilesCached: total localization requests that were served from local > caches. Cache hits. > LocalizedBytesMissed: total bytes downloaded from DFS due to cache misses. > LocalizedBytesCached: total bytes satisfied from local caches. > Localized(Files|Bytes)CachedRatio: percentage of localized (files|bytes) that > were served out of cache: ratio = 100 * caches / (caches + misses) > LocalizationDownloadNanos: total elapsed time in nanoseconds for a container > to go from ResourceRequestTransition to LocalizedTransition -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10369) Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG
[ https://issues.apache.org/jira/browse/YARN-10369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17167426#comment-17167426 ] Hudson commented on YARN-10369: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18479 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18479/]) YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId (bibinchundatt: rev 5d8600e80ad7864b332b60d5a01585fdf00848ee) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/NMTokenSecretManagerInRM.java > Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG > -- > > Key: YARN-10369 > URL: https://issues.apache.org/jira/browse/YARN-10369 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.4.0 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10369.001.patch, YARN-10369.002.patch > > > This message is logged at the info level, but it doesn't really add much > information. > We changed this to DEBUG internally years ago and haven't missed it. > {noformat} > 2020-07-27 21:51:29,027 INFO [RM Event dispatcher] > security.NMTokenSecretManagerInRM > (NMTokenSecretManagerInRM.java:createAndGetNMToken(200)) - Sending NMToken > for nodeId : localhost.localdomain:45454 for container : > container_1595886659189_0001_01_01 > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10343) Legacy RM UI should include labeled metrics for allocated, total, and reserved resources.
[ https://issues.apache.org/jira/browse/YARN-10343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17166693#comment-17166693 ] Hudson commented on YARN-10343: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18477 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18477/]) YARN-10343. Legacy RM UI should include labeled metrics for allocated, (jhung: rev 3eaf62726ffe90b3b096798fe501abd1ed0c5f15) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ResourceUsage.java > Legacy RM UI should include labeled metrics for allocated, total, and > reserved resources. > - > > Key: YARN-10343 > URL: https://issues.apache.org/jira/browse/YARN-10343 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.10.0, 3.2.1, 3.1.3 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > Fix For: 3.2.2, 2.10.1, 3.4.0, 3.3.1, 3.1.5 > > Attachments: Screen Shot 2020-07-07 at 1.00.22 PM.png, Screen Shot > 2020-07-07 at 1.03.26 PM.png, YARN-10343.000.patch, YARN-10343.001.patch, > YARN-10343.branch-2.10.001.patch, YARN-10343.branch-3.2.001.patch > > > The current legacy RM UI only includes resources metrics for the default > partition. If a cluster has labeled nodes, those are not included in the > resource metrics for allocated, total, and reserved resources. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10208) Add capacityScheduler metric for NODE_UPDATE interval
[ https://issues.apache.org/jira/browse/YARN-10208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17166337#comment-17166337 ] Hudson commented on YARN-10208: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18476 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18476/]) YARN-10208. Add capacityScheduler metric for NODE_UPDATE interval. (bibinchundatt: rev 5dadf963d3639cc6d37902d9c7beaacdafac0e9c) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerMetrics.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestCapacitySchedulerMetrics.java > Add capacityScheduler metric for NODE_UPDATE interval > - > > Key: YARN-10208 > URL: https://issues.apache.org/jira/browse/YARN-10208 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Pranjal Protim Borah >Assignee: Pranjal Protim Borah >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10208.001.patch, YARN-10208.002.patch, > YARN-10208.003.patch, YARN-10208.004.patch, YARN-10208.005.patch, > YARN-10208.006.patch, YARN-10208.007.patch > > > Metric measuring average time interval between node heartbeats in capacity > scheduler on node update event. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10366) Yarn rmadmin help message shows two labels for one node for --replaceLabelsOnNode
[ https://issues.apache.org/jira/browse/YARN-10366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165488#comment-17165488 ] Hudson commented on YARN-10366: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18474 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18474/]) YARN-10366. Fix Yarn rmadmin help message shows two labels for one node (pjoseph: rev 60a254621a3d07a6c3c4611ea86cffd9625e8fb2) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java > Yarn rmadmin help message shows two labels for one node for > --replaceLabelsOnNode > - > > Key: YARN-10366 > URL: https://issues.apache.org/jira/browse/YARN-10366 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Tanu Ajmera >Assignee: Tanu Ajmera >Priority: Major > Fix For: 3.4.0 > > Attachments: Screenshot 2020-07-24 at 4.07.10 PM.png, > YARN-10366-001.patch > > > In the help message of “yarn rmadmin” , looks like one node can be assign > with two labels, which is not consistent with the “Each node can have only > one node label” -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10362) Javadoc for TimelineReaderAuthenticationFilterInitializer is broken
[ https://issues.apache.org/jira/browse/YARN-10362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165291#comment-17165291 ] Hudson commented on YARN-10362: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18473 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18473/]) YARN-10362. Javadoc for TimelineReaderAuthenticationFilterInitializer is (aajisaka: rev d02be17a269cb7f5ac51910802683a1729510250) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/security/TimelineReaderAuthenticationFilterInitializer.java > Javadoc for TimelineReaderAuthenticationFilterInitializer is broken > --- > > Key: YARN-10362 > URL: https://issues.apache.org/jira/browse/YARN-10362 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Minor > Fix For: 3.4.0 > > Attachments: HADOOP-17148.000.patch > > > mvn javadoc:javadoc fails for > TimelineReaderAuthenticationFilterInitializer.java > {code:java} > [ERROR] > /Users/sri/projects/hadoop-mirror/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/security/TimelineReaderAuthenticationFilterInitializer.java:39: > error: value does not refer to a constant > [ERROR]* {@value TimelineAuthenticationFilterInitializer#PREFIX}. > [ERROR] ^ > [ERROR] > /Users/sri/projects/hadoop-mirror/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/security/TimelineReaderAuthenticationFilterInitializer.java:39: > error: reference not found > [ERROR]* {@value TimelineAuthenticationFilterInitializer#PREFIX}. > {code} > This issue seems to be caused by changes in YARN-10339 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10367) Failed to get nodejs 10.21.0 when building docker image
[ https://issues.apache.org/jira/browse/YARN-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17165255#comment-17165255 ] Hudson commented on YARN-10367: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18472 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18472/]) YARN-10367. Failed to get nodejs 10.21.0 when building docker image (github: rev e277d338da2675262b4d6985e08c78e408b927a3) * (edit) dev-support/docker/Dockerfile_aarch64 * (edit) dev-support/docker/Dockerfile > Failed to get nodejs 10.21.0 when building docker image > --- > > Key: YARN-10367 > URL: https://issues.apache.org/jira/browse/YARN-10367 > Project: Hadoop YARN > Issue Type: Bug > Components: build, webapp >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Blocker > Fix For: 3.4.0, 3.3.1 > > > https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/213/console > {noformat} > Version '10.21.0-1nodesource1' for 'nodejs' was not found > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4771) Some containers can be skipped during log aggregation after NM restart
[ https://issues.apache.org/jira/browse/YARN-4771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164670#comment-17164670 ] Hudson commented on YARN-4771: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18470 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18470/]) YARN-4771. Some containers can be skipped during log aggregation after (ebadger: rev ac5f21dbef0f0ad4210e4027f53877760fa606a5) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java > Some containers can be skipped during log aggregation after NM restart > -- > > Key: YARN-4771 > URL: https://issues.apache.org/jira/browse/YARN-4771 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.10.0, 3.2.1, 3.1.3 >Reporter: Jason Darrell Lowe >Assignee: Jim Brennan >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-4771.001.patch, YARN-4771.002.patch, > YARN-4771.003.patch > > > A container can be skipped during log aggregation after a work-preserving > nodemanager restart if the following events occur: > # Container completes more than > yarn.nodemanager.duration-to-track-stopped-containers milliseconds before the > restart > # At least one other container completes after the above container and before > the restart -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10319) Record Last N Scheduler Activities from ActivitiesManager
[ https://issues.apache.org/jira/browse/YARN-10319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164184#comment-17164184 ] Hudson commented on YARN-10319: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18468 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18468/]) YARN-10319. Record Last N Scheduler Activities from ActivitiesManager (pjoseph: rev 247eb0979b6a3a723ea9a249ba4db1ee079eb909) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/RouterWebServices.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/FederationInterceptorREST.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/DefaultRequestInterceptorREST.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWSConsts.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesSchedulerActivities.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/JAXBContextResolver.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/ActivitiesTestUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServiceProtocol.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/MockRESTRequestInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/PassThroughRESTRequestInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/activities/ActivitiesManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/BulkActivitiesInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/webapp/BaseRouterWebServicesTest.java > Record Last N Scheduler Activities from ActivitiesManager > - > > Key: YARN-10319 > URL: https://issues.apache.org/jira/browse/YARN-10319 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Labels: activitiesmanager > Fix For: 3.4.0 > > Attachments: Screen Shot 2020-06-18 at 1.26.31 PM.png, > YARN-10319-001-WIP.patch, YARN-10319-002.patch, YARN-10319-003.patch, > YARN-10319-004.patch, YARN-10319-005.patch, YARN-10319-006.patch > > > ActivitiesManager records a call flow for a given nodeId or a last call flow. > This is useful when debugging the issue live where the user queries with > right nodeId. But capturing last N scheduler activities during the issue > period can help to debug the issue offline. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10315) Avoid sending RMNodeResourceupdate event if resource is same
[ https://issues.apache.org/jira/browse/YARN-10315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17163460#comment-17163460 ] Hudson commented on YARN-10315: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18467 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18467/]) YARN-10315. Avoid sending RMNodeResourceupdate event if resource is (bibinchundatt: rev bfcd775381f1e0b94b17ce3cfca7eade95df1ea8) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java > Avoid sending RMNodeResourceupdate event if resource is same > > > Key: YARN-10315 > URL: https://issues.apache.org/jira/browse/YARN-10315 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin Chundatt >Assignee: Sushil Ks >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10315.001.patch, YARN-10315.002.patch > > > When the node is in DECOMMISSIONING state the RMNodeResourceUpdateEvent is > send for every heartbeat . Which will result in scheduler resource update. > Avoid sending the same. > Scheduler node resource update iterates through all the queues for resource > update which is costly.. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10353) Log vcores used and cumulative cpu in containers monitor
[ https://issues.apache.org/jira/browse/YARN-10353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17161450#comment-17161450 ] Hudson commented on YARN-10353: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18456 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18456/]) [YARN-10353] Log vcores used and cumulative cpu in containers monitor. (ebadger: rev 736bed6d6d20a17b522a0686ca3fd2d97e7e6838) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java > Log vcores used and cumulative cpu in containers monitor > > > Key: YARN-10353 > URL: https://issues.apache.org/jira/browse/YARN-10353 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.4.0 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10353.001.patch, YARN-10353.002.patch > > > We currently log the percentage/cpu and percentage/cpus-used-by-yarn in the > Containers Monitor log. It would be useful to also log vcores used vs vcores > assigned, and total accumulated CPU time. > For example, currently we have an audit log that looks like this: > {noformat} > 2020-07-16 20:33:51,550 DEBUG [Container Monitor] ContainersMonitorImpl.audit > (ContainersMonitorImpl.java:recordUsage(651)) - Resource usage of ProcessTree > 809 for container-id container_1594931466123_0002_01_07: 309.5 MB of 2 GB > physical memory used; 2.8 GB of 4.2 GB virtual memory used CPU:143.0905 > CPU/core:35.772625 > {noformat} > The proposal is to add two more fields to show vCores and Cumulative CPU ms: > {noformat} > 2020-07-16 20:33:51,550 DEBUG [Container Monitor] ContainersMonitorImpl.audit > (ContainersMonitorImpl.java:recordUsage(651)) - Resource usage of ProcessTree > 809 for container-id container_1594931466123_0002_01_07: 309.5 MB of 2 GB > physical memory used; 2.8 GB of 4.2 GB virtual memory used CPU:143.0905 > CPU/core:35.772625 vCores:2/1 CPU-ms:4180 > {noformat} > This is a snippet of a log from one of our clusters running branch-2.8 with a > similar change. > {noformat} > 2020-07-16 21:00:02,240 [Container Monitor] DEBUG > ContainersMonitorImpl.audit: Memory usage of ProcessTree 5267 for > container-id container_e04_1594079801456_1397450_01_001992: 1.6 GB of 2.5 GB > physical memory used; 3.8 GB of 5.3 GB virtual memory used. CPU usage: 18 of > 10 CPU vCores used. Cumulative CPU time: 157410 > 2020-07-16 21:00:02,269 [Container Monitor] DEBUG > ContainersMonitorImpl.audit: Memory usage of ProcessTree 18801 for > container-id container_e04_1594079801456_1390375_01_19: 413.2 MB of 2.5 > GB physical memory used; 3.8 GB of 5.3 GB virtual memory used. CPU usage: 0 > of 10 CPU vCores used. Cumulative CPU time: 113830 > 2020-07-16 21:00:02,298 [Container Monitor] DEBUG > ContainersMonitorImpl.audit: Memory usage of ProcessTree 5279 for > container-id container_e04_1594079801456_1397450_01_001991: 2.2 GB of 2.5 GB > physical memory used; 3.8 GB of 5.3 GB virtual memory used. CPU usage: 17 of > 10 CPU vCores used. Cumulative CPU time: 128630 > 2020-07-16 21:00:02,339 [Container Monitor] DEBUG > ContainersMonitorImpl.audit: Memory usage of ProcessTree 24189 for > container-id container_e04_1594079801456_1390430_01_000415: 392.7 MB of 2.5 > GB physical memory used; 3.8 GB of 5.3 GB virtual memory used. CPU usage: 0 > of 10 CPU vCores used. Cumulative CPU time: 96060 > 2020-07-16 21:00:02,367 [Container Monitor] DEBUG > ContainersMonitorImpl.audit: Memory usage of ProcessTree 6751 for > container-id container_e04_1594079801456_1397923_01_003248: 1.3 GB of 3 GB > physical memory used; 4.3 GB of 6.3 GB virtual memory used. CPU usage: 12 of > 10 CPU vCores used. Cumulative CPU time: 116820 > 2020-07-16 21:00:02,396 [Container Monitor] DEBUG > ContainersMonitorImpl.audit: Memory usage of ProcessTree 12138 for > container-id container_e04_1594079801456_1397760_01_44: 4.4 GB of 6 GB > physical memory used; 6.9 GB of 12.6 GB virtual memory used. CPU usage: 15 of > 10 CPU vCores used. Cumulative CPU time: 45900 > 2020-07-16 21:00:02,424 [Container Monitor] DEBUG > ContainersMonitorImpl.audit: Memory usage of ProcessTree 101918 for > container-id container_e04_1594079801456_1391130_01_002378: 2.4 GB of 4 GB > physical memory used; 5.8 GB of 8.4 GB virtual memory used. CPU usage: 13 of > 10 CPU vCores used. Cumulative CPU time: 2572390 > 2020-07-16 21:00:02,456 [Container Monitor] DEBUG > ContainersMonitorImpl.audit: Memory usage of ProcessTree 26596 for > container-id container_e04_1594079801456_1390446_01_000665: 418.6 MB of 2.5 > G
[jira] [Commented] (YARN-10339) Timeline Client in Nodemanager gets 403 errors when simple auth is used in kerberos environments
[ https://issues.apache.org/jira/browse/YARN-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17159398#comment-17159398 ] Hudson commented on YARN-10339: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18445 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18445/]) YARN-10339. Fix TimelineClient in NodeManager failing when Simple Http (pjoseph: rev cc71d50b219c1cc682b4185ea739b485e519501f) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineConnector.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/security/TestTimelineAuthenticationFilterForV1.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/timeline/security/TestTimelineAuthenticationFilterInitializer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/security/TestTimelineAuthFilterForV2.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/timeline/security/TimelineAuthenticationFilterInitializer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClientImpl.java > Timeline Client in Nodemanager gets 403 errors when simple auth is used in > kerberos environments > > > Key: YARN-10339 > URL: https://issues.apache.org/jira/browse/YARN-10339 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineclient >Affects Versions: 3.1.0 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10339.001.patch, YARN-10339.002.patch > > > We get below errors in NodeManager logs whenever we set > yarn.timeline-service.http-authentication.type=simple in a cluster which has > kerberos enabled. There are use cases where simple auth is used only in > timeline server for convenience although kerberos is enabled. > {code:java} > 2020-05-20 20:06:30,181 ERROR impl.TimelineV2ClientImpl > (TimelineV2ClientImpl.java:putObjects(321)) - Response from the timeline > server is not successful, HTTP error code: 403, Server response: > {"exception":"ForbiddenException","message":"java.lang.Exception: The owner > of the posted timeline entities is not > set","javaClassName":"org.apache.hadoop.yarn.webapp.ForbiddenException"} > {code} > This seems to affect the NM timeline publisher which uses > TimelineV2ClientImpl. Doing a simple auth directly to timeline service via > curl works fine. So this issue is in the authenticator configuration in > timeline client. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10350) TestUserGroupMappingPlacementRule fails
[ https://issues.apache.org/jira/browse/YARN-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17157910#comment-17157910 ] Hudson commented on YARN-10350: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18439 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18439/]) YARN-10350. TestUserGroupMappingPlacementRule fails (aajisaka: rev 8980c23fc2cdc3e30eb2066fc4a16dca4db819ec) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SimpleGroupsMapping.java > TestUserGroupMappingPlacementRule fails > --- > > Key: YARN-10350 > URL: https://issues.apache.org/jira/browse/YARN-10350 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Reporter: Akira Ajisaka >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-10350.001.patch, YARN-10350.002.patch > > > TestUserGroupMappingPlacementRule fails on trunk: > {noformat} > [INFO] Running > org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule > [ERROR] Tests run: 31, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: > 2.662 s <<< FAILURE! - in > org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule > [ERROR] > testResolvedQueueIsNotManaged(org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule) > Time elapsed: 0.03 s <<< ERROR! > java.lang.Exception: Unexpected exception, > expected but > was > at > org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:28) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) > at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) > Caused by: java.lang.AssertionError: Queue expected: but was: > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:118) > at > org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule.verifyQueueMapping(TestUserGroupMappingPlacementRule.java:236) > at > org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule.testResolvedQueueIsNotManaged(TestUserGroupMappingPlacementRule.java:516) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19) > ... 18 more > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubs
[jira] [Commented] (YARN-10348) Allow RM to always cancel tokens after app completes
[ https://issues.apache.org/jira/browse/YARN-10348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17157032#comment-17157032 ] Hudson commented on YARN-10348: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18431 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18431/]) YARN-10348. Allow RM to always cancel tokens after app completes. (ebadger: rev 48f90115b5ecb37f814af281f09bb404361b2bba) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java > Allow RM to always cancel tokens after app completes > > > Key: YARN-10348 > URL: https://issues.apache.org/jira/browse/YARN-10348 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.10.0, 3.1.3 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10348.001.patch, YARN-10348.002.patch > > > (Note: this change was originally done on our internal branch by [~daryn]). > The RM currently has an option for a client to specify disabling token > cancellation when a job completes. This feature was an initial attempt to > address the use case of a job launching sub-jobs (ie. oozie launcher) and the > original job finishing prior to the sub-job(s) completion - ex. original job > completion triggered premature cancellation of tokens needed by the sub-jobs. > Many years ago, [~daryn] added a more robust implementation to ref count > tokens ([YARN-3055]). This prevented premature cancellation of the token > until all apps using the token complete, and invalidated the need for a > client to specify cancel=false. Unfortunately the config option was not > removed. > We have seen cases where oozie "java actions" and some users were explicitly > disabling token cancellation. This can lead to a buildup of defunct tokens > that may overwhelm the ZK buffer used by the KDC's backing store. At which > point the KMS fails to connect to ZK and is unable to issue/validate new > tokens - rendering the KDC only able to authenticate pre-existing tokens. > Production incidents have occurred due to the buffer size issue. > To avoid these issues, the RM should have the option to ignore/override the > client's request to not cancel tokens. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10297) TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently
[ https://issues.apache.org/jira/browse/YARN-10297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17156926#comment-17156926 ] Hudson commented on YARN-10297: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18429 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18429/]) YARN-10297. (ericp: rev 0427100b7543d412f4fafe631b7ace289662d28c) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestContinuousScheduling.java > TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails > intermittently > --- > > Key: YARN-10297 > URL: https://issues.apache.org/jira/browse/YARN-10297 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jim Brennan >Priority: Major > Attachments: YARN-10297.001.patch, YARN-10297.002.patch, > YARN-10297.003.patch > > > After YARN-6492, testFairSchedulerContinuousSchedulingInitTime fails > intermittently when running {{mvn test -Dtest=TestContinuousScheduling}} > {noformat}[INFO] Running > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling > [ERROR] Tests run: 7, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.682 > s <<< FAILURE! - in > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling > [ERROR] > testFairSchedulerContinuousSchedulingInitTime(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling) > Time elapsed: 0.194 s <<< ERROR! > org.apache.hadoop.metrics2.MetricsException: Metrics source > PartitionQueueMetrics,partition= already exists! > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152) > at > org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125) > at > org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.getPartitionMetrics(QueueMetrics.java:362) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.incrPendingResources(QueueMetrics.java:601) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updatePendingResources(AppSchedulingInfo.java:388) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.internalAddResourceRequests(AppSchedulingInfo.java:320) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.internalAddResourceRequests(AppSchedulingInfo.java:347) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo.updateResourceRequests(AppSchedulingInfo.java:183) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.updateResourceRequests(SchedulerApplicationAttempt.java:456) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.allocate(FairScheduler.java:898) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling.testFairSchedulerContinuousSchedulingInitTime(TestContinuousScheduling.java:375) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10333) YarnClient obtain Delegation Token for Log Aggregation Path
[ https://issues.apache.org/jira/browse/YARN-10333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154290#comment-17154290 ] Hudson commented on YARN-10333: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18422 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18422/]) YARN-10333. YarnClient obtain Delegation Token for Log Aggregation Path. (sunilg: rev 5dd270e2085c8e8c3428287ed6f0c541a5548a31) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml > YarnClient obtain Delegation Token for Log Aggregation Path > --- > > Key: YARN-10333 > URL: https://issues.apache.org/jira/browse/YARN-10333 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10333-001.patch, YARN-10333-002.patch, > YARN-10333-003.patch > > > There are use cases where Yarn Log Aggregation Path is configured to a > FileSystem like S3 or ABFS different from what is configured in fs.defaultFS > (HDFS). Log Aggregation fails as the client has token only for fs.defaultFS > and not for log aggregation path. > This Jira is to improve YarnClient by obtaining delegation token for log > aggregation path and add it to the Credential of Container Launch Context > similar to how it does for Timeline Delegation Token. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10341) Yarn Service Container Completed event doesn't get processed
[ https://issues.apache.org/jira/browse/YARN-10341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154281#comment-17154281 ] Hudson commented on YARN-10341: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18421 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18421/]) YARN-10341. Yarn Service Container Completed event doesn't get (brahma: rev dfe60392c91be21f574c1659af22f5c381b2675a) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestServiceAM.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ServiceScheduler.java > Yarn Service Container Completed event doesn't get processed > - > > Key: YARN-10341 > URL: https://issues.apache.org/jira/browse/YARN-10341 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Critical > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10341.001.patch, YARN-10341.002.patch, > YARN-10341.003.patch, YARN-10341.004.patch > > > If there 10 workers running and if containers get killed , after a while we > see that there are just 9 workers runnning. This is due to CONTAINER > COMPLETED Event is not processed on AM side. > Issue is in below code: > {code:java} > public void onContainersCompleted(List statuses) { > for (ContainerStatus status : statuses) { > ContainerId containerId = status.getContainerId(); > ComponentInstance instance = > liveInstances.get(status.getContainerId()); > if (instance == null) { > LOG.warn( > "Container {} Completed. No component instance exists. > exitStatus={}. diagnostics={} ", > containerId, status.getExitStatus(), status.getDiagnostics()); > return; > } > ComponentEvent event = > new ComponentEvent(instance.getCompName(), CONTAINER_COMPLETED) > .setStatus(status).setInstance(instance) > .setContainerId(containerId); > dispatcher.getEventHandler().handle(event); > } > {code} > If component instance doesnt exist for a container, it doesnt iterate over > other containers as its returning from method -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10344) Sync netty versions in hadoop-yarn-csi
[ https://issues.apache.org/jira/browse/YARN-10344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17154190#comment-17154190 ] Hudson commented on YARN-10344: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18420 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18420/]) YARN-10344. Sync netty versions in hadoop-yarn-csi. (#2126) (github: rev 10d218934c9bc143bf8578c92cdbd6df6a4d3b98) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml > Sync netty versions in hadoop-yarn-csi > -- > > Key: YARN-10344 > URL: https://issues.apache.org/jira/browse/YARN-10344 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Affects Versions: 3.3.0 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Fix For: 3.4.0, 3.3.1 > > > netty-all is now 4.1.50.Final but the other netty libraries are 4.1.42.Final: > {noformat} > [INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ hadoop-yarn-csi > --- > [INFO] org.apache.hadoop:hadoop-yarn-csi:jar:3.3.0 > [INFO] +- com.google.guava:guava:jar:20.0:compile > [INFO] +- com.google.protobuf:protobuf-java:jar:3.6.1:compile > [INFO] +- io.netty:netty-all:jar:4.1.50.Final:compile > [INFO] +- io.grpc:grpc-core:jar:1.26.0:compile > [INFO] | +- io.grpc:grpc-api:jar:1.26.0:compile (version selected from > constraint [1.26.0,1.26.0]) > [INFO] | | +- io.grpc:grpc-context:jar:1.26.0:compile > [INFO] | | +- > com.google.errorprone:error_prone_annotations:jar:2.3.3:compile > [INFO] | | \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile > [INFO] | +- com.google.code.gson:gson:jar:2.2.4:compile > [INFO] | +- com.google.android:annotations:jar:4.1.1.4:compile > [INFO] | +- io.perfmark:perfmark-api:jar:0.19.0:compile > [INFO] | +- io.opencensus:opencensus-api:jar:0.24.0:compile > [INFO] | \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.24.0:compile > [INFO] +- io.grpc:grpc-protobuf:jar:1.26.0:compile > [INFO] | +- com.google.api.grpc:proto-google-common-protos:jar:1.12.0:compile > [INFO] | \- io.grpc:grpc-protobuf-lite:jar:1.26.0:compile > [INFO] +- io.grpc:grpc-stub:jar:1.26.0:compile > [INFO] +- io.grpc:grpc-netty:jar:1.26.0:compile > [INFO] | +- io.netty:netty-codec-http2:jar:4.1.42.Final:compile (version > selected from constraint [4.1.42.Final,4.1.42.Final]) > [INFO] | | +- io.netty:netty-common:jar:4.1.42.Final:compile > [INFO] | | +- io.netty:netty-buffer:jar:4.1.42.Final:compile > [INFO] | | +- io.netty:netty-transport:jar:4.1.42.Final:compile > [INFO] | | | \- io.netty:netty-resolver:jar:4.1.42.Final:compile > [INFO] | | +- io.netty:netty-codec:jar:4.1.42.Final:compile > [INFO] | | +- io.netty:netty-handler:jar:4.1.42.Final:compile > [INFO] | | \- io.netty:netty-codec-http:jar:4.1.42.Final:compile > [INFO] | \- io.netty:netty-handler-proxy:jar:4.1.42.Final:compile > [INFO] | \- io.netty:netty-codec-socks:jar:4.1.42.Final:compile > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8047) RMWebApp make external class pluggable
[ https://issues.apache.org/jira/browse/YARN-8047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17153315#comment-17153315 ] Hudson commented on YARN-8047: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18418 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18418/]) YARN-8047. RMWebApp make external class pluggable. (pjoseph: rev 3a4d05b850449c51a13f3a15fe0d756fdf50b4b2) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml > RMWebApp make external class pluggable > -- > > Key: YARN-8047 > URL: https://issues.apache.org/jira/browse/YARN-8047 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bibin Chundatt >Assignee: Bilwa S T >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-8047-001.patch, YARN-8047-002.patch, > YARN-8047-003.patch, YARN-8047.004.patch, YARN-8047.005.patch, > YARN-8047.006.patch > > > JIra should make sure we should be able to plugin webservices and web pages > of scheduler in Resourcemanager > * RMWebApp allow to bind external classes > * RMController allow to plugin scheduler classes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10337) TestRMHATimelineCollectors fails on hadoop trunk
[ https://issues.apache.org/jira/browse/YARN-10337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17152693#comment-17152693 ] Hudson commented on YARN-10337: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18416 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18416/]) YARN-10337. Fix failing testcase TestRMHATimelineCollectors. (pjoseph: rev 2bbd00dff498027241a5d84713f4e3f13ac45e65) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMHATimelineCollectors.java > TestRMHATimelineCollectors fails on hadoop trunk > > > Key: YARN-10337 > URL: https://issues.apache.org/jira/browse/YARN-10337 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test, yarn >Reporter: Ahmed Hussein >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10337.001.patch > > > {{TestRMHATimelineCollectors}} has been failing on trunk. I see it frequently > in the qbt reports and the yetus reprts > {code:bash} > [INFO] Running > org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.95 > s <<< FAILURE! - in > org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors > [ERROR] > testRebuildCollectorDataOnFailover(org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors) > Time elapsed: 5.615 s <<< ERROR! > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover(TestRMHATimelineCollectors.java:105) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:80) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) > at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) > [INFO] > [INFO] Results: > [INFO] > [ERROR] Errors: > [ERROR] TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover:105 > NullPointer > [INFO] > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0 > [INFO] > [ERROR] There are test failures. > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-iss
[jira] [Commented] (YARN-10330) Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule
[ https://issues.apache.org/jira/browse/YARN-10330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17149394#comment-17149394 ] Hudson commented on YARN-10330: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18402 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18402/]) YARN-10330. Add missing test scenarios to (snemeth: rev 04abd0eb17b58e321893e8651ec596e9f7ac786f) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SimpleGroupsMapping.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestAppNameMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java > Add missing test scenarios to TestUserGroupMappingPlacementRule and > TestAppNameMappingPlacementRule > --- > > Key: YARN-10330 > URL: https://issues.apache.org/jira/browse/YARN-10330 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, capacityscheduler, test >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10330-001.patch, YARN-10330-002.patch, > YARN-10330-003.patch, YARN-10330-004.patch, YARN-10330-branch-3.3-001.patch > > > After running {{TestUserGroupMappingPlacementRule}} with EclEmma, it turned > out that there are at least 8-10 missing test scenarios that are not covered. > Since we're planning to enhance mapping rule logic with extra features, it is > crucial to have good coverage so that we can verify backward compatibility. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10325) Document max-parallel-apps for Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17149364#comment-17149364 ] Hudson commented on YARN-10325: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18401 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18401/]) YARN-10325. Document max-parallel-apps for Capacity Scheduler. (snemeth: rev 9b5557a9e811f04b964aa3a31ba8846a907d26f9) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md > Document max-parallel-apps for Capacity Scheduler > - > > Key: YARN-10325 > URL: https://issues.apache.org/jira/browse/YARN-10325 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10325-001.patch, YARN-10325-branch-3.3.001.patch > > > New feature introduced by YARN-9930 should be reflected in the upstream > documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9809) NMs should supply a health status when registering with RM
[ https://issues.apache.org/jira/browse/YARN-9809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148914#comment-17148914 ] Hudson commented on YARN-9809: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18394 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18394/]) YARN-9809. Added node manager health status to resource manager (eyang: rev e8dc862d3856e9eaea124c625dade36f1dd53fe2) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/NodeManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/health/TimedHealthReporterService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/resourcetracker/TestNMReconnect.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestEventFlow.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/logaggregationstatus/TestRMAppLogAggregationStatus.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/proto/yarn_server_common_service_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/RegisterNodeManagerRequest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestNMProxy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerHealth.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/health/NodeHealthScriptRunner.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeStartedEvent.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesNodes.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/BaseContainerManagerTest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNM.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/mai
[jira] [Commented] (YARN-10331) Upgrade node.js to 10.21.0
[ https://issues.apache.org/jira/browse/YARN-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148417#comment-17148417 ] Hudson commented on YARN-10331: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18391 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18391/]) YARN-10331. Upgrade node.js to 10.21.0. (#2106) (github: rev cd188ea9f0e807df1e2cc13f62be3e4c956b1e69) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/pom.xml * (edit) dev-support/docker/Dockerfile * (edit) dev-support/docker/Dockerfile_aarch64 > Upgrade node.js to 10.21.0 > -- > > Key: YARN-10331 > URL: https://issues.apache.org/jira/browse/YARN-10331 > Project: Hadoop YARN > Issue Type: Bug > Components: build, yarn-ui-v2 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Fix For: 3.4.0, 3.3.1 > > > YARN-10036 upgraded Node.js to 8.17.0, but Node.js 8.x is already EoL. > https://nodejs.org/en/about/releases/ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9903) Support reservations continue looking for Node Labels
[ https://issues.apache.org/jira/browse/YARN-9903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17148084#comment-17148084 ] Hudson commented on YARN-9903: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18389 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18389/]) YARN-9903: Support reservations continue looking for Node Labels. (ericp: rev 74fc13cf91818a70f434401244f7560c4db3a676) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java > Support reservations continue looking for Node Labels > - > > Key: YARN-9903 > URL: https://issues.apache.org/jira/browse/YARN-9903 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Tarun Parimi >Assignee: Jim Brennan >Priority: Major > Attachments: YARN-9903-branch-3.2.001.patch, YARN-9903.001.patch, > YARN-9903.002.patch, YARN-9903.003.patch, YARN-9903.004.patch > > > YARN-1769 brought in reservations continue looking feature which improves the > several resource reservation scenarios. However, it is not handled currently > when nodes have a label assigned to them. This is useful and in many cases > necessary even for Node Labels. So we should look to support this for node > labels also. > For example, in AbstractCSQueue.java, we have the below TODO. > {code:java} > // TODO, now only consider reservation cases when the node has no label > if (this.reservationsContinueLooking && nodePartition.equals( > RMNodeLabelsManager.NO_LABEL) && Resources.greaterThan( resourceCalculator, > clusterResource, resourceCouldBeUnreserved, Resources.none())) { > {code} > cc [~sunilg] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10328) Too many ZK Curator NodeExists exception logs in YARN Service AM logs
[ https://issues.apache.org/jira/browse/YARN-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17147938#comment-17147938 ] Hudson commented on YARN-10328: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18388 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18388/]) YARN-10328. Fixed ZK Curator NodeExists exception in YARN service AM (eyang: rev 0be26811f3db49abb62d12e6a051a31553495da8) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/registry/YarnRegistryViewForProviders.java > Too many ZK Curator NodeExists exception logs in YARN Service AM logs > - > > Key: YARN-10328 > URL: https://issues.apache.org/jira/browse/YARN-10328 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10328.001.patch > > > Following debug logs are printed everytime when component is started. > {code:java} > [pool-6-thread-3] DEBUG zk.CuratorService - path already present: > /registry/users/server/services/yarn-service/default-worker/components > org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = > NodeExists for > /registry/users/hetuserver/services/yarn-service/default-worker/components > at org.apache.zookeeper.KeeperException.create(KeeperException.java:128) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1480) > at > org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740) > at > org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723) > at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109) > at > org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720) > at > org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484) > at > org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474) > at > org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454) > at > org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44) > at > org.apache.hadoop.registry.client.impl.zk.CuratorService.zkMkPath(CuratorService.java:587) > at > org.apache.hadoop.registry.client.impl.zk.RegistryOperationsService.mknode(RegistryOperationsService.java:99) > at > org.apache.hadoop.yarn.service.registry.YarnRegistryViewForProviders.putComponent(YarnRegistryViewForProviders.java:146) > at > org.apache.hadoop.yarn.service.registry.YarnRegistryViewForProviders.putComponent(YarnRegistryViewForProviders.java:128) > at > org.apache.hadoop.yarn.service.component.instance.ComponentInstance.updateServiceRecord(ComponentInstance.java:511) > at > org.apache.hadoop.yarn.service.component.instance.ComponentInstance.updateContainerStatus(ComponentInstance.java:449) > at > org.apache.hadoop.yarn.service.component.instance.ComponentInstance$ContainerStatusRetriever.run(ComponentInstance.java:620) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6526) Refactoring SQLFederationStateStore by avoiding to recreate a connection at every call
[ https://issues.apache.org/jira/browse/YARN-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146407#comment-17146407 ] Hudson commented on YARN-6526: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18384 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18384/]) YARN-6526. Refactoring SQLFederationStateStore by avoiding to recreate a (brahma: rev 2c03524fa4be754aa95889d4ac0f5d57dca8cda8) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/utils/FederationStateStoreUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/SQLFederationStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/TestSQLFederationStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/TestZookeeperFederationStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/metrics/FederationStateStoreClientMetrics.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/FederationStateStoreBaseTest.java > Refactoring SQLFederationStateStore by avoiding to recreate a connection at > every call > -- > > Key: YARN-6526 > URL: https://issues.apache.org/jira/browse/YARN-6526 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Giovanni Matteo Fumarola >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-6526.001.patch, YARN-6526.002.patch, > YARN-6526.003.patch, YARN-6526.004.patch, YARN-6526.005.patch, > YARN-6526.006.patch, YARN-6526.007.patch, YARN-6526.008.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10327) Remove duplication of checking for invalid application ID in TestLogsCLI
[ https://issues.apache.org/jira/browse/YARN-10327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146251#comment-17146251 ] Hudson commented on YARN-10327: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18383 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18383/]) YARN-10327. Remove duplication of checking for invalid application ID in (snemeth: rev e0c1d8a96905bfbedbddde9000fc08ce2af1d277) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java > Remove duplication of checking for invalid application ID in TestLogsCLI > > > Key: YARN-10327 > URL: https://issues.apache.org/jira/browse/YARN-10327 > Project: Hadoop YARN > Issue Type: Test >Reporter: Hudáky Márton Gyula >Assignee: Hudáky Márton Gyula >Priority: Trivial > Fix For: 3.4.0 > > Attachments: YARN-10327.001.patch > > > TestLogsCLI has a separate function to test for invalid application ID > (#testInvalidApplicationId) and another (#testLogsCLIWithInvalidArgs) to test > multiple invalid arguments (including application ID). One of them should be > eliminated. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10318) ApplicationHistory Web UI incorrect column indexing
[ https://issues.apache.org/jira/browse/YARN-10318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146240#comment-17146240 ] Hudson commented on YARN-10318: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18382 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18382/]) YARN-10318. ApplicationHistory Web UI incorrect column indexing. (snemeth: rev 6459cc73e6c0878be51c0f2f70ec9868325dec34) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppsBlock.java > ApplicationHistory Web UI incorrect column indexing > --- > > Key: YARN-10318 > URL: https://issues.apache.org/jira/browse/YARN-10318 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Minor > Fix For: 3.4.0 > > Attachments: Screenshot 2020-06-25 at 10.15.32.png, > YARN-10318.001.patch, image-2020-06-16-17-14-55-921.png > > > The ApplicationHistory UI is broken due to an incorrect column indexing. This > bug was probably introduced in YARN-10038, which presumes, that the table > contains the application tag column (which is true for RM Web UI, but not for > AH Web UI). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10277) CapacityScheduler test TestUserGroupMappingPlacementRule should build proper hierarchy
[ https://issues.apache.org/jira/browse/YARN-10277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146157#comment-17146157 ] Hudson commented on YARN-10277: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18381 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18381/]) YARN-10277. CapacityScheduler test TestUserGroupMappingPlacementRule (snemeth: rev 33d3df7ee2b3942560e9affe1409854ac3e8bb96) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java > CapacityScheduler test TestUserGroupMappingPlacementRule should build proper > hierarchy > -- > > Key: YARN-10277 > URL: https://issues.apache.org/jira/browse/YARN-10277 > Project: Hadoop YARN > Issue Type: Task >Reporter: Gergely Pollak >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10277.001.patch, YARN-10277.002.patch, > YARN-10277.003.patch, YARN-10277.branch-3.3.001.patch > > > Since the CapacityScheduler internal implementation depends more and more on > queue being hierarchical, the test gets really hard to maintain. A lot of > test cases were failing because they used non existing queues, but the older > placement rule solution ignored missing parents, but since the leaf queue > change in CS, we must be able to get a full path for any queue, since all > queues are referenced by their full path. > This test should reflect this and instead of creating and expecting the > existance of fictional queues, it should create a proper queue hierarchy, > with a way to describe it better. > Currently we set up a bunch of mockito "when" statements to simulate the > queue behavior, but this is a hassle to maintain, and easy to miss a few > method. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10279) Avoid unnecessary QueueMappingEntity creations
[ https://issues.apache.org/jira/browse/YARN-10279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17145042#comment-17145042 ] Hudson commented on YARN-10279: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18379 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18379/]) YARN-10279. Avoid unnecessary QueueMappingEntity creations. Contributed (snemeth: rev 6a8fd73b273629d0c7c071cf4d090f67d9b96fe4) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/QueuePlacementRuleUtils.java > Avoid unnecessary QueueMappingEntity creations > -- > > Key: YARN-10279 > URL: https://issues.apache.org/jira/browse/YARN-10279 > Project: Hadoop YARN > Issue Type: Task >Reporter: Gergely Pollak >Assignee: Hudáky Márton Gyula >Priority: Minor > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10279.001.patch, YARN-10279.003.patch, > YARN-10279.004.patch, YARN-10279.005.patch, YARN-10279.006.patch > > > In CS UserGroupMappingPlacementRule and AppNameMappingPlacementRule classes > we create new instances of QueueMappingEntity class. In some cases we simply > copy the already received class, so we just duplicate it, which is > unnecessary since the class is immutable. > This is just a minor improvement, probably doesn't have much impact, but > still puts some unnecessary load on GC. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10316) FS-CS converter: convert maxAppsDefault, maxRunningApps settings
[ https://issues.apache.org/jira/browse/YARN-10316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17142824#comment-17142824 ] Hudson commented on YARN-10316: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18376 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18376/]) YARN-10316. FS-CS converter: convert maxAppsDefault, maxRunningApps (snemeth: rev 03f855e3e7a4505362e221c8a07ae9317af773d0) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigRuleHandler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigRuleHandler.java > FS-CS converter: convert maxAppsDefault, maxRunningApps settings > > > Key: YARN-10316 > URL: https://issues.apache.org/jira/browse/YARN-10316 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Labels: fs2cs > Fix For: 3.4.0 > > Attachments: YARN-10136-001.patch, YARN-10316-002.patch, > YARN-10316-003.patch, YARN-10316-branch-3.3.001.patch > > > In YARN-9930, support for maximum running applications (called "max parallel > apps") has been introduced. > The converter now can handle the following settings in {{fair-scheduler.xml}}: > * {{}} per user > * {{}} per queue > * {{}} > * {{}} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10321) Break down TestUserGroupMappingPlacementRule#testMapping into test scenarios
[ https://issues.apache.org/jira/browse/YARN-10321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17141769#comment-17141769 ] Hudson commented on YARN-10321: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18372 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18372/]) YARN-10321. Break down TestUserGroupMappingPlacementRule#testMapping (pjoseph: rev ce1008fe61a70d23845c05fad0380ae4906e74f5) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java > Break down TestUserGroupMappingPlacementRule#testMapping into test scenarios > > > Key: YARN-10321 > URL: https://issues.apache.org/jira/browse/YARN-10321 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10321.001.patch > > > org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule#testMapping > is very large and hard to read/maintain and moreover, error-prone. > We should break this testcase down into several separate testcases. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9460) QueueACLsManager and ReservationsACLManager should not use instanceof checks
[ https://issues.apache.org/jira/browse/YARN-9460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17141103#comment-17141103 ] Hudson commented on YARN-9460: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18370 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18370/]) YARN-9460. QueueACLsManager and ReservationsACLManager should not use (surendralilhore: rev b2facc84a1b48b9dcbe0816e120778d2100b320e) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/CapacityReservationsACLsManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/GenericQueueACLsManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/QueueACLsManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/AbstractReservationSystem.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/package-info.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/ReservationsACLsManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/FairReservationsACLsManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/CapacityQueueACLsManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/FairQueueACLsManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMTokens.java > QueueACLsManager and ReservationsACLManager should not use instanceof checks > > > Key: YARN-9460 > URL: https://issues.apache.org/jira/browse/YARN-9460 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Szilard Nemeth >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-9460.001.patch, YARN-9460.002.patch, > YARN-9460.003.patch, YARN-9460.004.patch, YARN-9460.005.patch > > > QueueACLsManager and ReservationsACLManager should not use instanceof checks > for the scheduler type. > Rather, we should abstract this into two classes: Capacity and Fair variants > of these ACL classes. > QueueACLsManager and ReservationsACLManager could be abstract classes, but > the implementation is the decision of one who will work on this jira. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9930) Support max running app logic for CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-9930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17140526#comment-17140526 ] Hudson commented on YARN-9930: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18365 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18365/]) YARN-9930. Support max running app logic for CapacityScheduler. (snemeth: rev 469841446f921f3da5bbd96cf83b3a808dde8084) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueStateManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSMaxRunningAppsEnforcer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCSMaxRunningAppsEnforcer.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerMaxParallelApps.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/TestReservationSystem.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueState.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java > Support max running app logic for CapacityScheduler > --- > > Key: YARN-9930 > URL: https://issues.apache.org/jira/browse/YARN-9930 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.1.0, 3.1.1 >Reporter: zhoukang >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-9930-001.patch, YARN-9930-002.patch, > YARN-9930-003.patch, YARN-9930-004.patch, YARN-9930-005.patch, > YARN-9930-006.patch, YARN-9930-POC01.patch, YARN-9930-POC02.patch, > YARN-9930-POC03.patch, YARN-9930-POC04.patch, YARN-9930-POC05.patch, > screenshot-1.png > > > In FairScheduler, there has limitation for max running which will let > application pending. > But in CapacityScheduler there has no feature like max running app.Only got > max app,and jobs will be rejected directly on client. > This jira i want to implement this semantic for CapacityScheduler. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10308) Update javadoc and variable names for keytab in yarn services as it supports filesystems other than hdfs and local file system
[ https://issues.apache.org/jira/browse/YARN-10308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138619#comment-17138619 ] Hudson commented on YARN-10308: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18359 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18359/]) YARN-10308. Update javadoc and variable names for YARN service. (eyang: rev 89689c52c39cdcc498d04508dbd235c6036ec17c) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/KerberosPrincipal.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/client/ServiceClient.java > Update javadoc and variable names for keytab in yarn services as it supports > filesystems other than hdfs and local file system > -- > > Key: YARN-10308 > URL: https://issues.apache.org/jira/browse/YARN-10308 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10308.001.patch, YARN-10308.002.patch > > > 1. Below description should be updated > {code:java} > @ApiModelProperty(value = "The URI of the kerberos keytab. It supports two " + > "schemes \"hdfs\" and \"file\". If the URI starts with \"hdfs://\" " + > "scheme, it indicates the path on hdfs where the keytab is stored. The > " + > "keytab will be localized by YARN and made available to AM in its > local" + > " directory. If the URI starts with \"file://\" scheme, it indicates a > " + > "path on the local host where the keytab is presumbaly installed by " + > "admins upfront. ") > public String getKeytab() { > return keytab; > } > {code} > 2. Variables below are still named on hdfs which is confusing > {code:java} > if ("file".equals(keytabURI.getScheme())) { > LOG.info("Using a keytab from localhost: " + keytabURI); > } else { > Path keytabOnhdfs = new Path(keytabURI); > if (!fileSystem.getFileSystem().exists(keytabOnhdfs)) { > LOG.warn(service.getName() + "'s keytab (principalName = " > + principalName + ") doesn't exist at: " + keytabOnhdfs); > return; > } > LocalResource keytabRes = fileSystem.createAmResource(keytabOnhdfs, > LocalResourceType.FILE); > localResource.put(String.format(YarnServiceConstants.KEYTAB_LOCATION, > service.getName()), keytabRes); > LOG.info("Adding " + service.getName() + "'s keytab for " > + "localization, uri = " + keytabOnhdfs); > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10281) Redundant QueuePath usage in UserGroupMappingPlacementRule and AppNameMappingPlacementRule
[ https://issues.apache.org/jira/browse/YARN-10281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138412#comment-17138412 ] Hudson commented on YARN-10281: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18357 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18357/]) YARN-10281. Redundant QueuePath usage in UserGroupMappingPlacementRule (snemeth: rev 5b1a56f9f1aec7d75b14a60d0c42192b04407356) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/AppNameMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/QueuePlacementRuleUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/QueueMapping.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/QueuePath.java > Redundant QueuePath usage in UserGroupMappingPlacementRule and > AppNameMappingPlacementRule > -- > > Key: YARN-10281 > URL: https://issues.apache.org/jira/browse/YARN-10281 > Project: Hadoop YARN > Issue Type: Task >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10281.001.patch, YARN-10281.002.patch, > YARN-10281.003.patch, YARN-10281.004.patch, YARN-10281.branch-3.3.001.patch > > > We use the QueuePath and QueueMapping (or QueueMappingEntity) objects in the > aforementioned classes, but these technically store the same kind of > information, yet we keep converting between them, let's examine if we can use > only the QueueMapping(Entity) instead, since that holds more information. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10314) YarnClient throws NoClassDefFoundError for WebSocketException with only shaded client jars
[ https://issues.apache.org/jira/browse/YARN-10314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138090#comment-17138090 ] Hudson commented on YARN-10314: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18355 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18355/]) YARN-10314. YarnClient throws NoClassDefFoundError for (github: rev fc4ebb0499fe1095b87ff782c265e9afce154266) * (edit) hadoop-client-modules/hadoop-client-minicluster/pom.xml * (edit) hadoop-client-modules/hadoop-client-runtime/pom.xml > YarnClient throws NoClassDefFoundError for WebSocketException with only > shaded client jars > -- > > Key: YARN-10314 > URL: https://issues.apache.org/jira/browse/YARN-10314 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 3.3.0 >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Blocker > Fix For: 3.3.0 > > > After YARN-8778, with only shaded hadoop client jars in classpath Unable to > submit job. > CC: [~ayushtkn] confirmed the same. Hive 4.0 doesnot work due to this, shaded > client is necessary there to avoid guava jar's conflicts. > {noformat} > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/shaded/org/eclipse/jetty/websocket/api/WebSocketException > at > org.apache.hadoop.yarn.client.api.YarnClient.createYarnClient(YarnClient.java:92) > at > org.apache.hadoop.mapred.ResourceMgrDelegate.(ResourceMgrDelegate.java:109) > at org.apache.hadoop.mapred.YARNRunner.(YARNRunner.java:153) > at > org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:130) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:109) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:102) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1545) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1541) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845) > at org.apache.hadoop.mapreduce.Job.connect(Job.java:1541) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1570) > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1594) > at org.apache.hadoop.examples.WordCount.main(WordCount.java:87) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.shaded.org.eclipse.jetty.websocket.api.WebSocketException > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:418) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) > at java.lang.ClassLoader.loadClass(ClassLoader.java:351) > ... 16 more > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10293) Reserved Containers not allocated from available space of other nodes in CandidateNodeSet in MultiNodePlacement (YARN-10259)
[ https://issues.apache.org/jira/browse/YARN-10293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17134262#comment-17134262 ] Hudson commented on YARN-10293: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18347 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18347/]) YARN-10293. Reserved Containers not allocated from available space of (taoyang1: rev 7c4de59fc10953170bbef9a320ce70bcddae8bba) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerMultiNodes.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerMultiNodesWithPreemption.java > Reserved Containers not allocated from available space of other nodes in > CandidateNodeSet in MultiNodePlacement (YARN-10259) > > > Key: YARN-10293 > URL: https://issues.apache.org/jira/browse/YARN-10293 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10293-001.patch, YARN-10293-002.patch, > YARN-10293-003-WIP.patch, YARN-10293-004.patch, YARN-10293-005.patch > > > Reserved Containers not allocated from available space of other nodes in > CandidateNodeSet in MultiNodePlacement. YARN-10259 has fixed two issues > related to it > https://issues.apache.org/jira/browse/YARN-10259?focusedCommentId=17105987&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17105987 > Have found one more bug in the CapacityScheduler.java code which causes the > same issue with slight difference in the repro. > *Repro:* > *Nodes : Available : Used* > Node1 - 8GB, 8vcores - 8GB. 8cores > Node2 - 8GB, 8vcores - 8GB. 8cores > Node3 - 8GB, 8vcores - 8GB. 8cores > Queues -> A and B both 50% capacity, 100% max capacity > MultiNode enabled + Preemption enabled > 1. JobA submitted to A queue and which used full cluster 24GB and 24 vcores > 2. JobB Submitted to B queue with AM size of 1GB > {code} > 2020-05-21 12:12:27,313 INFO > org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=systest > IP=172.27.160.139 OPERATION=Submit Application Request > TARGET=ClientRMService RESULT=SUCCESS APPID=application_1590046667304_0005 > CALLERCONTEXT=CLI QUEUENAME=dummy > {code} > 3. Preemption happens and used capacity is lesser than 1.0f > {code} > 2020-05-21 12:12:48,222 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptMetrics: > Non-AM container preempted, current > appAttemptId=appattempt_1590046667304_0004_01, > containerId=container_e09_1590046667304_0004_01_24, > resource= > {code} > 4. JobB gets a Reserved Container as part of > CapacityScheduler#allocateOrReserveNewContainer > {code} > 2020-05-21 12:12:48,226 INFO > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: > container_e09_1590046667304_0005_01_01 Container Transitioned from NEW to > RESERVED > 2020-05-21 12:12:48,226 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp: > Reserved container=container_e09_1590046667304_0005_01_01, on node=host: > tajmera-fullnodes-3.tajmera-fullnodes.root.hwx.site:8041 #containers=8 > available= used= with > resource= > {code} > *Why RegularContainerAllocator reserved the container when the used capacity > is <= 1.0f ?* > {code} > The reason is even though the container is preempted - nodemanager has to > stop the container and heartbeat and update the available and unallocated > resources to ResourceManager. > {code} > 5. Now, no new allocation happens and reserved container stays at reserved. > After reservation the used capacity becomes 1.0f, below will be in a loop and > no new allocate or reserve happens. The reserved container cannot be > allocated as reserved node does not have space. node2 has space for 1GB, > 1vcore but CapacityScheduler#allocateOrReserveNewContainers not getting > called causing the Hang. > *[INFINITE LOOP] CapacityScheduler#allocateContainersOnMultiNodes -> > CapacityScheduler#allocateFromReservedContainer -> Re-reserve the container > on node* > {code} > 2020-05-21 12:13:33,242 INFO > org.apache.hadoop.yarn.server.resourcemanager.s
[jira] [Commented] (YARN-10312) Add support for yarn logs -logFile to retain backward compatibility
[ https://issues.apache.org/jira/browse/YARN-10312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133701#comment-17133701 ] Hudson commented on YARN-10312: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18346 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18346/]) YARN-10312. Add support for yarn logs -logFile to retain backward (ebadger: rev fed6fecd3a9e24efc20f9221505da35a7e1949c7) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java > Add support for yarn logs -logFile to retain backward compatibility > --- > > Key: YARN-10312 > URL: https://issues.apache.org/jira/browse/YARN-10312 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.10.0, 3.4.1 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Labels: compatibility > Attachments: YARN-10312.001.patch > > > The YARN CLI logs command line option {{-logFiles}} was changed to > {{-log_files}} in 2.9 and later releases. This change was made as part of > YARN-5363. > Verizon Media is in the process of moving from Hadoop-2.8 to Hadoop-2.10, and > while testing integration with Spark, we ran into this issue. We are > concerned that we will run into more cases of this as we roll out to > production, and rather than break user scripts, we'd prefer to add > {{-logFiles}} as an alias of {{-log_files}}. If both are provided, > {{-logFiles}} will be ignored. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10300) appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat
[ https://issues.apache.org/jira/browse/YARN-10300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17129725#comment-17129725 ] Hudson commented on YARN-10300: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18342 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18342/]) YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails (ericp: rev 56247db3022705635580c4d2f8b0abde109f954f) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java > appMasterHost not set in RM ApplicationSummary when AM fails before first > heartbeat > --- > > Key: YARN-10300 > URL: https://issues.apache.org/jira/browse/YARN-10300 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Attachments: YARN-10300.001.patch, YARN-10300.002.patch, > YARN-10300.003.patch > > > {noformat} > 2020-05-23 14:09:10,086 INFO resourcemanager.RMAppManager$ApplicationSummary: > appId=application_1586003420099_12444961,name=job_name,user=username,queue=queuename,state=FAILED,trackingUrl=https > > ://cluster:port/applicationhistory/app/application_1586003420099_12444961,appMasterHost=N/A,startTime=1590241207309,finishTime=1590242950085,finalStatus=FAILED,memorySeconds=13750,vcoreSeconds=67,preemptedMemorySeconds=0,preemptedVcoreSeconds=0,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources= vCores:0>,applicationType=MAPREDUCE > {noformat} > {{appMasterHost=N/A}} should have the AM hostname instead of N/A -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10292) FS-CS converter: add an option to enable asynchronous scheduling in CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-10292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126761#comment-17126761 ] Hudson commented on YARN-10292: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18330 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18330/]) YARN-10292. FS-CS converter: add an option to enable asynchronous (snemeth: rev 8abff5151a7c157355930475dbea2dc535d51c19) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSYarnSiteConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/ConversionOptions.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSYarnSiteConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigArgumentHandler.java > FS-CS converter: add an option to enable asynchronous scheduling in > CapacityScheduler > - > > Key: YARN-10292 > URL: https://issues.apache.org/jira/browse/YARN-10292 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10292.001.patch, YARN-10292.002.patch > > > FS doesn't have an equivalent setting to the CapacityScheduler's > yarn.scheduler.capacity.schedule-asynchronously.enable option so the FS to CS > converter won't add this to the yarn-site.xml. An optional command line > switch should be added to support this option during migration. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10274) Merge QueueMapping and QueueMappingEntity
[ https://issues.apache.org/jira/browse/YARN-10274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126601#comment-17126601 ] Hudson commented on YARN-10274: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18328 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18328/]) YARN-10274. Merge QueueMapping and QueueMappingEntity. Contributed by (snemeth: rev 8b146c17b35532af7b9685aa1896d2b4f4fadc05) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/AppNameMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestPlacementManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestAppNameMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/QueueMapping.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerQueueMappingFactory.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/QueueMappingEntity.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/QueuePlacementRuleUtils.java > Merge QueueMapping and QueueMappingEntity > - > > Key: YARN-10274 > URL: https://issues.apache.org/jira/browse/YARN-10274 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10274.001.patch, YARN-10274.002.patch, > YARN-10274.003.patch > > > The role, usage and internal behaviour of these classes are almost identical, > but it makes no sense to keep both of them. One is used by UserGroup > placement rule definitions the other is used by Application placement rules. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10296) Make ContainerPBImpl#getId/setId synchronized
[ https://issues.apache.org/jira/browse/YARN-10296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17126547#comment-17126547 ] Hudson commented on YARN-10296: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18327 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18327/]) YARN-10296. Make ContainerPBImpl#getId/setId synchronized. Contributed (snemeth: rev a4835db95a2cca255bf0e40fa5d016526ef03857) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerPBImpl.java > Make ContainerPBImpl#getId/setId synchronized > - > > Key: YARN-10296 > URL: https://issues.apache.org/jira/browse/YARN-10296 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10296.001.patch, YARN-10296.002.patch > > > ContainerPBImpl getId and setId methods can be accessed from multiple > threads. In order to avoid any simultaneous accesses and race conditions > these methods should be synchronized. > The idea came from the issue described in YARN-10295, however that patch is > only applicable to branch-3.2 and 3.1. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10254) CapacityScheduler incorrect User Group Mapping after leaf queue change
[ https://issues.apache.org/jira/browse/YARN-10254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17124044#comment-17124044 ] Hudson commented on YARN-10254: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18315 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18315/]) YARN-10254. CapacityScheduler incorrect User Group Mapping after leaf (snemeth: rev b5efdea4fd385453fd9f9da7106e908d2a9a3812) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerQueueMappingFactory.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java > CapacityScheduler incorrect User Group Mapping after leaf queue change > -- > > Key: YARN-10254 > URL: https://issues.apache.org/jira/browse/YARN-10254 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10254.001.patch, YARN-10254.002.patch, > YARN-10254.003.patch, YARN-10254.004.patch, YARN-10254.005.patch, > YARN-10254.branch-3.3.001.patch > > > YARN-9879 and YARN-10198 introduced some major changes to user group mapping, > and some of them unfortunately had some negative impact on the way mapping > works. > In some cases incorrect PlacementContexts were created, where full queue path > was passed as leaf queue name. This affects how the yarn cli app list > displays the queues. > u:%user:%primary_group.%user mapping fails with an incorrect validation error > when the %primary_group parent queue was a managed parent. > Group based rules in certain cases are mapped to root.[primary_group] rules, > loosing the ability to create deeper structures. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10284) Add lazy initialization of LogAggregationFileControllerFactory in LogServlet
[ https://issues.apache.org/jira/browse/YARN-10284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17123851#comment-17123851 ] Hudson commented on YARN-10284: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18314 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18314/]) YARN-10284. Add lazy initialization of (snemeth: rev aa6d13455b9435fb6c6d8f942c2b278dfada8f0c) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/webapp/TestLogServlet.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogServlet.java > Add lazy initialization of LogAggregationFileControllerFactory in LogServlet > > > Key: YARN-10284 > URL: https://issues.apache.org/jira/browse/YARN-10284 > Project: Hadoop YARN > Issue Type: Sub-task > Components: log-aggregation, yarn >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10284.001.patch, YARN-10284.002.patch, > YARN-10284.003.patch, YARN-10284.004.patch, YARN-10284.branch-3.3.001.patch > > > Suppose the {{mapred}} user has no access to the remote folder. Pinging the > JHS if it's online in every few seconds will produce the following entry in > the log: > {noformat} > 2020-05-19 00:17:20,331 WARN > org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController: > Unable to determine if the filesystem supports append operation > java.nio.file.AccessDeniedException: test-bucket: > org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: There is no mapped role > for the group(s) associated with the authenticated user. (user: mapred) > at > org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:204) > [...] > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:513) > at > org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController.getRollOverLogMaxSize(LogAggregationIndexedFileController.java:1157) > at > org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController.initInternal(LogAggregationIndexedFileController.java:149) > at > org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileController.initialize(LogAggregationFileController.java:135) > at > org.apache.hadoop.yarn.logaggregation.filecontroller.LogAggregationFileControllerFactory.(LogAggregationFileControllerFactory.java:139) > at > org.apache.hadoop.yarn.server.webapp.LogServlet.(LogServlet.java:66) > at > org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.(HsWebServices.java:99) > at > org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices$$FastClassByGuice$$1eb8d5d6.newInstance() > at > com.google.inject.internal.cglib.reflect.$FastConstructor.newInstance(FastConstructor.java:40) > [...] > at > org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) > at java.lang.Thread.run(Thread.java:748) > {noformat} > We should only create the {{LogAggregationFactory}} instance when we actually > need it, not every time the {{LogServlet}} object is instantiated (so > definitely not in the constructor). In this way we prevent pressure on the > S3A auth side, especially if the authentication request is a costly operation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10286) PendingContainers bugs in the scheduler outputs
[ https://issues.apache.org/jira/browse/YARN-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17123827#comment-17123827 ] Hudson commented on YARN-10286: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18313 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18313/]) YARN-10286. PendingContainers bugs in the scheduler outputs. Contributed (snemeth: rev e0a0741ac86dcc4f98ec2f9739b70b3697a4d0c0) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerQueueInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRest.md > PendingContainers bugs in the scheduler outputs > --- > > Key: YARN-10286 > URL: https://issues.apache.org/jira/browse/YARN-10286 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Andras Gyori >Priority: Critical > Fix For: 3.4.0 > > Attachments: YARN-10286.001.patch > > > There are some problems with the {{ws/v1/cluster/scheduler}} output of the > ResourceManager: > Even though the pendingContainers field is > [documented|https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API] > in the FairScheduler output, it is > [missing|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerQueueInfo.java] > from the actual output. In case of the CapacityScheduler this field > [exists|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java], > but it is missing from the documentation. Let's fix both cases! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10108) FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping
[ https://issues.apache.org/jira/browse/YARN-10108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117685#comment-17117685 ] Hudson commented on YARN-10108: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18298 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18298/]) YARN-10108. FS-CS converter: nestedUserQueue with default rule results (snemeth: rev 593af878c006953a2df5da504e59aa65505db438) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoQueueCreation.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java > FS-CS converter: nestedUserQueue with default rule results in invalid queue > mapping > --- > > Key: YARN-10108 > URL: https://issues.apache.org/jira/browse/YARN-10108 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Gergely Pollak >Priority: Major > Labels: fs2cs > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10108.001.patch, YARN-10108.002.patch, > YARN-10108.003.patch, YARN-10108.004.patch, YARN-10108.branch-3.3.001.patch, > YARN-10108.branch-3.3.002.patch > > > FS Queue Placement Policy > {code:java} > > > > > > {code} > gets mapped to an invalid CS queue mapping "u:%user:root.users.%user" > RM fails to start with above queue mapping in CS > {code:java} > 2020-01-28 00:19:12,889 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting > ResourceManager > org.apache.hadoop.service.ServiceStateException: java.io.IOException: mapping > contains invalid or non-leaf queue [%user] and invalid parent queue > [root.users] > at > org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:173) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:829) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1247) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:324) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1534) > Caused by: java.io.IOException: mapping contains invalid or non-leaf queue > [%user] and invalid parent queue [root.users] > at > org.apache.hadoop.yarn.server.resourcemanager.placement.QueuePlacementRuleUtils.validateQueueMappingUnderParentQueue(QueuePlacementRuleUtils.java:48) > at > org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.validateAndGetAutoCreatedQueueMapping(UserGroupMappingPlacementRule.java:363) > at > org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.initialize(UserGroupMappingPlacementRule.java:300) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getUserGroupMappingPlacementRule(CapacityScheduler.java:671) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.updatePlacementRules(CapacityScheduler.java:712) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:753) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:361) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:426) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) > ... 7 more > {code} > QueuePlacementConverter#handleNestedRule has to be fixed. > {code:java} > else if (pr instanceof DefaultPlacementRule) { > DefaultPlacementRule defaultRule = (DefaultPlacementRule) pr; > mapping.append("u:" + USER + ":") > .append(defaultRule.defaultQueueName) > .append("." + USER); > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) -
[jira] [Commented] (YARN-6492) Generate queue metrics for each partition
[ https://issues.apache.org/jira/browse/YARN-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117179#comment-17117179 ] Hudson commented on YARN-6492: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18297 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18297/]) YARN-6492. Generate queue metrics for each partition. Contributed by (jhung: rev c30c23cb665761e997bcfc1dc00908f70b069fa2) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/PartitionQueueMetrics.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoCreatedQueueBase.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestPartitionQueueMetrics.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueMetrics.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestNodeLabelContainerAllocation.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoAppAttempt.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ContainerUpdateContext.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoQueueCreation.java > Generate queue metrics for each partition > - > > Key: YARN-6492 > URL: https://issues.apache.org/jira/browse/YARN-6492 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Reporter: Jonathan Hung >Assignee: Manikandan R >Priority: Major > Fix For: 3.4.0 > > Attachments: PartitionQueueMetrics_default_partition.txt, > PartitionQueueMetrics_x_partition.txt, PartitionQueueMetrics_y_partition.txt, > YARN-6492-junits.patch, YARN-6492.001.patch, YARN-6492.002.patch, > YARN-6492.003.patch, YARN-6492.004.patch, YARN-6492.005.WIP.patch, > YARN-6492.006.WIP.patch, YARN-6492.007.WIP.patch, YARN-6492.008.WIP.patch, > YARN-6492.009.WIP.patch, YARN-6492.010.WIP.patch, YARN-6492.011.WIP.patch, > YARN-6492.012.WIP.patch, YARN-6492.013.patch, partition_metrics.txt > > > We are interested in having queue metrics for all partitions. Right now each > queue has one QueueMetrics object which captures metrics either in default > partition or across all partitions. (After YARN-6467 it will be in default > partition) > But having the partition metrics would be very useful. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10228) Yarn Service fails if am java opts contains ZK authentication file path
[ https://issues.apache.org/jira/browse/YARN-10228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112429#comment-17112429 ] Hudson commented on YARN-10228: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18281 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18281/]) YARN-10228. Relax restriction of file path character in (eyang: rev 726b8e324b6fc99aac5a26fbbc7edd26a3a25479) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/ServiceApiUtil.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/utils/TestServiceApiUtil.java > Yarn Service fails if am java opts contains ZK authentication file path > --- > > Key: YARN-10228 > URL: https://issues.apache.org/jira/browse/YARN-10228 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Fix For: 3.3.0, 3.4.0, 3.3.1 > > Attachments: YARN-10228.001.patch > > > If i configure > {code:java} > yarn.service.am.java.opts=-Xmx768m > -Djava.security.auth.login.config=/opt/hadoop/etc/jaas-zk.conf > {code} > Invalid character error is getting printed . > This is due to jvm opts validation added in YARN-9718 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9606) Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient
[ https://issues.apache.org/jira/browse/YARN-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111817#comment-17111817 ] Hudson commented on YARN-9606: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18279 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18279/]) YARN-9606. Set sslfactory for AuthenticatedURL() while creating (pjoseph: rev cef0756929458f65c4f6731fe21d659100bcc8ea) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/util/WebServiceClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/webapp/TestWebServiceClient.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/util/TestWebServiceClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/Router.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServiceClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/RouterWebServiceUtil.java > Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient > -- > > Key: YARN-9606 > URL: https://issues.apache.org/jira/browse/YARN-9606 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-9606-001.patch, YARN-9606-002.patch, > YARN-9606.003.patch, YARN-9606.004.patch, YARN-9606.005.patch, > YARN-9606.006.patch, YARN-9606.007.patch, YARN-9606.008.patch > > > Yarn logs fails for running containers > > > {quote} > > > > Unable to fetch log files list > Exception in thread "main" java.io.IOException: > com.sun.jersey.api.client.ClientHandlerException: > javax.net.ssl.SSLHandshakeException: Error while authenticating with > endpoint: > [https://vm2:65321/ws/v1/node/containers/container_e05_1559802125016_0001_01_08/logs] > at > org.apache.hadoop.yarn.client.cli.LogsCLI.getContainerLogFiles(LogsCLI.java:543) > at > org.apache.hadoop.yarn.client.cli.LogsCLI.getMatchedContainerLogFiles(LogsCLI.java:1338) > at > org.apache.hadoop.yarn.client.cli.LogsCLI.getMatchedOptionForRunningApp(LogsCLI.java:1514) > at > org.apache.hadoop.yarn.client.cli.LogsCLI.fetchContainerLogs(LogsCLI.java:1052) > at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:367) > at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:152) > at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:399) > {quote} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9606) Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient
[ https://issues.apache.org/jira/browse/YARN-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17110026#comment-17110026 ] Hudson commented on YARN-9606: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18266 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18266/]) Revert "YARN-9606. Set sslfactory for AuthenticatedURL() while creating (aajisaka: rev b65815d6914996fed25bd9fef4d37d00828bc664) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml > Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient > -- > > Key: YARN-9606 > URL: https://issues.apache.org/jira/browse/YARN-9606 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-9606-001.patch, YARN-9606-002.patch, > YARN-9606.003.patch > > > Yarn logs fails for running containers > > > {quote} > > > > Unable to fetch log files list > Exception in thread "main" java.io.IOException: > com.sun.jersey.api.client.ClientHandlerException: > javax.net.ssl.SSLHandshakeException: Error while authenticating with > endpoint: > [https://vm2:65321/ws/v1/node/containers/container_e05_1559802125016_0001_01_08/logs] > at > org.apache.hadoop.yarn.client.cli.LogsCLI.getContainerLogFiles(LogsCLI.java:543) > at > org.apache.hadoop.yarn.client.cli.LogsCLI.getMatchedContainerLogFiles(LogsCLI.java:1338) > at > org.apache.hadoop.yarn.client.cli.LogsCLI.getMatchedOptionForRunningApp(LogsCLI.java:1514) > at > org.apache.hadoop.yarn.client.cli.LogsCLI.fetchContainerLogs(LogsCLI.java:1052) > at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:367) > at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:152) > at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:399) > {quote} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9606) Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient
[ https://issues.apache.org/jira/browse/YARN-9606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107350#comment-17107350 ] Hudson commented on YARN-9606: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #18257 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18257/]) YARN-9606. Set sslfactory for AuthenticatedURL() while creating (pjoseph: rev 7836bc4c3533e93e7adc0c7da0659bc04bdf2494) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/pom.xml > Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient > -- > > Key: YARN-9606 > URL: https://issues.apache.org/jira/browse/YARN-9606 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-9606-001.patch, YARN-9606-002.patch, > YARN-9606.003.patch > > > Yarn logs fails for running containers > > > {quote} > > > > Unable to fetch log files list > Exception in thread "main" java.io.IOException: > com.sun.jersey.api.client.ClientHandlerException: > javax.net.ssl.SSLHandshakeException: Error while authenticating with > endpoint: > [https://vm2:65321/ws/v1/node/containers/container_e05_1559802125016_0001_01_08/logs] > at > org.apache.hadoop.yarn.client.cli.LogsCLI.getContainerLogFiles(LogsCLI.java:543) > at > org.apache.hadoop.yarn.client.cli.LogsCLI.getMatchedContainerLogFiles(LogsCLI.java:1338) > at > org.apache.hadoop.yarn.client.cli.LogsCLI.getMatchedOptionForRunningApp(LogsCLI.java:1514) > at > org.apache.hadoop.yarn.client.cli.LogsCLI.fetchContainerLogs(LogsCLI.java:1052) > at org.apache.hadoop.yarn.client.cli.LogsCLI.runCommand(LogsCLI.java:367) > at org.apache.hadoop.yarn.client.cli.LogsCLI.run(LogsCLI.java:152) > at org.apache.hadoop.yarn.client.cli.LogsCLI.main(LogsCLI.java:399) > {quote} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10259) Reserved Containers not allocated from available space of other nodes in CandidateNodeSet in MultiNodePlacement
[ https://issues.apache.org/jira/browse/YARN-10259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107239#comment-17107239 ] Hudson commented on YARN-10259: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18256 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18256/]) YARN-10259. Fix reservation logic in Multi Node Placement. (pjoseph: rev 6ce295b78737aca8103912121d54f318cb5d36ef) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerMultiNodes.java > Reserved Containers not allocated from available space of other nodes in > CandidateNodeSet in MultiNodePlacement > --- > > Key: YARN-10259 > URL: https://issues.apache.org/jira/browse/YARN-10259 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 3.2.0, 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10259-001.patch, YARN-10259-002.patch, > YARN-10259-003.patch > > > Reserved Containers are not allocated from the available space of other nodes > in CandidateNodeSet in MultiNodePlacement. > *Repro:* > 1. MultiNode Placement Enabled. > 2. Two nodes h1 and h2 with 8GB > 3. Submit app1 AM (5GB) which gets placed in h1 and app2 AM (5GB) which gets > placed in h2. > 4. Submit app3 AM which is reserved in h1 > 5. Kill app2 which frees space in h2. > 6. app3 AM never gets ALLOCATED > RM logs shows YARN-8127 fix rejecting the allocation proposal for app3 AM on > h2 as it expects the assignment to be on same node where reservation has > happened. > {code} > 2020-05-05 18:49:37,264 DEBUG [AsyncDispatcher event handler] > scheduler.SchedulerApplicationAttempt > (SchedulerApplicationAttempt.java:commonReserve(573)) - Application attempt > appattempt_1588684773609_0003_01 reserved container > container_1588684773609_0003_01_01 on node host: h1:1234 #containers=1 > available= used=. This attempt > currently has 1 reserved containers at priority 0; currentReservation > > 2020-05-05 18:49:37,264 INFO [AsyncDispatcher event handler] > fica.FiCaSchedulerApp (FiCaSchedulerApp.java:apply(670)) - Reserved > container=container_1588684773609_0003_01_01, on node=host: h1:1234 > #containers=1 available= used= > with resource= >RESERVED=[(Application=appattempt_1588684773609_0003_01; > Node=h1:1234; Resource=)] > > 2020-05-05 18:49:38,283 DEBUG [Time-limited test] > allocator.RegularContainerAllocator > (RegularContainerAllocator.java:assignContainer(514)) - assignContainers: > node=h2 application=application_1588684773609_0003 priority=0 > pendingAsk=,repeat=1> > type=OFF_SWITCH > 2020-05-05 18:49:38,285 DEBUG [Time-limited test] fica.FiCaSchedulerApp > (FiCaSchedulerApp.java:commonCheckContainerAllocation(371)) - Try to allocate > from reserved container container_1588684773609_0003_01_01, but node is > not reserved >ALLOCATED=[(Application=appattempt_1588684773609_0003_01; > Node=h2:1234; Resource=)] > {code} > Attached testcase which reproduces the issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10265) Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue
[ https://issues.apache.org/jira/browse/YARN-10265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107179#comment-17107179 ] Hudson commented on YARN-10265: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18255 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18255/]) YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix (surendralilhore: rev 1958cb7c2bed08bd295a41fb4400b869a547b667) * (edit) hadoop-hdfs-project/hadoop-hdfs-client/pom.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml * (edit) hadoop-project/pom.xml * (edit) hadoop-hdfs-project/hadoop-hdfs/pom.xml > Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue > -- > > Key: YARN-10265 > URL: https://issues.apache.org/jira/browse/YARN-10265 > Project: Hadoop YARN > Issue Type: Bug >Reporter: liusheng >Assignee: liusheng >Priority: Major > Fix For: 3.3.0, 3.4.0, 3.3.1 > > Attachments: YARN-10265.001.patch > > > In the issue: https://issues.apache.org/jira/browse/YARN-9898 we have added a > workaround using an non-official released netty-4.1.48 to fix the ARM support > issue. but just a few hours agon, Netty has release the 4.1.50 version which > is officially support ARM platform, please see: > [https://github.com/netty/netty/pull/9804] > > netty-4.1.50.Final release: > [https://github.com/netty/netty/releases/tag/netty-4.1.50.Final] > commits from netty-4.1.48 to netty-4.1.50: > [https://github.com/netty/netty/compare/netty-4.1.48.Final...netty-4.1.50.Final] > So, now it is better to upgrade the netty-dependency version of Hadoop to > 4.1.50 version. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9898) Dependency netty-all-4.1.27.Final doesn't support ARM platform
[ https://issues.apache.org/jira/browse/YARN-9898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106614#comment-17106614 ] Hudson commented on YARN-9898: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18254 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18254/]) YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM (ayushsaxena: rev 0918433b4da1affbe380988b8f63fca39bc0850b) * (edit) hadoop-project/pom.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml * (edit) hadoop-hdfs-project/hadoop-hdfs-client/pom.xml * (edit) hadoop-hdfs-project/hadoop-hdfs/pom.xml > Dependency netty-all-4.1.27.Final doesn't support ARM platform > -- > > Key: YARN-9898 > URL: https://issues.apache.org/jira/browse/YARN-9898 > Project: Hadoop YARN > Issue Type: Bug >Reporter: liusheng >Assignee: liusheng >Priority: Major > Fix For: 3.3.0, 3.4.0 > > Attachments: YARN-9898.001.patch, YARN-9898.002.patch, > YARN-9898.003.patch, YARN-9898.004.patch > > > Hadoop dependent the Netty package, but the *netty-all-4.1.27.Final* of > io.netty maven repo, cannot support ARM platform. > When run the test *TestCsiClient.testIdentityService* on ARM server, it will > raise error like following: > {code:java} > Caused by: java.io.FileNotFoundException: > META-INF/native/libnetty_transport_native_epoll_aarch_64.so > at > io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:161) > ... 45 more > Suppressed: java.lang.UnsatisfiedLinkError: no > netty_transport_native_epoll_aarch_64 in java.library.path > at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) > at java.lang.Runtime.loadLibrary0(Runtime.java:870) > at java.lang.System.loadLibrary(System.java:1122) > at > io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38) > at > io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243) > at > io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:124) > ... 45 more > Suppressed: java.lang.UnsatisfiedLinkError: no > netty_transport_native_epoll_aarch_64 in java.library.path > at > java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867) > at java.lang.Runtime.loadLibrary0(Runtime.java:870) > at java.lang.System.loadLibrary(System.java:1122) > at > io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38) > at > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263) > at java.security.AccessController.doPrivileged(Native > Method) > at > io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255) > at > io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233) > ... 46 more > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8942) PriorityBasedRouterPolicy throws exception if all sub-cluster weights have negative value
[ https://issues.apache.org/jira/browse/YARN-8942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106498#comment-17106498 ] Hudson commented on YARN-8942: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18251 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18251/]) YARN-8942. PriorityBasedRouterPolicy throws exception if all sub-cluster (inigoiri: rev 108ecf992f0004dd64a7143d1c400de1361b13f3) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/TestPriorityRouterPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/router/PriorityRouterPolicy.java > PriorityBasedRouterPolicy throws exception if all sub-cluster weights have > negative value > - > > Key: YARN-8942 > URL: https://issues.apache.org/jira/browse/YARN-8942 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.3.0 >Reporter: Akshay Agarwal >Assignee: Bilwa S T >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-8942.001.patch, YARN-8942.002.patch > > > In *PriorityBasedRouterPolicy* if all sub-cluster weights are *set to > negative values* it is throwing exception while running a job. > Ideally it should handle the negative priority as well according to the home > sub cluster selection process of the policy. > *Exception Details:* > {code:java} > java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Unable > to insert the ApplicationId application_1540356760422_0015 into the > FederationStateStore > at > org.apache.hadoop.yarn.server.router.RouterServerUtil.logAndThrowException(RouterServerUtil.java:56) > at > org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor.submitApplication(FederationClientInterceptor.java:418) > at > org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService.submitApplication(RouterClientRMService.java:218) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:282) > at > org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:579) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:872) > at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:818) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2678) > Caused by: > org.apache.hadoop.yarn.server.federation.store.exception.FederationStateStoreInvalidInputException: > Missing SubCluster Id information. Please try again by specifying Subcluster > Id information. > at > org.apache.hadoop.yarn.server.federation.store.utils.FederationMembershipStateStoreInputValidator.checkSubClusterId(FederationMembershipStateStoreInputValidator.java:247) > at > org.apache.hadoop.yarn.server.federation.store.utils.FederationApplicationHomeSubClusterStoreInputValidator.checkApplicationHomeSubCluster(FederationApplicationHomeSubClusterStoreInputValidator.java:160) > at > org.apache.hadoop.yarn.server.federation.store.utils.FederationApplicationHomeSubClusterStoreInputValidator.validate(FederationApplicationHomeSubClusterStoreInputValidator.java:65) > at > org.apache.hadoop.yarn.server.federation.store.impl.ZookeeperFederationStateStore.addApplicationHomeSubCluster(ZookeeperFederationStateStore.java:159) > at sun.reflect.GeneratedMethodAccessor30.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157) > at > org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) > at > org.apache.hadoop.
[jira] [Commented] (YARN-10154) CS Dynamic Queues cannot be configured with absolute resources
[ https://issues.apache.org/jira/browse/YARN-10154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106159#comment-17106159 ] Hudson commented on YARN-10154: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18249 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18249/]) YARN-10154. Addendum Patch which fixes below bugs (pjoseph: rev 450e5aa9dd49eae46a0e05151bbddc56083eafd5) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceWithAutoQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ManagedParentQueue.java > CS Dynamic Queues cannot be configured with absolute resources > -- > > Key: YARN-10154 > URL: https://issues.apache.org/jira/browse/YARN-10154 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.3 >Reporter: Sunil G >Assignee: Manikandan R >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10154.001.patch, YARN-10154.002.patch, > YARN-10154.003.patch, YARN-10154.addendum-001.patch, > YARN-10154.addendum-002.patch, YARN-10154.addendum-003.patch, > YARN-10154.addendum-004.patch > > > In CS, ManagedParent Queue and its template cannot take absolute resource > value like > [memory=8192,vcores=8] > Thsi Jira is to track and improve the configuration reading module of > DynamicQueue to support absolute resource values. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10260) Allow transitioning queue from DRAINING to RUNNING state
[ https://issues.apache.org/jira/browse/YARN-10260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105648#comment-17105648 ] Hudson commented on YARN-10260: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18243 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18243/]) YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. (jhung: rev fff1d2c1226ec23841b04dd478e8b97f31abbeba) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueState.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java > Allow transitioning queue from DRAINING to RUNNING state > > > Key: YARN-10260 > URL: https://issues.apache.org/jira/browse/YARN-10260 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Bilwa S T >Priority: Major > Fix For: 3.2.2, 2.10.1, 3.4.0, 3.3.1, 3.1.5 > > Attachments: YARN-10260.001.patch > > > We found in our cluster, a queue was erroneously stopped. Then queue is > internally in DRAINING state. It cannot be moved back to RUNNING state until > the queue is finished draining. For queues with large workloads, this can > block other apps from submitting to this queue for a long time. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9301) Too many InvalidStateTransitionException with SLS
[ https://issues.apache.org/jira/browse/YARN-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105546#comment-17105546 ] Hudson commented on YARN-9301: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18240 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18240/]) YARN-9301. Too many InvalidStateTransitionException with SLS. (inigoiri: rev 96bbc3bc972619bd830b2f935c06a1585a5470c6) * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/resourcemanager/MockAMLauncher.java > Too many InvalidStateTransitionException with SLS > - > > Key: YARN-9301 > URL: https://issues.apache.org/jira/browse/YARN-9301 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin Chundatt >Assignee: Bilwa S T >Priority: Major > Labels: simulator > Fix For: 3.4.0 > > Attachments: YARN-9301-001.patch, YARN-9301.002.patch > > > Too many InvalidStateTransistionExcetion > {noformat} > 19/02/13 17:44:43 ERROR rmcontainer.RMContainerImpl: Can't handle this event > at current state > org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: > LAUNCHED at RUNNING > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487) > at > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:483) > at > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:65) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.containerLaunchedOnNode(SchedulerApplicationAttempt.java:655) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.containerLaunchedOnNode(AbstractYarnScheduler.java:359) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.updateNewContainerInfo(AbstractYarnScheduler.java:1010) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.nodeUpdate(AbstractYarnScheduler.java:1112) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:1295) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1752) > at > org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.handle(SLSCapacityScheduler.java:205) > at > org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.handle(SLSCapacityScheduler.java:60) > at > org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66) > at java.lang.Thread.run(Thread.java:745) > 19/02/13 17:44:43 ERROR rmcontainer.RMContainerImpl: Invalid event LAUNCHED > on container container_1550059705491_0067_01_01 > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9301) Too many InvalidStateTransitionException with SLS
[ https://issues.apache.org/jira/browse/YARN-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105533#comment-17105533 ] Hudson commented on YARN-9301: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18239 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18239/]) YARN-9301. Too many InvalidStateTransitionException with SLS. (inigoiri: rev 9cbd0cd2a9268ff2e8fed0af335e9c4f91c5f601) * (edit) hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/resourcemanager/MockAMLauncher.java > Too many InvalidStateTransitionException with SLS > - > > Key: YARN-9301 > URL: https://issues.apache.org/jira/browse/YARN-9301 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin Chundatt >Assignee: Bilwa S T >Priority: Major > Labels: simulator > Fix For: 3.4.0 > > Attachments: YARN-9301-001.patch, YARN-9301.002.patch > > > Too many InvalidStateTransistionExcetion > {noformat} > 19/02/13 17:44:43 ERROR rmcontainer.RMContainerImpl: Can't handle this event > at current state > org.apache.hadoop.yarn.state.InvalidStateTransitionException: Invalid event: > LAUNCHED at RUNNING > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$500(StateMachineFactory.java:46) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:487) > at > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:483) > at > org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl.handle(RMContainerImpl.java:65) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.containerLaunchedOnNode(SchedulerApplicationAttempt.java:655) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.containerLaunchedOnNode(AbstractYarnScheduler.java:359) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.updateNewContainerInfo(AbstractYarnScheduler.java:1010) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.nodeUpdate(AbstractYarnScheduler.java:1112) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:1295) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1752) > at > org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.handle(SLSCapacityScheduler.java:205) > at > org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.handle(SLSCapacityScheduler.java:60) > at > org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66) > at java.lang.Thread.run(Thread.java:745) > 19/02/13 17:44:43 ERROR rmcontainer.RMContainerImpl: Invalid event LAUNCHED > on container container_1550059705491_0067_01_01 > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9017) PlacementRule order is not maintained in CS
[ https://issues.apache.org/jira/browse/YARN-9017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17101178#comment-17101178 ] Hudson commented on YARN-9017: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18224 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18224/]) YARN-9017. PlacementRule order is not maintained in CS. Contributed by (inigoiri: rev 35010120fbbcad8618f99abf7130e53f98879a33) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfigValidator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestPlacementManager.java > PlacementRule order is not maintained in CS > --- > > Key: YARN-9017 > URL: https://issues.apache.org/jira/browse/YARN-9017 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.4.0 >Reporter: Bibin Chundatt >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-9017.001.patch, YARN-9017.002.patch, > YARN-9017.003.patch > > > {{yarn.scheduler.queue-placement-rules}} doesn't work as expected in Capacity > Scheduler > {quote} > * **Queue Mapping Interface based on Default or User Defined Placement > Rules** - This feature allows users to map a job to a specific queue based on > some default placement rule. For instance based on user & group, or > application name. User can also define their own placement rule. > {quote} > As per current UserGroupMapping is always added in placementRule. > {{CapacityScheduler#updatePlacementRules}} > {code} > // Initialize placement rules > Collection placementRuleStrs = conf.getStringCollection( > YarnConfiguration.QUEUE_PLACEMENT_RULES); > List placementRules = new ArrayList<>(); > ... > // add UserGroupMappingPlacementRule if absent > distingushRuleSet.add(YarnConfiguration.USER_GROUP_PLACEMENT_RULE); > {code} > PlacementRule configuration order is not maintained -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8959) TestContainerResizing fails randomly
[ https://issues.apache.org/jira/browse/YARN-8959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17101054#comment-17101054 ] Hudson commented on YARN-8959: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18222 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18222/]) YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via (jeagles: rev 92e3ebb40199aec0890b868b8d6bf2d7fe90abbf) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java > TestContainerResizing fails randomly > > > Key: YARN-8959 > URL: https://issues.apache.org/jira/browse/YARN-8959 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin Chundatt >Assignee: Ahmed Hussein >Priority: Minor > Fix For: 3.2.2, 2.10.1, 3.4.0, 3.3.1, 3.1.5 > > Attachments: YARN-8959-branch-2.10.002.patch, > YARN-8959-branch-2.10.003.patch, YARN-8959-branch-2.10.004.patch, > YARN-8959.001.patch, YARN-8959.002.patch, YARN-8959.003.patch > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.testSimpleDecreaseContainer > {code} > testSimpleDecreaseContainer(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing) > Time elapsed: 0.348 s <<< FAILURE! > java.lang.AssertionError: expected:<1024> but was:<3072> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at org.junit.Assert.assertEquals(Assert.java:631) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.checkUsedResource(TestContainerResizing.java:1011) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.testSimpleDecreaseContainer(TestContainerResizing.java:210) > {code} > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.testIncreaseContainerUnreservedWhenContainerCompleted > {code} > testIncreaseContainerUnreservedWhenContainerCompleted(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing) > Time elapsed: 0.445 s <<< FAILURE! > java.lang.AssertionError: expected:<1024> but was:<7168> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at org.junit.Assert.assertEquals(Assert.java:631) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.checkUsedResource(TestContainerResizing.java:1011) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.testIncreaseContainerUnreservedWhenContainerCompleted(TestContainerResizing.java:729) > {code} > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.testExcessiveReservationWhenDecreaseSameContainer > {code} > testExcessiveReservationWhenDecreaseSameContainer(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing) > Time elapsed: 0.321 s <<< FAILURE! > java.lang.AssertionError: expected:<1024> but was:<2048> > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.failNotEquals(Assert.java:834) > at org.junit.Assert.assertEquals(Assert.java:645) > at org.junit.Assert.assertEquals(Assert.java:631) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.checkUsedResource(TestContainerResizing.java:1015) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing.testExcessiveReservationWhenDecreaseSameContainer(TestContainerResizing.java:623) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10257) FS-CS converter: skip increment properties for mem/vcores and fix DRF check
[ https://issues.apache.org/jira/browse/YARN-10257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099929#comment-17099929 ] Hudson commented on YARN-10257: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18218 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18218/]) YARN-10257. FS-CS converter: skip increment properties for mem/vcores (snemeth: rev cb6399c1095af52112cbf4356572d99923d694ae) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSYarnSiteConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverterMain.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/pom.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSYarnSiteConverter.java > FS-CS converter: skip increment properties for mem/vcores and fix DRF check > --- > > Key: YARN-10257 > URL: https://issues.apache.org/jira/browse/YARN-10257 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10257-001.patch, YARN-10257-002.patch > > > Two issues have been discovered during fs2cs testing: > 1. The conversion of allocation increment properties are not needed: > {{yarn.scheduler.increment-allocation-mb}} > {{yarn.scheduler.increment-allocation-vcores}} > {{yarn.resource-types.memory-mb.increment-allocation}} > {{yarn.resource-types.vcores.increment-allocation}} > 2. The following piece of code is incorrect - the default scheduling policy > can be different from DRF, which is a problem if DRF is used everywhere else: > {code} > private boolean isDrfUsed(FairScheduler fs) { > FSQueue rootQueue = fs.getQueueManager().getRootQueue(); > AllocationConfiguration allocConf = fs.getAllocationConfiguration(); > String defaultPolicy = allocConf.getDefaultSchedulingPolicy().getName(); > if (DominantResourceFairnessPolicy.NAME.equals(defaultPolicy)) { > return true; > } else { > return isDrfUsedOnQueueLevel(rootQueue); > } > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10160) Add auto queue creation related configs to RMWebService#CapacitySchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-10160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099803#comment-17099803 ] Hudson commented on YARN-10160: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18217 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18217/]) YARN-10160. Add auto queue creation related configs to (snemeth: rev 0debe55d6cf36b358c86c27d43991aa44baef4f2) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/webapp/WebServicesTestUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/LeafQueueTemplateInfo.java > Add auto queue creation related configs to > RMWebService#CapacitySchedulerQueueInfo > -- > > Key: YARN-10160 > URL: https://issues.apache.org/jira/browse/YARN-10160 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Major > Fix For: 3.4.0 > > Attachments: Screen Shot 2020-02-25 at 9.06.52 PM.png, > YARN-10160-001.patch, YARN-10160-002.patch, YARN-10160-003.patch, > YARN-10160-004.patch, YARN-10160-005.patch, YARN-10160-006.patch, > YARN-10160-007.patch, YARN-10160-008.patch, YARN-10160-009.patch > > > Add auto queue creation related configs to > RMWebService#CapacitySchedulerQueueInfo. > {code} > yarn.scheduler.capacity..auto-create-child-queue.enabled > yarn.scheduler.capacity..leaf-queue-template. > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10256) Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic
[ https://issues.apache.org/jira/browse/YARN-10256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099049#comment-17099049 ] Hudson commented on YARN-10256: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18213 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18213/]) YARN-10256. Refactor (jeagles: rev f5081a9a5d51fe781da97f07ac4dca8dd49c98e2) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/TestContainerSchedulerQueuing.java > Refactor > TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic > --- > > Key: YARN-10256 > URL: https://issues.apache.org/jira/browse/YARN-10256 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: refactoring, unit-test > Attachments: YARN-10256.001.patch > > > In 3.x, > {{TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic}} > has redundant assertions. Since the UT throws timeout exception, > {{GenericTestsUtils.waitFor()}} guarantees that the predicate is met > successfully. Otherwise, the UT would throw a timeout exception. > The redundant loop causes confusion in understanding the test unit and may > increase the possibility of failure in case the container terminates -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6553) Replace MockResourceManagerFacade with MockRM for AMRMProxy/Router tests
[ https://issues.apache.org/jira/browse/YARN-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17097861#comment-17097861 ] Hudson commented on YARN-6553: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18206 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18206/]) YARN-6553. Replace MockResourceManagerFacade with MockRM for (surendralilhore: rev 257b04ce4e153cb88dfc59eab63625909efdd56b) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestFederationClientInterceptorRetry.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestRouterClientRMService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestFederationClientInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestableFederationClientInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/rmadmin/MockRMAdminRequestInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/BaseRouterClientRMTest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/pom.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/MockClientRequestInterceptor.java > Replace MockResourceManagerFacade with MockRM for AMRMProxy/Router tests > > > Key: YARN-6553 > URL: https://issues.apache.org/jira/browse/YARN-6553 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-6553.001.patch, YARN-6553.002.patch, > YARN-6553.003.patch, YARN-6553.004.patch > > > Currently the AMRMProxy and Router tests use the > {{MockResourceManagerFacade}}. This jira proposes replacing it with > {{MockRM}} as is done in majority of the tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6973) Adding RM Cluster Id in ApplicationReport
[ https://issues.apache.org/jira/browse/YARN-6973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17095785#comment-17095785 ] Hudson commented on YARN-6973: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18200 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18200/]) YARN-6973. Adding RM Cluster Id in ApplicationReport. Contributed by (inigoiri: rev d125d3910843eeaa25dd09fae493c6fd258757e5) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationReport.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationReportPBImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java > Adding RM Cluster Id in ApplicationReport > -- > > Key: YARN-6973 > URL: https://issues.apache.org/jira/browse/YARN-6973 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-6973.001.patch, YARN-6973.002.patch, > YARN-6973.003.patch > > > Adding RM Cluster Id in ApplicationReport. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10247) Application priority queue ACLs are not respected
[ https://issues.apache.org/jira/browse/YARN-10247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17095490#comment-17095490 ] Hudson commented on YARN-10247: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18198 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18198/]) YARN-10247. Application priority queue ACLs are not respected. (snemeth: rev 410c605aec308a2ccd903f60aade3aaeefcaa610) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationPriorityACLs.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java > Application priority queue ACLs are not respected > - > > Key: YARN-10247 > URL: https://issues.apache.org/jira/browse/YARN-10247 > Project: Hadoop YARN > Issue Type: Task > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G >Priority: Blocker > Fix For: 3.3.0, 3.4.0 > > Attachments: YARN-10247.0001.patch > > > This is a regression from queue path jira. > App priority acls are not working correctly. > {code:java} > yarn.scheduler.capacity.root.B.acl_application_max_priority=[user=john > group=users max_priority=4] > {code} > max_priority enforcement is not working. For user john, maximum supported > priority is 4. However I can submit like priority 6 for this user. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10215) Endpoint for obtaining direct URL for the logs
[ https://issues.apache.org/jira/browse/YARN-10215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17094701#comment-17094701 ] Hudson commented on YARN-10215: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18194 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18194/]) YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed (snemeth: rev ab3642955971dec1ce285f39cf0f02e6cc64b4b2) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-jhs-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-app-jhs-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-jhs-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/helpers/log-adapter-helper.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-jhs-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/YarnWebServiceParams.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-jhs-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-redirect-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogServlet.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-jhs-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-jhs-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-app-redirect-log.js * (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/main/java/org/apache/hadoop/yarn/server/applicationhistoryservice/webapp/AHSWebServices.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-redirect-log.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-jhs-redirect-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/LogWebService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-app-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/controllers/yarn-app/logs.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-redirect-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-jhs-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-app-jhs-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-jhs-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-app-jhs-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-app-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/adapters/yarn-log.js > Endpoint for obtaining direct URL for the logs > -- > > Key: YARN-10215 > URL: https://issues.apache.org/jira/browse/YARN-10215 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Andras Gyori >Priority: Major > Fix For: 3.3.0, 3.4.0 > > Attachments: YARN-10025.001.patch, YARN-10025.002.patch, > YARN-10025.003.patch, YARN-10025.004.patch > > > If CORS protected UIs are set up, there is an issue when the browser tries to > access the logs of a running container in the RM web UIv2. > Assuming ATS is not up, the browser follows the following call chain: > - Tries to access ATS, it fails, falls back to JHS > - From RM the browser received basic app info, we know that the application > is running > - From the JHS we got the list of containers and th
[jira] [Commented] (YARN-9848) revert YARN-4946
[ https://issues.apache.org/jira/browse/YARN-9848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093926#comment-17093926 ] Hudson commented on YARN-9848: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18191 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18191/]) YARN-9848. Revert YARN-4946. Contributed by Steven Rand. (aajisaka: rev 6ef01646ba9540248ffaf322f4678fbdac28fcd4) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/AppManagerTestBase.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java > revert YARN-4946 > > > Key: YARN-9848 > URL: https://issues.apache.org/jira/browse/YARN-9848 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation, resourcemanager >Reporter: Steven Rand >Assignee: Steven Rand >Priority: Blocker > Fix For: 3.3.0 > > Attachments: YARN-9848-01.patch, YARN-9848.002.patch, > YARN-9848.003.patch > > > In YARN-4946, we've been discussing a revert due to the potential for keeping > more applications in the state store than desired, and the potential to > greatly increase RM recovery times. > > I'm in favor of reverting the patch, but other ideas along the lines of > YARN-9571 would work as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4946) RM should not consider an application as COMPLETED when log aggregation is not in a terminal state
[ https://issues.apache.org/jira/browse/YARN-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093927#comment-17093927 ] Hudson commented on YARN-4946: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18191 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18191/]) YARN-9848. Revert YARN-4946. Contributed by Steven Rand. (aajisaka: rev 6ef01646ba9540248ffaf322f4678fbdac28fcd4) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/MockRMApp.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/AppManagerTestBase.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java > RM should not consider an application as COMPLETED when log aggregation is > not in a terminal state > -- > > Key: YARN-4946 > URL: https://issues.apache.org/jira/browse/YARN-4946 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 2.8.0 >Reporter: Robert Kanter >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.2.0 > > Attachments: YARN-4946.001.patch, YARN-4946.002.patch, > YARN-4946.003.patch, YARN-4946.004.patch > > > MAPREDUCE-6415 added a tool that combines the aggregated log files for each > Yarn App into a HAR file. When run, it seeds the list by looking at the > aggregated logs directory, and then filters out ineligible apps. One of the > criteria involves checking with the RM that an Application's log aggregation > status is not still running and has not failed. When the RM "forgets" about > an older completed Application (e.g. RM failover, enough time has passed, > etc), the tool won't find the Application in the RM and will just assume that > its log aggregation succeeded, even if it actually failed or is still running. > We can solve this problem by doing the following: > The RM should not consider an app to be fully completed (and thus removed > from its history) until the aggregation status has reached a terminal state > (e.g. SUCCEEDED, FAILED, TIME_OUT). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10156) Fix typo 'complaint' which means quite different in Federation.md
[ https://issues.apache.org/jira/browse/YARN-10156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093526#comment-17093526 ] Hudson commented on YARN-10156: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18188 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18188/]) YARN-10156. Destroy Jersey Client in TimelineConnector. (pjoseph: rev 9224568b0ea9c97071d45e1c26241c64efdd2926) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineConnector.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java > Fix typo 'complaint' which means quite different in Federation.md > - > > Key: YARN-10156 > URL: https://issues.apache.org/jira/browse/YARN-10156 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation, federation >Reporter: Sungpeo Kook >Assignee: Sungpeo Kook >Priority: Minor > Fix For: 3.3.0, 2.9.3, 3.2.2, 3.1.4, 2.10.1 > > > [https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/Federation.html#Running_Applications_across_Sub-Clusters] > There is a typo in Fedration.md that could be confusing. > {quote}{{`be complaint to`}} should be {{`be compliant with`}} on the context. > {quote} > I think that it need to be corrected. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10194) YARN RMWebServices /scheduler-conf/validate leaks ZK Connections
[ https://issues.apache.org/jira/browse/YARN-10194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17093063#comment-17093063 ] Hudson commented on YARN-10194: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18186 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18186/]) YARN-10194. YARN RMWebServices /scheduler-conf/validate leaks ZK (snemeth: rev f91e21ac109e753e76d19c5c872c59a767b7b837) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ZKConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfigValidator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/FSSchedulerConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/InMemoryConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/YarnConfigurationStore.java > YARN RMWebServices /scheduler-conf/validate leaks ZK Connections > > > Key: YARN-10194 > URL: https://issues.apache.org/jira/browse/YARN-10194 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 3.3.0 >Reporter: Akhil PB >Assignee: Prabhu Joseph >Priority: Blocker > Fix For: 3.3.0, 3.4.0 > > Attachments: YARN-10194-001.patch, YARN-10194-002.patch, > YARN-10194-003.patch, YARN-10194-004.patch, YARN-10194-005.patch > > > YARN RMWebServices /scheduler-conf/validate leaks ZK Connections. Validation > API creates a new CapacityScheduler and missed to close after the validation. > Every CapacityScheduler#init opens MutableCSConfigurationProvider which opens > ZKConfigurationStore and creates a ZK Connection. > *ZK LOGS* > {code} > -03-12 16:45:51,881 WARN org.apache.zookeeper.server.NIOServerCnxnFactory: [2 > times] Error accepting new connection: Too many connections from > /172.27.99.64 - max is 60 > 2020-03-12 16:45:52,449 WARN > org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new > connection: Too many connections from /172.27.99.64 - max is 60 > 2020-03-12 16:45:52,710 WARN > org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new > connection: Too many connections from /172.27.99.64 - max is 60 > 2020-03-12 16:45:52,876 WARN > org.apache.zookeeper.server.NIOServerCnxnFactory: [4 times] Error accepting > new connection: Too many connections from /172.27.99.64 - max is 60 > 2020-03-12 16:45:53,068 WARN > org.apache.zookeeper.server.NIOServerCnxnFactory: [2 times] Error accepting > new connection: Too many connections from /172.27.99.64 - max is 60 > 2020-03-12 16:45:53,391 WARN > org.apache.zookeeper.server.NIOServerCnxnFactory: [2 times] Error accepting > new connection: Too many connections from /172.27.99.64 - max is 60 > 2020-03-12 16:45:54,008 WARN > org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new > connection: Too many connections from /172.27.99.64 - max is 60 > 2020-03-12 16:45:54,287 WARN > org.apache.zookeeper.server.NIOServerCnxnFactory: Error accepting new > connection: Too many connections from /172.27.99.64 - max is 60 > 2020-03-12 16:45:54,483 WARN > org.apache.zookeeper.server.NIOServerCnxnFactory: [4 times] Error accepting > new connection: Too many connections from /172.27.99.64 - max is 60 > {code} > And there is an another bug in ZKConfigurationStore which has not handled > close() of ZKCuratorManager. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10237) Add isAbsoluteResource config for queue in scheduler response
[ https://issues.apache.org/jira/browse/YARN-10237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091662#comment-17091662 ] Hudson commented on YARN-10237: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18179 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18179/]) YARN-10237. Add isAbsoluteResource config for queue in scheduler (snemeth: rev d4874585f4eb8620b64855d466ae82148ded0038) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java > Add isAbsoluteResource config for queue in scheduler response > - > > Key: YARN-10237 > URL: https://issues.apache.org/jira/browse/YARN-10237 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 3.4.0 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10237-001.patch, YARN-10237-002.patch, > YARN-10237-003.patch > > > Internal Config Management tools have difficulty in managing the capacity > scheduler queue configs if user toggles between Absolute Resource to > Percentage or vice versa. > This jira is to expose if a queue is configured in absolute resource or not > as part of scheduler response. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10223) Duplicate jersey-test-framework-core dependency in yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-10223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17091059#comment-17091059 ] Hudson commented on YARN-10223: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18176 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18176/]) YARN-10223. Remove jersey-test-framework-core dependency from (github: rev 9827ff2961e249e8be9023f39d91e1c0fbd23d61) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml > Duplicate jersey-test-framework-core dependency in yarn-server-common > - > > Key: YARN-10223 > URL: https://issues.apache.org/jira/browse/YARN-10223 > Project: Hadoop YARN > Issue Type: Bug > Components: build >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Minor > > The following warning appears in maven log. > {noformat} > [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must > be unique: > com.sun.jersey.jersey-test-framework:jersey-test-framework-core:jar -> > version (?) vs 1.19 @ line 148, column 17 > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10240) Prevent Fatal CancelledException in TimelineV2Client when stopping
[ https://issues.apache.org/jira/browse/YARN-10240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17088351#comment-17088351 ] Hudson commented on YARN-10240: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18170 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18170/]) YARN-10240. Prevent Fatal CancelledException in TimelineV2Client when (pjoseph: rev 60fa15366e8f2d59f4dc8e7beaa6edcbbcb9c18f) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineV2ClientImpl.java > Prevent Fatal CancelledException in TimelineV2Client when stopping > -- > > Key: YARN-10240 > URL: https://issues.apache.org/jira/browse/YARN-10240 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ATSv2 >Reporter: Tarun Parimi >Assignee: Tarun Parimi >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10240.001.patch > > > When the timeline client is stopped, it will cancel all sync EntityHolders > after waiting for a drain timeout. > {code:java} > // if some entities were not drained then we need interrupt > // the threads which had put sync EntityHolders to the > queue. > EntitiesHolder nextEntityInTheQueue = null; > while ((nextEntityInTheQueue = > timelineEntityQueue.poll()) != null) { > nextEntityInTheQueue.cancel(true); > } > {code} > We only handle interrupted exception here. > {code:java} > if (sync) { > // In sync call we need to wait till its published and if any error > then > // throw it back > try { > entitiesHolder.get(); > } catch (ExecutionException e) { > throw new YarnException("Failed while publishing entity", > e.getCause()); > } catch (InterruptedException e) { > Thread.currentThread().interrupt(); > throw new YarnException("Interrupted while publishing entity", e); > } > } > {code} > But calling nextEntityInTheQueue.cancel(true) will result in > entitiesHolder.get() throwing a CancelledException which is not handled. This > can result in FATAL error in NM. We need to prevent this. > {code:java} > FATAL event.AsyncDispatcher (AsyncDispatcher.java:dispatch(203)) - Error in > dispatcher thread > java.util.concurrent.CancellationException > at java.util.concurrent.FutureTask.report(FutureTask.java:121) > at java.util.concurrent.FutureTask.get(FutureTask.java:192) > at > org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher.dispatchEntities(TimelineV2ClientImpl.java:545) > at > org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.putEntities(TimelineV2ClientImpl.java:149) > at > org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.putEntity(NMTimelinePublisher.java:348) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9996) Code cleanup in QueueAdminConfigurationMutationACLPolicy
[ https://issues.apache.org/jira/browse/YARN-9996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086375#comment-17086375 ] Hudson commented on YARN-9996: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18159 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18159/]) YARN-9996. Code cleanup in QueueAdminConfigurationMutationACLPolicy. (snemeth: rev 92b28e38b7bcfb70a3e805ff40ef73d27ba24788) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/QueueAdminConfigurationMutationACLPolicy.java > Code cleanup in QueueAdminConfigurationMutationACLPolicy > > > Key: YARN-9996 > URL: https://issues.apache.org/jira/browse/YARN-9996 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Siddharth Ahuja >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-9996.001.patch > > > Method 'isMutationAllowed' contains many uses of substring and lastIndexOf. > These could be extracted and simplified. > Also, some logging could be added as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10189) Code cleanup in LeveldbRMStateStore
[ https://issues.apache.org/jira/browse/YARN-10189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086370#comment-17086370 ] Hudson commented on YARN-10189: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18158 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18158/]) YARN-10189. Code cleanup in LeveldbRMStateStore. Contributed by Benjamin (snemeth: rev 76900b4f5b7ad8d11baf505abc43f7b24d364071) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestLeveldbRMStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/LeveldbRMStateStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/LeveldbConfigurationStore.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/DBManager.java > Code cleanup in LeveldbRMStateStore > --- > > Key: YARN-10189 > URL: https://issues.apache.org/jira/browse/YARN-10189 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10189.001.patch, YARN-10189.POC001.patch, > YARN-10189.POC002.patch > > > Some things can be improved: > * throws Exception declaration can be removed from > LeveldbRMStateStore.initInternal method > * key variable is redundant in LeveldbRMStateStore.dbStoreVersion > * try can use automatic Resource management in > LeveldbRMStateStore.loadReservationState/loadRMDTSecretManagerKeys/loadRMDTSecretManagerTokens/loadRMApps/... > etc > * there were some methods which were copied to LeveldbConfigurationStore > (ie: openDatabase, storeVersion, loadVersion, CompactionTimerClass nested > class), a helper class could be created to reduce the duplicated code > * Any other cleanup -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9954) Configurable max application tags and max tag length
[ https://issues.apache.org/jira/browse/YARN-9954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085960#comment-17085960 ] Hudson commented on YARN-9954: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18156 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18156/]) YARN-9954. Configurable max application tags and max tag length. (jhung: rev 360105482258569e5e31c49d023d2bbdc3ef38cd) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationSubmissionContextPBImpl.java > Configurable max application tags and max tag length > > > Key: YARN-9954 > URL: https://issues.apache.org/jira/browse/YARN-9954 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Bilwa S T >Priority: Major > Fix For: 3.3.0, 3.2.2, 3.1.4, 2.10.1, 3.4.0 > > Attachments: YARN-9954-branch-3.3.patch, > YARN-9954-branch-3.3.v2.patch, YARN-9954-branch-3.3.v3.patch, > YARN-9954.001.patch, YARN-9954.002.patch, YARN-9954.003.patch > > > Currently max tags and max tag length is hardcoded, it should be configurable > {noformat} > @Evolving > public static final int APPLICATION_MAX_TAGS = 10; > @Evolving > public static final int APPLICATION_MAX_TAG_LENGTH = 100; {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10154) CS Dynamic Queues cannot be configured with absolute resources
[ https://issues.apache.org/jira/browse/YARN-10154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17085758#comment-17085758 ] Hudson commented on YARN-10154: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18154 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18154/]) YARN-10154. CS Dynamic Queues cannot be configured with absolute (sunilg: rev 2fe122e322afd35dfae6e45b3e7fdaddd20a411f) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoCreatedQueueBase.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractAutoCreatedLeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedLeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/queuemanagement/GuaranteedOrZeroCapacityOverTimePolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceWithAutoQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ManagedParentQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestAbsoluteResourceConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java > CS Dynamic Queues cannot be configured with absolute resources > -- > > Key: YARN-10154 > URL: https://issues.apache.org/jira/browse/YARN-10154 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.3 >Reporter: Sunil G >Assignee: Manikandan R >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10154.001.patch, YARN-10154.002.patch, > YARN-10154.003.patch > > > In CS, ManagedParent Queue and its template cannot take absolute resource > value like > [memory=8192,vcores=8] > Thsi Jira is to track and improve the configuration reading module of > DynamicQueue to support absolute resource values. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9999) TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, general code cleanup
[ https://issues.apache.org/jira/browse/YARN-?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084020#comment-17084020 ] Hudson commented on YARN-: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18147 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18147/]) YARN-. TestFSSchedulerConfigurationStore: Extend from (snemeth: rev 4db598e0e61bb13baf9ec7f854c0435cf5c71aa2) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestFSSchedulerConfigurationStore.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/ConfigurationStoreBaseTest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/PersistentConfigurationStoreBaseTest.java > TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, > general code cleanup > --- > > Key: YARN- > URL: https://issues.apache.org/jira/browse/YARN- > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Benjamin Teke >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-.001.patch, YARN-.002.patch, > YARN-.003.patch, YARN-.004.patch > > > All config store tests are extended from ConfigurationStoreBaseTest: > * TestInMemoryConfigurationStore > * TestLeveldbConfigurationStore > * TestZKConfigurationStore > TestFSSchedulerConfigurationStore should also extend from it. > Additionally, some general code cleanup can be applied as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10233) [YARN UI2] No Logs were found in "YARN Daemon Logs" page
[ https://issues.apache.org/jira/browse/YARN-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083918#comment-17083918 ] Hudson commented on YARN-10233: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18146 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18146/]) YARN-10233. Fix YARN UI2 Daemon Logs (pjoseph: rev 94f7470c71104af1871a22c34fbbce14a60c33ab) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/.gitignore * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/serializers/yarn-rm-log.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-tools/yarn-rm-log.hbs * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/templates/yarn-tools.hbs * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/models/yarn-rm-log.js > [YARN UI2] No Logs were found in "YARN Daemon Logs" page > > > Key: YARN-10233 > URL: https://issues.apache.org/jira/browse/YARN-10233 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Blocker > Fix For: 3.3.0 > > Attachments: YARN-10233.001.patch, > YARN_UI2_Tools_Daemon_Logs_Page_Fixed.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10234) FS-CS converter: don't enable auto-create queue property for root
[ https://issues.apache.org/jira/browse/YARN-10234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083826#comment-17083826 ] Hudson commented on YARN-10234: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18145 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18145/]) YARN-10234. FS-CS converter: don't enable auto-create queue property for (snemeth: rev 55fcbcb5c2a096f98f273fda52ae25ecaa1d8bb6) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java > FS-CS converter: don't enable auto-create queue property for root > - > > Key: YARN-10234 > URL: https://issues.apache.org/jira/browse/YARN-10234 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Fix For: 3.3.0, 3.4.0 > > Attachments: YARN-10234-001.patch, YARN-10234-002.patch > > > The auto-create-child-queue property should not be enabled for root, > otherwise it creates an exception inside capacity scheduler. > {noformat} > 2020-04-14 09:48:54,117 INFO org.apache.hadoop.ha.ActiveStandbyElector: > Trying to re-establish ZK session > 2020-04-14 09:48:54,117 ERROR > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Received > RMFatalEvent of type TRANSITION_TO_ACTIVE_FAILED, caused by failure to > refresh configuration settings: org.apache.hadoop.ha.ServiceFailedException: > RefreshAll operation failed > at > org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:772) > at > org.apache.hadoop.yarn.server.resourcemanager.AdminService.transitionToActive(AdminService.java:307) > at > org.apache.hadoop.yarn.server.resourcemanager.ActiveStandbyElectorBasedElectorService.becomeActive(ActiveStandbyElectorBasedElectorService.java:144) > at > org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:896) > at > org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:476) > at > org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:636) > at > org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) > Caused by: java.io.IOException: Failed to re-init queues : null > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:467) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:489) > at > org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:430) > at > org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshAll(AdminService.java:761) > ... 6 more > Caused by: java.lang.ClassCastException > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10219) YARN service placement constraints is broken
[ https://issues.apache.org/jira/browse/YARN-10219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17082936#comment-17082936 ] Hudson commented on YARN-10219: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18140 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18140/]) YARN-10219. Fix YARN Native Service Placement Constraints with Node (pjoseph: rev c791b0e90e0d9c7cb05d162d605e0679942bcbfb) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestYarnNativeServices.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/utils/TestServiceApiUtil.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/Component.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/ServiceApiUtil.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/YarnServiceAPI.md > YARN service placement constraints is broken > > > Key: YARN-10219 > URL: https://issues.apache.org/jira/browse/YARN-10219 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.0, 3.2.0, 3.1.1, 3.1.2, 3.3.0, 3.2.1, 3.1.3 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Blocker > Attachments: YARN-10219.001.patch, YARN-10219.002.patch, > YARN-10219.003.patch, YARN-10219.004.patch, YARN-10219.005.patch > > > YARN service placement constraint does not work with node label nor node > attributes. Example of placement constraints: > {code} > "placement_policy": { > "constraints": [ > { > "type": "AFFINITY", > "scope": "NODE", > "node_attributes": { > "label":["genfile"] > }, > "target_tags": [ > "ping" > ] > } > ] > }, > {code} > Node attribute added: > {code} ./bin/yarn nodeattributes -add "host-3.example.com:label=genfile" > {code} > Scheduling activities shows: > {code} Node does not match partition or placement constraints, > unsatisfied PC expression="in,node,ping", target-type=ALLOCATION_TAG > > 1 > host-3.example.com:45454{code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10212) Create separate configuration for max global AM attempts
[ https://issues.apache.org/jira/browse/YARN-10212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17079664#comment-17079664 ] Hudson commented on YARN-10212: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18135 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18135/]) YARN-10212. Create separate configuration for max global AM attempts. (jhung: rev 23481ad378de7f8e95eabefbd102825f757714b8) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java > Create separate configuration for max global AM attempts > > > Key: YARN-10212 > URL: https://issues.apache.org/jira/browse/YARN-10212 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Bilwa S T >Priority: Major > Fix For: 3.3.0, 3.2.2, 3.1.4, 2.10.1, 3.4.0 > > Attachments: YARN-10212.001.patch, YARN-10212.002.patch, > YARN-10212.003.patch, YARN-10212.004.patch > > > Right now user's default max AM attempts is set to the same as global max AM > attempts: > {noformat} > int globalMaxAppAttempts = conf.getInt(YarnConfiguration.RM_AM_MAX_ATTEMPTS, > YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS); {noformat} > If we want to increase global max AM attempts, it will also increase the > default. So we should create a separate global AM max attempts config to > separate the two. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9995) Code cleanup in TestSchedConfCLI
[ https://issues.apache.org/jira/browse/YARN-9995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17079378#comment-17079378 ] Hudson commented on YARN-9995: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18132 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18132/]) YARN-9995. Code cleanup in TestSchedConfCLI. Contributed by Bilwa S T. (snemeth: rev c04a484aedb09bdac7484908be1b8fcd1add2e30) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestSchedConfCLI.java > Code cleanup in TestSchedConfCLI > > > Key: YARN-9995 > URL: https://issues.apache.org/jira/browse/YARN-9995 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Bilwa S T >Priority: Minor > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-9995.001.patch, YARN-9995.002.patch, > YARN-9995.003.patch, YARN-9995.004.patch > > > Some tests are too verbose: > - add / delete / remove queues testcases: Creating SchedConfUpdateInfo > instances could be simplified with a helper method or something like that. > - Some fields can be converted to local variables: sysOutStream, sysOut, > sysErr, csConf > - Any additional cleanup -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10226) NPE in Capacity Scheduler while using %primary_group queue mapping
[ https://issues.apache.org/jira/browse/YARN-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17079350#comment-17079350 ] Hudson commented on YARN-10226: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18131 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18131/]) YARN-10226. NPE in Capacity Scheduler while using %primary_group queue (sunilg: rev 6fb29d0f1710cbef021e58846eb5c67a8d0b1f87) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java > NPE in Capacity Scheduler while using %primary_group queue mapping > -- > > Key: YARN-10226 > URL: https://issues.apache.org/jira/browse/YARN-10226 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Fix For: 3.3.0, 3.4.0 > > Attachments: YARN-10226-001.patch > > > If we use the following queue mapping: > {{u:%user:%primary_group}} > then we get a NPE inside ResourceManager: > {noformat} > 2020-04-06 11:59:13,883 ERROR resourcemanager.ResourceManager > (ResourceManager.java:serviceStart(881)) - Failed to load/recover state > java.lang.NullPointerException > at > java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.getQueue(CapacitySchedulerQueueManager.java:138) > at > org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.getContextForPrimaryGroup(UserGroupMappingPlacementRule.java:163) > at > org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.getPlacementForUser(UserGroupMappingPlacementRule.java:118) > at > org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule.getPlacementForApp(UserGroupMappingPlacementRule.java:227) > at > org.apache.hadoop.yarn.server.resourcemanager.placement.PlacementManager.placeApplication(PlacementManager.java:67) > at > org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.placeApplication(RMAppManager.java:827) > at > org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:378) > at > org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recoverApplication(RMAppManager.java:367) > at > org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.recover(RMAppManager.java:594) > ... > {noformat} > We to check if parent queue is null in > {{UserGroupMappingPlacementRule.getContextForPrimaryGroup()}}. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5277) When localizers fail due to resource timestamps being out, provide more diagnostics
[ https://issues.apache.org/jira/browse/YARN-5277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17079181#comment-17079181 ] Hudson commented on YARN-5277: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18129 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18129/]) YARN-5277. When localizers fail due to resource timestamps being out, (snemeth: rev 8e4517f251e9f12917f2f457831664905015612f) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestFSDownload.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java > When localizers fail due to resource timestamps being out, provide more > diagnostics > --- > > Key: YARN-5277 > URL: https://issues.apache.org/jira/browse/YARN-5277 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.8.0 >Reporter: Steve Loughran >Assignee: Siddharth Ahuja >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-5277.001.patch, YARN-5277.002.patch > > > When an NM fails a resource D/L as the timestamps are wrong, there's not much > info, just two long values. > It would be good to also include the local time values, *and the current wall > time*. These are the things people need to know when trying to work out what > went wrong -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10063) Usage output of container-executor binary needs to include --http/--https argument
[ https://issues.apache.org/jira/browse/YARN-10063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077739#comment-17077739 ] Hudson commented on YARN-10063: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18126 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18126/]) YARN-10063. Add container-executor arguments --http/--https to usage. (wilfreds: rev 2214005c0f11955b2c50c4d2d4bd14947dd797ba) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c > Usage output of container-executor binary needs to include --http/--https > argument > -- > > Key: YARN-10063 > URL: https://issues.apache.org/jira/browse/YARN-10063 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Siddharth Ahuja >Assignee: Siddharth Ahuja >Priority: Minor > Attachments: YARN-10063.001.patch, YARN-10063.002.patch, > YARN-10063.003.patch, YARN-10063.004.patch > > > YARN-8448/YARN-6586 seems to have introduced a new option - "\--http" > (default) and "\--https" that is possible to be passed in to the > container-executor binary, see : > https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c#L564 > and > https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c#L521 > however, the usage output seems to have missed this: > https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c#L74 > Raising this jira to improve this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10207) CLOSE_WAIT socket connection leaks during rendering of (corrupted) aggregated logs on the JobHistoryServer Web UI
[ https://issues.apache.org/jira/browse/YARN-10207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077332#comment-17077332 ] Hudson commented on YARN-10207: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18124 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18124/]) YARN-10207. CLOSE_WAIT socket connection leaks during rendering of (snemeth: rev bffb43b00e14a23d96f08b5a5df01e7f760b11ed) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java > CLOSE_WAIT socket connection leaks during rendering of (corrupted) aggregated > logs on the JobHistoryServer Web UI > - > > Key: YARN-10207 > URL: https://issues.apache.org/jira/browse/YARN-10207 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Siddharth Ahuja >Assignee: Siddharth Ahuja >Priority: Major > Fix For: 3.3.0, 3.4.0 > > Attachments: YARN-10207.001.patch, YARN-10207.002.patch, > YARN-10207.003.patch, YARN-10207.004.patch, YARN-10207.branch-3.2.001.patch > > > File descriptor leaks are observed coming from the JobHistoryServer process > while it tries to render a "corrupted" aggregated log on the JHS Web UI. > Issue reproduced using the following steps: > # Ran a sample Hadoop MR Pi job, it had the id - > application_1582676649923_0026. > # Copied an aggregated log file from HDFS to local FS: > {code} > hdfs dfs -get > /tmp/logs/systest/logs/application_1582676649923_0026/_8041 > {code} > # Updated the TFile metadata at the bottom of this file with some junk to > corrupt the file : > *Before:* > {code} > > ^@^GVERSION*(^@&container_1582676649923_0026_01_03^F^Dnone^A^Pª5²ª5²^C^Qdata:BCFile.index^Dnoneª5þ^M^M^Pdata:TFile.index^Dnoneª5È66^Odata:TFile.meta^Dnoneª5Â^F^F^@^@^@^@^@^B6^K^@^A^@^@Ñ^QÓh<91>µ×¶9ßA@<92>ºáP > {code} > *After:* > {code} > > ^@^GVERSION*(^@&container_1582676649923_0026_01_03^F^Dnone^A^Pª5²ª5²^C^Qdata:BCFile.index^Dnoneª5þ^M^M^Pdata:TFile.index^Dnoneª5È66^Odata:TFile.meta^Dnoneª5Â^F^F^@^@^@^@^@^B6^K^@^A^@^@Ñ^QÓh<91>µ×¶9ßA@<92>ºáPblah > {code} > Notice "blah" (junk) added at the very end. > # Remove the existing aggregated log file that will need to be replaced by > our modified copy from step 3 (as otherwise HDFS will prevent it from placing > the file with the same name as it already exists): > {code} > hdfs dfs -rm -r -f > /tmp/logs/systest/logs/application_1582676649923_0026/_8041 > {code} > # Upload the corrupted aggregated file back to HDFS: > {code} > hdfs dfs -put _8041 > /tmp/logs/systest/logs/application_1582676649923_0026 > {code} > # Visit HistoryServer Web UI > # Click on job_1582676649923_0026 > # Click on "logs" link against the AM (assuming the AM ran on nm_hostname) > # Review the JHS logs, following exception will be seen: > {code} > 2020-03-24 20:03:48,484 ERROR org.apache.hadoop.yarn.webapp.View: Error > getting logs for job_1582676649923_0026 > java.io.IOException: Not a valid BCFile. > at > org.apache.hadoop.io.file.tfile.BCFile$Magic.readAndVerify(BCFile.java:927) > at > org.apache.hadoop.io.file.tfile.BCFile$Reader.(BCFile.java:628) > at > org.apache.hadoop.io.file.tfile.TFile$Reader.(TFile.java:804) > at > org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat$LogReader.(AggregatedLogFormat.java:588) > at > org.apache.hadoop.yarn.logaggregation.filecontroller.tfile.TFileAggregatedLogsBlock.render(TFileAggregatedLogsBlock.java:111) > at > org.apache.hadoop.yarn.logaggregation.filecontroller.tfile.LogAggregationTFileController.renderAggregatedLogsBlock(LogAggregationTFileController.java:341) > at > org.apache.hadoop.yarn.webapp.log.AggregatedLogsBlock.render(AggregatedLogsBlock.java:117) > at > org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:69) > at > org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:79) > at org.apache.hadoop.yarn.webapp.View.render(View.java:235) > at > org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:49) > at > org.apache.hadoop.yarn.webapp.hamlet2.HamletImpl$EImp._v(HamletImpl.java:117) > at > org.apache.hadoop.yarn.webapp.hamlet2.Hamlet$TD.__(Hamlet.java:848) > at > org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:71) >
[jira] [Commented] (YARN-10001) Add explanation of unimplemented methods in InMemoryConfigurationStore
[ https://issues.apache.org/jira/browse/YARN-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077270#comment-17077270 ] Hudson commented on YARN-10001: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18123 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18123/]) YARN-10001. Add explanation of unimplemented methods in (snemeth: rev 45362a9f4cbe512ee4cd6b7f65aa47d59fee612e) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/InMemoryConfigurationStore.java > Add explanation of unimplemented methods in InMemoryConfigurationStore > -- > > Key: YARN-10001 > URL: https://issues.apache.org/jira/browse/YARN-10001 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Siddharth Ahuja >Priority: Major > Fix For: 3.3.0, 3.4.0 > > Attachments: YARN-10001.001.patch, YARN-10001.002.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10120) In Federation Router Nodes/Applications/About pages throws 500 exception when https is enabled
[ https://issues.apache.org/jira/browse/YARN-10120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17074379#comment-17074379 ] Hudson commented on YARN-10120: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18114 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18114/]) YARN-10120. Amendment fix for Java Doc. (pjoseph: rev 7a39da3537f09b13246a27667199d629b5e62a5b) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServiceClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/DefaultRequestInterceptorREST.java > In Federation Router Nodes/Applications/About pages throws 500 exception when > https is enabled > -- > > Key: YARN-10120 > URL: https://issues.apache.org/jira/browse/YARN-10120 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Reporter: Sushanta Sen >Assignee: Bilwa S T >Priority: Critical > Fix For: 3.4.0 > > Attachments: YARN-10120-YARN-7402.patch, > YARN-10120-addendum-01.patch, YARN-10120.001.patch, YARN-10120.002.patch > > > In Federation Router Nodes/Applications/About pages throws 500 exception when > https is enabled. > yarn.router.webapp.https.address =router ip:8091 > {noformat} > 2020-02-07 16:38:49,990 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error > handling URI: /cluster/apps > java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:166) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at > com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287) > at > com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277) > at > com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182) > at > com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) > at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829) > at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) > at > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) > at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) > at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) > at > com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) > at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) > at > org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644) > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) > at > org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1622) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) > at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:583) > at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) > at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionH
[jira] [Commented] (YARN-10120) In Federation Router Nodes/Applications/About pages throws 500 exception when https is enabled
[ https://issues.apache.org/jira/browse/YARN-10120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17073710#comment-17073710 ] Hudson commented on YARN-10120: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18112 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18112/]) YARN-10120. Https Support in Router WebServiceClient. (pjoseph: rev 25361b077bd34606259dac5a00c41faddd2dfc7d) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/AboutBlock.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/DefaultRequestInterceptorREST.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/RouterWebServiceUtil.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/AppsBlock.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/pom.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/NodesBlock.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/WebServiceClient.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/webapp/TestWebServiceClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/Router.java > In Federation Router Nodes/Applications/About pages throws 500 exception when > https is enabled > -- > > Key: YARN-10120 > URL: https://issues.apache.org/jira/browse/YARN-10120 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Reporter: Sushanta Sen >Assignee: Bilwa S T >Priority: Critical > Fix For: 3.3.0 > > Attachments: YARN-10120.001.patch, YARN-10120.002.patch > > > In Federation Router Nodes/Applications/About pages throws 500 exception when > https is enabled. > yarn.router.webapp.https.address =router ip:8091 > {noformat} > 2020-02-07 16:38:49,990 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error > handling URI: /cluster/apps > java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:166) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at > com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:287) > at > com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:277) > at > com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:182) > at > com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) > at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:941) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:875) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829) > at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) > at > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:119) > at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:133) > at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:130) > at > com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:203) > at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:130) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) > at > org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1767) > at > org.apache.hadoop.security.authentication.server.Authenti
[jira] [Commented] (YARN-10202) Fix documentation about NodeAttributes.
[ https://issues.apache.org/jira/browse/YARN-10202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072469#comment-17072469 ] Hudson commented on YARN-10202: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18106 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18106/]) YARN-10202. Fix documentation about NodeAttributes. Contributed by Sen (aajisaka: rev c162648aff68552d87db8a013b850c17fee762c0) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeAttributes.md > Fix documentation about NodeAttributes. > --- > > Key: YARN-10202 > URL: https://issues.apache.org/jira/browse/YARN-10202 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 3.2.1 >Reporter: Sen Zhao >Assignee: Sen Zhao >Priority: Minor > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-10202.001.patch > > > {noformat:title=NodeAttributes.md} > The above SchedulingRequest requests for 1 container on nodes that must > satisfy following constraints: > 1. Node attribute *`rm.yarn.io/python`* doesn't exist on the node or it exist > but its value is not equal to 3 > 2. Node attribute *`rm.yarn.io/java`* must exist on the node and its value is > equal to 1.8 > {noformat} > should be > {noformat} > The above SchedulingRequest requests for 1 container on nodes that must > satisfy following constraints: > 1. Node attribute *`rm.yarn.io/python`* doesn't exist on the node or it exist > but its value is not equal to 3 > 2. Node attribute *`rm.yarn.io/java`* must exist on the node and its value is > equal to 1.8 > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10210) Add a RMFailoverProxyProvider that does DNS resolution on failover
[ https://issues.apache.org/jira/browse/YARN-10210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067778#comment-17067778 ] Hudson commented on YARN-10210: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18095 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18095/]) YARN-10210. Add a RMFailoverProxyProvider that does DNS resolution on (github: rev 50f7f6dfd10031df142cb9105bddf2d0cd74f017) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestNoHaRMFailoverProxyProvider.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/AutoRefreshNoHARMFailoverProxyProvider.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/AutoRefreshRMFailoverProxyProvider.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestRMFailoverProxyProvider.java > Add a RMFailoverProxyProvider that does DNS resolution on failover > -- > > Key: YARN-10210 > URL: https://issues.apache.org/jira/browse/YARN-10210 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.1.2 >Reporter: Roger Liu >Assignee: Roger Liu >Priority: Major > Fix For: 3.3.0 > > > In Kubernetes, the a node may go down and then come back later with a > different IP address. YARN clients which are already running will be unable > to rediscover the node after it comes back up due to caching the original IP > address. This is problematic for cases such as Spark HA on Kubernetes, as the > node containing the resource manager may go down and come back up, meaning > existing node managers must then also be restarted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10043) FairOrderingPolicy Improvements
[ https://issues.apache.org/jira/browse/YARN-10043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17067645#comment-17067645 ] Hudson commented on YARN-10043: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18094 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18094/]) YARN-10043. FairOrderingPolicy Improvements. Contributed by Manikandan R (snemeth: rev 3d5ade1839205db1c4a11f73bd02d847187f48ca) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/SchedulableEntity.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/MockSchedulableEntity.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FairOrderingPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java > FairOrderingPolicy Improvements > --- > > Key: YARN-10043 > URL: https://issues.apache.org/jira/browse/YARN-10043 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-10043.001.patch, YARN-10043.002.patch, > YARN-10043.003.patch, YARN-10043.004.patch > > > FairOrderingPolicy can be improved by using some of the approaches (only > relevant) implemented in FairSharePolicy of FS. This improvement has > significance in FS to CS migration context. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org