[jira] [Commented] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016632#comment-17016632
]
Hudson commented on YARN-10070:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17871 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17871/])
YARN-10070. Fix NPE if no queue mapping defined for proxy user when (pjoseph:
rev a0ff42d7612e744e0bf88aa14078ea3ab6bcce49)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
> NPE if no rule is defined and application-tag-based-placement is enabled
>
>
> Key: YARN-10070
> URL: https://issues.apache.org/jira/browse/YARN-10070
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-10070.001.patch, YARN-10070.002.patch,
> YARN-10070.003.patch
>
>
> If there is no rule defined for a user NPE is thrown by the following line.
> {code:java}
> String queue = placementManager
> .placeApplication(context, usernameUsedForPlacement).getQueue();{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016626#comment-17016626
]
Prabhu Joseph commented on YARN-10070:
--
Thanks [~kmarton] for the patch. Thanks [~adam.antal] for the review.
Have committed the [^YARN-10070.003.patch] (after fixing the checkstyle issue)
to trunk. Will resolve the Jira.
> NPE if no rule is defined and application-tag-based-placement is enabled
>
>
> Key: YARN-10070
> URL: https://issues.apache.org/jira/browse/YARN-10070
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-10070.001.patch, YARN-10070.002.patch,
> YARN-10070.003.patch
>
>
> If there is no rule defined for a user NPE is thrown by the following line.
> {code:java}
> String queue = placementManager
> .placeApplication(context, usernameUsedForPlacement).getQueue();{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9743) [JDK11] TestTimelineWebServices.testContextFactory fails
[
https://issues.apache.org/jira/browse/YARN-9743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016600#comment-17016600
]
Akira Ajisaka commented on YARN-9743:
-
Thanks [~kmarton] for providing the patch. Some comments:
* jaxb-core and activation dependency seem not to be required.
* I think jaxb-impl should be runtime scope instead of compile.
* In Apache Hadoop, the version of the dependencies are managed by the
dependencyManagement section of hadoop-project module. Would you move the
version to the dependencyManagement section?
> [JDK11] TestTimelineWebServices.testContextFactory fails
>
>
> Key: YARN-9743
> URL: https://issues.apache.org/jira/browse/YARN-9743
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineservice
>Affects Versions: 3.2.0
>Reporter: Adam Antal
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-9743.001.patch, YARN-9743.002.patch
>
>
> Tested on OpenJDK 11.0.2 on a Mac.
> Stack trace:
> {noformat}
> [ERROR] Tests run: 29, Failures: 0, Errors: 3, Skipped: 0, Time elapsed:
> 36.016 s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices
> [ERROR]
> testContextFactory(org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices)
> Time elapsed: 1.031 s <<< ERROR!
> java.lang.ClassNotFoundException: com.sun.xml.internal.bind.v2.ContextFactory
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:315)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.newContext(ContextFactory.java:85)
> at
> org.apache.hadoop.yarn.server.applicationhistoryservice.webapp.ContextFactory.createContext(ContextFactory.java:112)
> at
> org.apache.hadoop.yarn.server.timeline.webapp.TestTimelineWebServices.testContextFactory(TestTimelineWebServices.java:1039)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (YARN-9970) Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping
[
https://issues.apache.org/jira/browse/YARN-9970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016580#comment-17016580
]
Hadoop QA commented on YARN-9970:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
49s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
8s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 91 unchanged - 7 fixed = 92 total (was 98) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 14s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 26s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisherForV2 |
| |
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:0f25cbbb251 |
| JIRA Issue | YARN-9970 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12991070/YARN-9970-branch-3.2.011.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 50d8ae7fe1e8 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / 45b23fd |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_232 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Updated] (YARN-10087) ATS possible NPE on REST API when data is missing
[
https://issues.apache.org/jira/browse/YARN-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wilfred Spiegelenburg updated YARN-10087:
-
Attachment: ats_stack.txt
> ATS possible NPE on REST API when data is missing
> -
>
> Key: YARN-10087
> URL: https://issues.apache.org/jira/browse/YARN-10087
> Project: Hadoop YARN
> Issue Type: Bug
> Components: ATSv2
>Reporter: Wilfred Spiegelenburg
>Priority: Major
> Attachments: ats_stack.txt
>
>
> If the data stored by the ATS is not complete REST calls to the ATS can
> return a NPE instead of results.
> {{{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException"}}}
> The issue shows up when the ATS was down for a short period and in that time
> new applications were started. This causes certain parts of the application
> data to be missing in the ATS store. In most cases this is not a problem and
> data will be returned but when you start filtering data the filtering fails
> throwing the NPE.
> In this case the request was for:
> {{http://:8188/ws/v1/applicationhistory/apps?user=hive'}}
> If certain pieces of data are missing the ATS should not even consider
> returning that data, filtered or not. We should not display partial or
> incomplete data.
> In case of the missing user information ACL checks cannot be correctly
> performed and we could see more issues.
> A similar issue was fixed in YARN-7118 where the queue details were missing.
> It just _skips_ the app to prevent the NPE but that is not the correct thing
> when the user is missing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10087) ATS possible NPE on REST API when data is missing
[
https://issues.apache.org/jira/browse/YARN-10087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016530#comment-17016530
]
Wilfred Spiegelenburg commented on YARN-10087:
--
Attached the NPE as logged in the ATS logs.
The logs point to these lines of code in the release that was running:
{code:java}
188 if (userQuery != null && !userQuery.isEmpty()) {
189 if (!appReport.getUser().equals(userQuery)) {
190 continue;
191 }
192 } {code}
> ATS possible NPE on REST API when data is missing
> -
>
> Key: YARN-10087
> URL: https://issues.apache.org/jira/browse/YARN-10087
> Project: Hadoop YARN
> Issue Type: Bug
> Components: ATSv2
>Reporter: Wilfred Spiegelenburg
>Priority: Major
> Attachments: ats_stack.txt
>
>
> If the data stored by the ATS is not complete REST calls to the ATS can
> return a NPE instead of results.
> {{{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException"}}}
> The issue shows up when the ATS was down for a short period and in that time
> new applications were started. This causes certain parts of the application
> data to be missing in the ATS store. In most cases this is not a problem and
> data will be returned but when you start filtering data the filtering fails
> throwing the NPE.
> In this case the request was for:
> {{http://:8188/ws/v1/applicationhistory/apps?user=hive'}}
> If certain pieces of data are missing the ATS should not even consider
> returning that data, filtered or not. We should not display partial or
> incomplete data.
> In case of the missing user information ACL checks cannot be correctly
> performed and we could see more issues.
> A similar issue was fixed in YARN-7118 where the queue details were missing.
> It just _skips_ the app to prevent the NPE but that is not the correct thing
> when the user is missing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9970) Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping
[ https://issues.apache.org/jira/browse/YARN-9970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-9970: --- Attachment: YARN-9970-branch-3.2.011.patch > Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping > - > > Key: YARN-9970 > URL: https://issues.apache.org/jira/browse/YARN-9970 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9970-branch-3.2.010.patch, > YARN-9970-branch-3.2.011.patch, YARN-9970.001.patch, YARN-9970.002.patch, > YARN-9970.003.patch, YARN-9970.004.patch, YARN-9970.005.patch, > YARN-9970.006.patch, YARN-9970.007.patch, YARN-9970.008.patch, > YARN-9970.009.patch > > > Scope of this Jira is to refactor > TestUserGroupMappingPlacementRule#verifyQueueMapping and QueueMapping class > as discussed in > https://issues.apache.org/jira/browse/YARN-9865?focusedCommentId=16971482=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16971482 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9970) Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping
[ https://issues.apache.org/jira/browse/YARN-9970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016521#comment-17016521 ] Manikandan R commented on YARN-9970: Fixed some checkstyle issues. Junit failures doesn't seem to be related to this patch. > Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping > - > > Key: YARN-9970 > URL: https://issues.apache.org/jira/browse/YARN-9970 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9970-branch-3.2.010.patch, YARN-9970.001.patch, > YARN-9970.002.patch, YARN-9970.003.patch, YARN-9970.004.patch, > YARN-9970.005.patch, YARN-9970.006.patch, YARN-9970.007.patch, > YARN-9970.008.patch, YARN-9970.009.patch > > > Scope of this Jira is to refactor > TestUserGroupMappingPlacementRule#verifyQueueMapping and QueueMapping class > as discussed in > https://issues.apache.org/jira/browse/YARN-9865?focusedCommentId=16971482=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16971482 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9512) [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath fails because of ClassCastException.
[
https://issues.apache.org/jira/browse/YARN-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016516#comment-17016516
]
Hudson commented on YARN-9512:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17869 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17869/])
YARN-9512. [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath (github:
rev 14c2c3d69d1864889f2d20ec79e2bf8e62e56a17)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestAuxServices.java
> [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath fails because of
> ClassCastException.
> --
>
> Key: YARN-9512
> URL: https://issues.apache.org/jira/browse/YARN-9512
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Siyao Meng
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: 3.3.0
>
>
> Found in maven JDK 11 unit test run. Compiled on JDK 8:
> {code}
> [ERROR]
> testCustomizedAuxServiceClassPath(org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices)
> Time elapsed: 0.019 s <<< ERROR!java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices$ServiceC.getMetaData(TestAuxServices.java:197)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:315)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices.testCustomizedAuxServiceClassPath(TestAuxServices.java:344)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9892) Capacity scheduler: support DRF ordering policy on queue level
[
https://issues.apache.org/jira/browse/YARN-9892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016511#comment-17016511
]
Manikandan R commented on YARN-9892:
[~leftnoteasy] Thanks for sharing your views.
{quote}For the patch uploaded in the Jira, it handles a small area only, which
is sort apps.
{quote}
{quote}To me, we should map both FairScheduler's fair/drf to fair in
CapacityScheduler.
{quote}
True. YARN-10043 and YARN-10049 has been created to bridge these gaps. If we
can consolidate all these three JIRA's, can we say it is feature complete? Have
been syncing with [~sunilg] offline as well on this. As you said, still there
could be gaps which can be addressed.
> Capacity scheduler: support DRF ordering policy on queue level
> --
>
> Key: YARN-9892
> URL: https://issues.apache.org/jira/browse/YARN-9892
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-9892-003.patch, YARN-9892.001.patch,
> YARN-9892.002.patch
>
>
> Capacity scheduler does not support DRF (Dominant Resource Fairness) ordering
> policy on queue level. Only "fifo" and "fair" are accepted for
> {{yarn.scheduler.capacity..ordering-policy}}.
> DRF can only be used globally if
> {{yarn.scheduler.capacity.resource-calculator}} is set to
> DominantResourceCalculator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9512) [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath fails because of ClassCastException.
[
https://issues.apache.org/jira/browse/YARN-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-9512:
Summary: [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath fails
because of ClassCastException. (was: [JDK11]
TestAuxServices#testCustomizedAuxServiceClassPath ClassCastException: class
jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
java.net.URLClassLoader)
> [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath fails because of
> ClassCastException.
> --
>
> Key: YARN-9512
> URL: https://issues.apache.org/jira/browse/YARN-9512
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Siyao Meng
>Assignee: Akira Ajisaka
>Priority: Major
>
> Found in maven JDK 11 unit test run. Compiled on JDK 8:
> {code}
> [ERROR]
> testCustomizedAuxServiceClassPath(org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices)
> Time elapsed: 0.019 s <<< ERROR!java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices$ServiceC.getMetaData(TestAuxServices.java:197)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:315)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices.testCustomizedAuxServiceClassPath(TestAuxServices.java:344)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10087) ATS possible NPE on REST API when data is missing
Wilfred Spiegelenburg created YARN-10087:
Summary: ATS possible NPE on REST API when data is missing
Key: YARN-10087
URL: https://issues.apache.org/jira/browse/YARN-10087
Project: Hadoop YARN
Issue Type: Bug
Components: ATSv2
Reporter: Wilfred Spiegelenburg
If the data stored by the ATS is not complete REST calls to the ATS can return
a NPE instead of results.
{{{"exception":"NullPointerException","javaClassName":"java.lang.NullPointerException"}}}
The issue shows up when the ATS was down for a short period and in that time
new applications were started. This causes certain parts of the application
data to be missing in the ATS store. In most cases this is not a problem and
data will be returned but when you start filtering data the filtering fails
throwing the NPE.
In this case the request was for:
{{http://:8188/ws/v1/applicationhistory/apps?user=hive'}}
If certain pieces of data are missing the ATS should not even consider
returning that data, filtered or not. We should not display partial or
incomplete data.
In case of the missing user information ACL checks cannot be correctly
performed and we could see more issues.
A similar issue was fixed in YARN-7118 where the queue details were missing. It
just _skips_ the app to prevent the NPE but that is not the correct thing when
the user is missing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10074) Update netty to 4.1.42Final in yarn-csi
[ https://issues.apache.org/jira/browse/YARN-10074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016247#comment-17016247 ] Wei-Chiu Chuang commented on YARN-10074: It's a pretty trivial change so here it is. > Update netty to 4.1.42Final in yarn-csi > --- > > Key: YARN-10074 > URL: https://issues.apache.org/jira/browse/YARN-10074 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > Looks like HADOOP-16643 is not complete. > https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml#L32 > yarn-csi depends on netty-all 4.1.27Final > [~leosun08] would you be interested in providing another patch to update it > here? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10074) Update netty to 4.1.42Final in yarn-csi
[ https://issues.apache.org/jira/browse/YARN-10074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reassigned YARN-10074: -- Assignee: Wei-Chiu Chuang > Update netty to 4.1.42Final in yarn-csi > --- > > Key: YARN-10074 > URL: https://issues.apache.org/jira/browse/YARN-10074 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > Looks like HADOOP-16643 is not complete. > https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-csi/pom.xml#L32 > yarn-csi depends on netty-all 4.1.27Final > [~leosun08] would you be interested in providing another patch to update it > here? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10086) AWS AssumedRoleCredentialProvider needs ExternalId add
Jon Hartlaub created YARN-10086:
---
Summary: AWS AssumedRoleCredentialProvider needs ExternalId add
Key: YARN-10086
URL: https://issues.apache.org/jira/browse/YARN-10086
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 3.2.1
Reporter: Jon Hartlaub
AWS has added a security feature to the assume-role function in the form of the
"ExternalId" key in the AWS Java SDK
{{STSAssumeRoleSessionCredentialsProvider.Builder}} class. To support this
security feature, the hadoop aws {{AssumedRoleCredentialProvider}} needs a
patch to include this value from the configuration as well as an added Constant
to the {{org.apache.hadoop.fs.s3a.Constants}} file.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9970) Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping
[
https://issues.apache.org/jira/browse/YARN-9970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016202#comment-17016202
]
Hadoop QA commented on YARN-9970:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 7m
7s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} branch-3.2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
47s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} branch-3.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} branch-3.2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 3 new + 91 unchanged - 7 fixed = 94 total (was 98) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 43s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisherForV2 |
| |
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:0f25cbbb251 |
| JIRA Issue | YARN-9970 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12991014/YARN-9970-branch-3.2.010.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux bc6c03bf5d2f 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-3.2 / 1f464e2 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_232 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (YARN-9892) Capacity scheduler: support DRF ordering policy on queue level
[
https://issues.apache.org/jira/browse/YARN-9892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016173#comment-17016173
]
Peter Bacsko commented on YARN-9892:
[~leftnoteasy] thanks for sharing your thoughts - I admit I don't really have a
huge experience in this area, to me this looked like a logical extension, but
if there are such differences in the implementation (eg global calculator vs
local calculator) then we have to think about it.
> Capacity scheduler: support DRF ordering policy on queue level
> --
>
> Key: YARN-9892
> URL: https://issues.apache.org/jira/browse/YARN-9892
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-9892-003.patch, YARN-9892.001.patch,
> YARN-9892.002.patch
>
>
> Capacity scheduler does not support DRF (Dominant Resource Fairness) ordering
> policy on queue level. Only "fifo" and "fair" are accepted for
> {{yarn.scheduler.capacity..ordering-policy}}.
> DRF can only be used globally if
> {{yarn.scheduler.capacity.resource-calculator}} is set to
> DominantResourceCalculator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9892) Capacity scheduler: support DRF ordering policy on queue level
[
https://issues.apache.org/jira/browse/YARN-9892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016160#comment-17016160
]
Wangda Tan commented on YARN-9892:
--
Spent a bit time to look at the code, not fully dig into all details, but I
think there's significant different between FS and CS to handle resource
calculator:
In CS, resource calculator is initialized and used across all the logic.
In FS, there's a separate SchedulingPolicy, which is a wrapper of resource
calculator. It covers areas like compute shares, sort apps/queues, compute
headrooms, etc.
For the patch uploaded in the Jira, it handles a small area only, which is sort
apps.
I want to hear thoughts from [~wilfreds] that why the drf for different queues
is a P0 feature, to me DRF is just an natural extension of Fair (which
considers multiple resource types). I still think we should optimize the logic
and knobs to the users. Cluster admin has to re-tune lots of queue capacities
and SLAs after conversion, we cannot maintain all the different behaviors.
And since there's still a global configuration of ResourceCalculator, which is
used by
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils#getNormalizedResource
when the request came in to the scheduler. I'm a bit confused that
SchedulingPolicy could conflict with the global ResourceCalculator.
> Capacity scheduler: support DRF ordering policy on queue level
> --
>
> Key: YARN-9892
> URL: https://issues.apache.org/jira/browse/YARN-9892
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-9892-003.patch, YARN-9892.001.patch,
> YARN-9892.002.patch
>
>
> Capacity scheduler does not support DRF (Dominant Resource Fairness) ordering
> policy on queue level. Only "fifo" and "fair" are accepted for
> {{yarn.scheduler.capacity..ordering-policy}}.
> DRF can only be used globally if
> {{yarn.scheduler.capacity.resource-calculator}} is set to
> DominantResourceCalculator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9872) DecommissioningNodesWatcher#update blocks the heartbeat processing
[
https://issues.apache.org/jira/browse/YARN-9872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016157#comment-17016157
]
Hadoop QA commented on YARN-9872:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
21s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 22s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
26s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}101m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| | Nullcheck of context at line 140 of value previously dereferenced in
org.apache.hadoop.yarn.server.resourcemanager.DecommissioningNodesWatcher.update(RMNode,
NodeStatus) At DecommissioningNodesWatcher.java:140 of value previously
dereferenced in
org.apache.hadoop.yarn.server.resourcemanager.DecommissioningNodesWatcher.update(RMNode,
NodeStatus) At DecommissioningNodesWatcher.java:[line 140] |
| Failed junit tests |
hadoop.yarn.server.resourcemanager.resourcetracker.TestRMNMRPCResponseId |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
|
| | hadoop.yarn.server.resourcemanager.TestNodeBlacklistingOnAMFailures |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestSchedulingRequestContainerAllocationAsync
|
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations |
| |
[jira] [Commented] (YARN-9892) Capacity scheduler: support DRF ordering policy on queue level
[
https://issues.apache.org/jira/browse/YARN-9892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016151#comment-17016151
]
Wangda Tan commented on YARN-9892:
--
[~pbacsko],
My concern is adding DRF only in application ordering policy is not meaningful,
All the resource requests of apps are normalized differently by resource
calculator implementations (default or drf):
org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator#normalize
When a global resource calculator is configured as "default", we will simply
ignore vcores in all calculation, including queue, nodes, apps, so what does
DRF mean when in that scenario?
To me, we should map both FairScheduler's fair/drf to fair in
CapacityScheduler. Because in CapacityScheduler it do app/queue sorting based
on global resource calculator, which could be fair (mem only, when default
calculator configured) or drf (all resource types, when DRC configured).
> Capacity scheduler: support DRF ordering policy on queue level
> --
>
> Key: YARN-9892
> URL: https://issues.apache.org/jira/browse/YARN-9892
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-9892-003.patch, YARN-9892.001.patch,
> YARN-9892.002.patch
>
>
> Capacity scheduler does not support DRF (Dominant Resource Fairness) ordering
> policy on queue level. Only "fifo" and "fair" are accepted for
> {{yarn.scheduler.capacity..ordering-policy}}.
> DRF can only be used globally if
> {{yarn.scheduler.capacity.resource-calculator}} is set to
> DominantResourceCalculator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9892) Capacity scheduler: support DRF ordering policy on queue level
[
https://issues.apache.org/jira/browse/YARN-9892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016143#comment-17016143
]
Peter Bacsko commented on YARN-9892:
[~leftnoteasy] Fair Scheduler supports all three scheduling policies (fifo,
fair, drf) on queue level. This is not the case in CS and this ticket was
created to address this. I was thinking that we want to close as many feature
gaps as possible, no? We also had a meeting with [~wilfreds] last week and he
identified this as a P0.
Do you have any specific concerns?
> Capacity scheduler: support DRF ordering policy on queue level
> --
>
> Key: YARN-9892
> URL: https://issues.apache.org/jira/browse/YARN-9892
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-9892-003.patch, YARN-9892.001.patch,
> YARN-9892.002.patch
>
>
> Capacity scheduler does not support DRF (Dominant Resource Fairness) ordering
> policy on queue level. Only "fifo" and "fair" are accepted for
> {{yarn.scheduler.capacity..ordering-policy}}.
> DRF can only be used globally if
> {{yarn.scheduler.capacity.resource-calculator}} is set to
> DominantResourceCalculator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[
https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016140#comment-17016140
]
Hadoop QA commented on YARN-10022:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
34s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 21 new + 130 unchanged - 9 fixed = 151 total (was 139)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 9s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}145m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | YARN-10022 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12990992/YARN-10022.WIP2.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux c1061527e5db 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2aa065d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_232 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/25396/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
[jira] [Commented] (YARN-9879) Allow multiple leaf queues with the same name in CS
[
https://issues.apache.org/jira/browse/YARN-9879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016121#comment-17016121
]
Wangda Tan commented on YARN-9879:
--
[~wilfreds], I agree with,
{quote}The behaviour inside the scheduler must all be based on the full queue
paths anyway.
{quote}
I also agree that we need to carefully think about queue mapping and queue
path. I would suggest moving queue mapping related changes to a different Jira
to avoid putting two big patches together. (If it already considered both
scenario we can keep it here).
> Allow multiple leaf queues with the same name in CS
> ---
>
> Key: YARN-9879
> URL: https://issues.apache.org/jira/browse/YARN-9879
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: DesignDoc_v1.pdf
>
>
> Currently the leaf queue's name must be unique regardless of its position in
> the queue hierarchy.
> Design doc and first proposal is being made, I'll attach it as soon as it's
> done.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9970) Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping
[ https://issues.apache.org/jira/browse/YARN-9970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikandan R updated YARN-9970: --- Attachment: YARN-9970-branch-3.2.010.patch > Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping > - > > Key: YARN-9970 > URL: https://issues.apache.org/jira/browse/YARN-9970 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9970-branch-3.2.010.patch, YARN-9970.001.patch, > YARN-9970.002.patch, YARN-9970.003.patch, YARN-9970.004.patch, > YARN-9970.005.patch, YARN-9970.006.patch, YARN-9970.007.patch, > YARN-9970.008.patch, YARN-9970.009.patch > > > Scope of this Jira is to refactor > TestUserGroupMappingPlacementRule#verifyQueueMapping and QueueMapping class > as discussed in > https://issues.apache.org/jira/browse/YARN-9865?focusedCommentId=16971482=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16971482 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9970) Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping
[ https://issues.apache.org/jira/browse/YARN-9970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016114#comment-17016114 ] Manikandan R commented on YARN-9970: [~pbacsko] Thanks. Attached patch for branch-3.2. > Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping > - > > Key: YARN-9970 > URL: https://issues.apache.org/jira/browse/YARN-9970 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9970-branch-3.2.010.patch, YARN-9970.001.patch, > YARN-9970.002.patch, YARN-9970.003.patch, YARN-9970.004.patch, > YARN-9970.005.patch, YARN-9970.006.patch, YARN-9970.007.patch, > YARN-9970.008.patch, YARN-9970.009.patch > > > Scope of this Jira is to refactor > TestUserGroupMappingPlacementRule#verifyQueueMapping and QueueMapping class > as discussed in > https://issues.apache.org/jira/browse/YARN-9865?focusedCommentId=16971482=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16971482 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9892) Capacity scheduler: support DRF ordering policy on queue level
[
https://issues.apache.org/jira/browse/YARN-9892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016112#comment-17016112
]
Wangda Tan commented on YARN-9892:
--
[~pbacsko], [~maniraj...@gmail.com] , thanks for working on this.
Sorry for chiming in late, I have several questions:
1) What is the use case of allowing app's DRF sort only? (While other
computations are using memory only). I saw it is under FS -> CS conversion
tool, but is it a feature we really want to support?
2) In CS, IIRC when DRF is disabled, all the resource accounting will only look
at memory. For example, a queue which set 100 vcore max limit will not be
respected, asking 100 vcore is the same as asking 0 vore. In that case, I don't
think look at DRF of apps is meaningful.
cc: [~sunilg] , [~wilfreds] to add some thoughts here.
> Capacity scheduler: support DRF ordering policy on queue level
> --
>
> Key: YARN-9892
> URL: https://issues.apache.org/jira/browse/YARN-9892
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-9892-003.patch, YARN-9892.001.patch,
> YARN-9892.002.patch
>
>
> Capacity scheduler does not support DRF (Dominant Resource Fairness) ordering
> policy on queue level. Only "fifo" and "fair" are accepted for
> {{yarn.scheduler.capacity..ordering-policy}}.
> DRF can only be used globally if
> {{yarn.scheduler.capacity.resource-calculator}} is set to
> DominantResourceCalculator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016091#comment-17016091
]
Hadoop QA commented on YARN-10070:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 85 unchanged - 0 fixed = 86 total (was 85) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 87m
48s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | YARN-10070 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12990981/YARN-10070.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f3fd8a388da8 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2aa065d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_232 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/25394/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25394/testReport/ |
| Max. process+thread count | 811 (vs. ulimit of 5500) |
| modules | C:
[jira] [Updated] (YARN-9872) DecommissioningNodesWatcher#update blocks the heartbeat processing
[ https://issues.apache.org/jira/browse/YARN-9872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilwa S T updated YARN-9872: Attachment: YARN-9872.001.patch > DecommissioningNodesWatcher#update blocks the heartbeat processing > -- > > Key: YARN-9872 > URL: https://issues.apache.org/jira/browse/YARN-9872 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin Chundatt >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-9872.001.patch > > > ResourceTrackerService handlers gettting blocked due to the synchronisation > at DecommissioningNodesWatcher#update -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9872) DecommissioningNodesWatcher#update blocks the heartbeat processing
[ https://issues.apache.org/jira/browse/YARN-9872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilwa S T updated YARN-9872: Attachment: (was: YARN-9872.001.patch) > DecommissioningNodesWatcher#update blocks the heartbeat processing > -- > > Key: YARN-9872 > URL: https://issues.apache.org/jira/browse/YARN-9872 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin Chundatt >Assignee: Bilwa S T >Priority: Major > > ResourceTrackerService handlers gettting blocked due to the synchronisation > at DecommissioningNodesWatcher#update -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8951) Defining default queue placement rule in allocations file with create="false" throws an NPE
[
https://issues.apache.org/jira/browse/YARN-8951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016036#comment-17016036
]
Szilard Nemeth commented on YARN-8951:
--
As per our offline discussion with [~wilfreds], YARN-8967 is most probably
fixed this bug as well.
It's relatively easy to validate it: I will run the attached testcase and check
the results. If it's green, we can close this jira.
> Defining default queue placement rule in allocations file with create="false"
> throws an NPE
> ---
>
> Key: YARN-8951
> URL: https://issues.apache.org/jira/browse/YARN-8951
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: default-placement-rule-with-create-false.patch
>
>
> If the default queue placement rule is defined with {{create="false"}} and a
> scheduling request is created for queue {{"root.default"}}, then
> {{FairScheduler#assignToQueue}} throws an NPE, while trying to construct an
> error message in the catch block of {{IllegalStateException}}, relying on the
> fact that the {{rmApp}} is not null but it is.
> Example of such a config file:
> {code:java}
>
>
>
> 1024mb,0vcores
>
>
>
>
>
> {code}
> This is suspicious, as there are some null checks for {{rmApp}} in the same
> method.
> Not sure if this is a special case for the tests or it is reproducable in a
> cluster, this needs further investigation.
> In any case, it's not good that we try to dereference the {{rmApp}} that is
> null.
> On the other hand, I'm not sure if the default queue placement rule with
> {{create="false"}} makes sense at all. Looking at the documentation
> ([https://hadoop.apache.org/docs/r3.1.0/hadoop-yarn/hadoop-yarn-site/FairScheduler.html):]
> {quote}default: the app is placed into the queue specified in the ‘queue’
> attribute of the default rule. *If ‘queue’ attribute is not specified, the
> app is placed into ‘root.default’ queue.*
> A queuePlacementPolicy element: which contains a list of rule elements that
> tell the scheduler how to place incoming apps into queues. Rules are applied
> in the order that they are listed. Rules may take arguments. *All rules
> accept the “create” argument, which indicates whether the rule can create a
> new queue. “Create” defaults to true; if set to false and the rule would
> place the app in a queue that is not configured in the allocations file, we
> continue on to the next rule.* The last rule must be one that can never issue
> a continue
> {quote}
> In this case, the rule has the queue property suppressed so the apps should
> be placed to the {{root.default}} queue (which is an undefined queue
> according to the config file), and create is false, meaning that the queue
> {{root.default}} cannot be created at all.
> *This seems to be a case of an invalid queue configuration file for me.*
> [~jlowe], [~leftnoteasy]: What is your take on this?
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9892) Capacity scheduler: support DRF ordering policy on queue level
[
https://issues.apache.org/jira/browse/YARN-9892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016013#comment-17016013
]
Peter Bacsko commented on YARN-9892:
Ok, remaining checkstyle is a nothingburger.
Test failure is about CS, but seems to be unrelated, I bet it's a flaky one.
[~snemeth] could you check this patch pls?
> Capacity scheduler: support DRF ordering policy on queue level
> --
>
> Key: YARN-9892
> URL: https://issues.apache.org/jira/browse/YARN-9892
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-9892-003.patch, YARN-9892.001.patch,
> YARN-9892.002.patch
>
>
> Capacity scheduler does not support DRF (Dominant Resource Fairness) ordering
> policy on queue level. Only "fifo" and "fair" are accepted for
> {{yarn.scheduler.capacity..ordering-policy}}.
> DRF can only be used globally if
> {{yarn.scheduler.capacity.resource-calculator}} is set to
> DominantResourceCalculator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10022: Attachment: YARN-10022.WIP2.patch > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.WIP.patch, YARN-10022.WIP2.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9892) Capacity scheduler: support DRF ordering policy on queue level
[
https://issues.apache.org/jira/browse/YARN-9892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17016006#comment-17016006
]
Hadoop QA commented on YARN-9892:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 140 unchanged - 0 fixed = 141 total (was 140)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 37s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 31s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.5 Server=19.03.5 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | YARN-9892 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12990970/YARN-9892-003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d71a1a76ceb3 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7f35100 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_232 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/25393/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
[jira] [Commented] (YARN-9462) TestResourceTrackerService.testNodeRemovalGracefully fails sporadically
[
https://issues.apache.org/jira/browse/YARN-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015986#comment-17015986
]
Prabhu Joseph commented on YARN-9462:
-
bq. Only a code review is needed for you?
Yes [~snemeth]. The issue still persists on trunk. The patch fixes the issue.
> TestResourceTrackerService.testNodeRemovalGracefully fails sporadically
> ---
>
> Key: YARN-9462
> URL: https://issues.apache.org/jira/browse/YARN-9462
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager, test
>Affects Versions: 3.2.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Minor
> Attachments:
> TestResourceTrackerService.testNodeRemovalGracefully.txt, YARN-9462-001.patch
>
>
> TestResourceTrackerService.testNodeRemovalGracefully fails sporadically
> {code}
> [ERROR]
> testNodeRemovalGracefully(org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService)
> Time elapsed: 3.385 s <<< FAILURE!
> java.lang.AssertionError: Shutdown nodes should be 0 now expected:<1> but
> was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService.testNodeRemovalUtilDecomToUntracked(TestResourceTrackerService.java:2318)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService.testNodeRemovalUtil(TestResourceTrackerService.java:2280)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService.testNodeRemovalGracefully(TestResourceTrackerService.java:2133)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10083) Provide utility to ask whether an application is in final status
[
https://issues.apache.org/jira/browse/YARN-10083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015963#comment-17015963
]
Adam Antal commented on YARN-10083:
---
Other set of UT failures are unrelated.
> Provide utility to ask whether an application is in final status
>
>
> Key: YARN-10083
> URL: https://issues.apache.org/jira/browse/YARN-10083
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-10083.001.patch, YARN-10083.002.patch,
> YARN-10083.002.patch
>
>
> This code part is severely duplicated across the Hadoop repo:
> {code:java}
> public static boolean isApplicationFinalState(YarnApplicationState
> appState) {
> return appState == YarnApplicationState.FINISHED
> || appState == YarnApplicationState.FAILED
> || appState == YarnApplicationState.KILLED;
> }
> {code}
> This functionality is used heavily by the log aggregation as well, so we may
> do some sanitizing here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015953#comment-17015953
]
Adam Antal commented on YARN-10070:
---
Thank you [~kmarton].
Pending on jenkins, but I'll give a +1 (non-binding) to patch v3.
> NPE if no rule is defined and application-tag-based-placement is enabled
>
>
> Key: YARN-10070
> URL: https://issues.apache.org/jira/browse/YARN-10070
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-10070.001.patch, YARN-10070.002.patch,
> YARN-10070.003.patch
>
>
> If there is no rule defined for a user NPE is thrown by the following line.
> {code:java}
> String queue = placementManager
> .placeApplication(context, usernameUsedForPlacement).getQueue();{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[
https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015939#comment-17015939
]
Hadoop QA commented on YARN-10022:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} YARN-10022 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10022 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12990979/YARN-10022.WIP.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25395/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Create RM Rest API to validate a CapacityScheduler Configuration
>
>
> Key: YARN-10022
> URL: https://issues.apache.org/jira/browse/YARN-10022
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-10022.WIP.patch
>
>
> RMWebService should expose a new api which gets a CapacityScheduler
> Configuration as an input, validates it and returns success / failure.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015937#comment-17015937
]
Kinga Marton commented on YARN-10070:
-
Thank you [~adam.antal] for the review. As we have discussed offline, I have
changed the Javadoc comment for the test cases from "_Test case for when..._"
to "_Test for the case when..._"
> NPE if no rule is defined and application-tag-based-placement is enabled
>
>
> Key: YARN-10070
> URL: https://issues.apache.org/jira/browse/YARN-10070
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-10070.001.patch, YARN-10070.002.patch,
> YARN-10070.003.patch
>
>
> If there is no rule defined for a user NPE is thrown by the following line.
> {code:java}
> String queue = placementManager
> .placeApplication(context, usernameUsedForPlacement).getQueue();{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10070) NPE if no rule is defined and application-tag-based-placement is enabled
[
https://issues.apache.org/jira/browse/YARN-10070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kinga Marton updated YARN-10070:
Attachment: YARN-10070.003.patch
> NPE if no rule is defined and application-tag-based-placement is enabled
>
>
> Key: YARN-10070
> URL: https://issues.apache.org/jira/browse/YARN-10070
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kinga Marton
>Assignee: Kinga Marton
>Priority: Major
> Attachments: YARN-10070.001.patch, YARN-10070.002.patch,
> YARN-10070.003.patch
>
>
> If there is no rule defined for a user NPE is thrown by the following line.
> {code:java}
> String queue = placementManager
> .placeApplication(context, usernameUsedForPlacement).getQueue();{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015922#comment-17015922 ] Kinga Marton commented on YARN-10022: - The attached patch is a Work in progress one, because I have to cover the changes with some test cases, and also I expect to have some checkstyle issues and some failing tests. Some comments related to the patch: * we need an API to decide whether a capacity scheduler configuration change is valid or not ad return the new merged configuration if it is a valid one. * my first idea was to extract all the verification steps fro the actual code and write a clear verification part. Unfortunately I couldn't extract all the steps, so at one point I need to create a new instance of CS and call reinitialise queues to be sure that all the validation steps are done. For this part I will open a follow up Jira to try to extract that part as well and remove that hacky part. * I have added a test file, what contains only the signature of the test cases I want to implement, any other test case ideas are very welcomed. :) > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.WIP.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10022: Attachment: (was: YARN-YARN-10022.001.patch) > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.WIP.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10022: Attachment: YARN-10022.WIP.patch > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-10022.WIP.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10022) Create RM Rest API to validate a CapacityScheduler Configuration
[ https://issues.apache.org/jira/browse/YARN-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kinga Marton updated YARN-10022: Attachment: YARN-YARN-10022.001.patch > Create RM Rest API to validate a CapacityScheduler Configuration > > > Key: YARN-10022 > URL: https://issues.apache.org/jira/browse/YARN-10022 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kinga Marton >Assignee: Kinga Marton >Priority: Major > Attachments: YARN-YARN-10022.001.patch > > > RMWebService should expose a new api which gets a CapacityScheduler > Configuration as an input, validates it and returns success / failure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10085) FS-CS converter: remove mixed ordering policy check
Peter Bacsko created YARN-10085:
---
Summary: FS-CS converter: remove mixed ordering policy check
Key: YARN-10085
URL: https://issues.apache.org/jira/browse/YARN-10085
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Peter Bacsko
Assignee: Peter Bacsko
When YARN-9892 gets committed, this part will become unnecessary:
{noformat}
// Validate ordering policy
if (queueConverter.isDrfPolicyUsedOnQueueLevel()) {
if (queueConverter.isFifoOrFairSharePolicyUsed()) {
throw new ConversionException(
"DRF ordering policy cannot be used together with fifo/fair");
} else {
capacitySchedulerConfig.set(
CapacitySchedulerConfiguration.RESOURCE_CALCULATOR_CLASS,
DominantResourceCalculator.class.getCanonicalName());
}
}
{noformat}
We will be able to freely mix fifo/fair/drf, so let's get rid of this strict
check and also rewrite {{FSQueueConverter.emitOrderingPolicy()}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7769) FS QueueManager should not create default queue at init
[ https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015895#comment-17015895 ] Szilard Nemeth commented on YARN-7769: -- Hi [~wilfreds], Can I take this over? > FS QueueManager should not create default queue at init > --- > > Key: YARN-7769 > URL: https://issues.apache.org/jira/browse/YARN-7769 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Wilfred Spiegelenburg >Assignee: Wilfred Spiegelenburg >Priority: Major > > Currently the FairScheduler QueueManager automatically creates the default > queue. However the default queue does not need to exist. We have two possible > cases which we should handle: > * Based on the placement rule "Default" the name for the default queue might > not be default and it should be created with a different name > * There might not be a "Default" placement rule at all which removes the need > to create the queue. > We should leave the creation of the default queue to the point in time that > we can assess if it is needed or not. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7913) Improve error handling when application recovery fails with exception
[ https://issues.apache.org/jira/browse/YARN-7913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015884#comment-17015884 ] Peter Bacsko commented on YARN-7913: +1 (non-binding). Patch looks good to me with all the explanation comments. > Improve error handling when application recovery fails with exception > - > > Key: YARN-7913 > URL: https://issues.apache.org/jira/browse/YARN-7913 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.0.0 >Reporter: Gergo Repas >Assignee: Wilfred Spiegelenburg >Priority: Major > Attachments: YARN-7913.000.poc.patch, YARN-7913.001.patch, > YARN-7913.002.patch, YARN-7913.003.patch > > > There are edge cases when the application recovery fails with an exception. > Example failure scenario: > * setup: a queue is a leaf queue in the primary RM's config and the same > queue is a parent queue in the secondary RM's config. > * When failover happens with this setup, the recovery will fail for > applications on this queue, and an APP_REJECTED event will be dispatched to > the async dispatcher. On the same thread (that handles the recovery), a > NullPointerException is thrown when the applicationAttempt is tried to be > recovered > (https://github.com/apache/hadoop/blob/55066cc53dc22b68f9ca55a0029741d6c846be0a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java#L494). > I don't see a good way to avoid the NPE in this scenario, because when the > NPE occurs the APP_REJECTED has not been processed yet, and we don't know > that the application recovery failed. > Currently the first exception will abort the recovery, and if there are X > applications, there will be ~X passive -> active RM transition attempts - the > passive -> active RM transition will only succeed when the last APP_REJECTED > event is processed on the async dispatcher thread. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10082) FS-CS converter: disable terminal placement rule checking
[
https://issues.apache.org/jira/browse/YARN-10082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015880#comment-17015880
]
Hudson commented on YARN-10082:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17866 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17866/])
YARN-10082. FS-CS converter: disable terminal placement rule checking.
(snemeth: rev 2aa065d98f36527d7769c9c58a923a706036391d)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/ConversionOptions.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalidplacementrules.xml
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigArgumentHandler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSConfigToCSConfigConverter.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigArgumentHandler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementPolicy.java
> FS-CS converter: disable terminal placement rule checking
> -
>
> Key: YARN-10082
> URL: https://issues.apache.org/jira/browse/YARN-10082
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Fix For: 3.3.0
>
> Attachments: YARN-10082-001.patch, YARN-10082-002.patch
>
>
> Before YARN-8967, {{QueuePlacementRule}} class had a method called
> {{isTerminal()}}. However, sometimes this method was hard-coded to return
> false, accepting such configurations as:
> {noformat}
>
>
>
>
>
>
>
> {noformat}
> It's because {{NestedUserQueue.isTerminal()}} always returns {{false}}.
> This changed after YARN-8967. Now, this configuration is not accepted because
> {{QueuePlacementPolicy.fromXml()}} calculates a list of terminal rules
> differently:
> https://github.com/apache/hadoop/blob/5257f50abb71905ef3068fd45541d00ce9e8f355/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementPolicy.java#L176-L183
> In order to migrate existing configuration that were created before
> YARN-8967, we need a new switch (at least in migration mode) in FS to turn
> off this validation, otherwise the tool will not be able to migrate these
> configs and the following exception will be thrown:
> {noformat}
> ~$ ./yarn fs2cs -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp
> WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
> 20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Output
> directory for yarn-site.xml and capacity-scheduler.xml is: /tmp
> 20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Conversion
> rules file is not defined, using default conversion config!
> 20/01/13 05:48:21 INFO converter.FSConfigToCSConfigConverter: Using
> explicitly defined fair-scheduler.xml
> WARNING: This feature is experimental and not intended for production use!
> 20/01/13 05:48:21 INFO conf.Configuration: resource-types.xml not found
> 20/01/13 05:48:21 INFO resource.ResourceUtils: Unable to find
> 'resource-types.xml'.
> 20/01/13 05:48:21 INFO security.YarnAuthorizationProvider:
> org.apache.hadoop.yarn.security.ConfiguredYarnAuthorizer is instantiated.
> 20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Minimum allocation =
>
> 20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Maximum allocation =
>
[jira] [Updated] (YARN-10082) FS-CS converter: disable terminal placement rule checking
[
https://issues.apache.org/jira/browse/YARN-10082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-10082:
--
Fix Version/s: 3.3.0
> FS-CS converter: disable terminal placement rule checking
> -
>
> Key: YARN-10082
> URL: https://issues.apache.org/jira/browse/YARN-10082
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Fix For: 3.3.0
>
> Attachments: YARN-10082-001.patch, YARN-10082-002.patch
>
>
> Before YARN-8967, {{QueuePlacementRule}} class had a method called
> {{isTerminal()}}. However, sometimes this method was hard-coded to return
> false, accepting such configurations as:
> {noformat}
>
>
>
>
>
>
>
> {noformat}
> It's because {{NestedUserQueue.isTerminal()}} always returns {{false}}.
> This changed after YARN-8967. Now, this configuration is not accepted because
> {{QueuePlacementPolicy.fromXml()}} calculates a list of terminal rules
> differently:
> https://github.com/apache/hadoop/blob/5257f50abb71905ef3068fd45541d00ce9e8f355/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementPolicy.java#L176-L183
> In order to migrate existing configuration that were created before
> YARN-8967, we need a new switch (at least in migration mode) in FS to turn
> off this validation, otherwise the tool will not be able to migrate these
> configs and the following exception will be thrown:
> {noformat}
> ~$ ./yarn fs2cs -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp
> WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
> 20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Output
> directory for yarn-site.xml and capacity-scheduler.xml is: /tmp
> 20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Conversion
> rules file is not defined, using default conversion config!
> 20/01/13 05:48:21 INFO converter.FSConfigToCSConfigConverter: Using
> explicitly defined fair-scheduler.xml
> WARNING: This feature is experimental and not intended for production use!
> 20/01/13 05:48:21 INFO conf.Configuration: resource-types.xml not found
> 20/01/13 05:48:21 INFO resource.ResourceUtils: Unable to find
> 'resource-types.xml'.
> 20/01/13 05:48:21 INFO security.YarnAuthorizationProvider:
> org.apache.hadoop.yarn.security.ConfiguredYarnAuthorizer is instantiated.
> 20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Minimum allocation =
>
> 20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Maximum allocation =
>
> 20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
> 20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
> 20/01/13 05:48:21 INFO fair.AllocationFileLoaderService: Loading allocation
> file file:/tmp/fair-scheduler.xml
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
> 20/01/13 05:48:22 INFO service.AbstractService: Service
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
> failed in state INITED
> java.io.IOException: Failed to initialize FairScheduler
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1438)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1479)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:206)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:101)
> at
>
[jira] [Commented] (YARN-10082) FS-CS converter: disable terminal placement rule checking
[
https://issues.apache.org/jira/browse/YARN-10082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015871#comment-17015871
]
Szilard Nemeth commented on YARN-10082:
---
Committed to trunk.
Thanks [~pbacsko] for your contribution!
> FS-CS converter: disable terminal placement rule checking
> -
>
> Key: YARN-10082
> URL: https://issues.apache.org/jira/browse/YARN-10082
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: YARN-10082-001.patch, YARN-10082-002.patch
>
>
> Before YARN-8967, {{QueuePlacementRule}} class had a method called
> {{isTerminal()}}. However, sometimes this method was hard-coded to return
> false, accepting such configurations as:
> {noformat}
>
>
>
>
>
>
>
> {noformat}
> It's because {{NestedUserQueue.isTerminal()}} always returns {{false}}.
> This changed after YARN-8967. Now, this configuration is not accepted because
> {{QueuePlacementPolicy.fromXml()}} calculates a list of terminal rules
> differently:
> https://github.com/apache/hadoop/blob/5257f50abb71905ef3068fd45541d00ce9e8f355/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementPolicy.java#L176-L183
> In order to migrate existing configuration that were created before
> YARN-8967, we need a new switch (at least in migration mode) in FS to turn
> off this validation, otherwise the tool will not be able to migrate these
> configs and the following exception will be thrown:
> {noformat}
> ~$ ./yarn fs2cs -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp
> WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
> 20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Output
> directory for yarn-site.xml and capacity-scheduler.xml is: /tmp
> 20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Conversion
> rules file is not defined, using default conversion config!
> 20/01/13 05:48:21 INFO converter.FSConfigToCSConfigConverter: Using
> explicitly defined fair-scheduler.xml
> WARNING: This feature is experimental and not intended for production use!
> 20/01/13 05:48:21 INFO conf.Configuration: resource-types.xml not found
> 20/01/13 05:48:21 INFO resource.ResourceUtils: Unable to find
> 'resource-types.xml'.
> 20/01/13 05:48:21 INFO security.YarnAuthorizationProvider:
> org.apache.hadoop.yarn.security.ConfiguredYarnAuthorizer is instantiated.
> 20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Minimum allocation =
>
> 20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Maximum allocation =
>
> 20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
> 20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
> 20/01/13 05:48:21 INFO fair.AllocationFileLoaderService: Loading allocation
> file file:/tmp/fair-scheduler.xml
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
> 20/01/13 05:48:22 INFO service.AbstractService: Service
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
> failed in state INITED
> java.io.IOException: Failed to initialize FairScheduler
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1438)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1479)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSConfigToCSConfigConverter.convert(FSConfigToCSConfigConverter.java:206)
> at
>
[jira] [Comment Edited] (YARN-10082) FS-CS converter: disable terminal placement rule checking
[
https://issues.apache.org/jira/browse/YARN-10082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015869#comment-17015869
]
Szilard Nemeth edited comment on YARN-10082 at 1/15/20 11:40 AM:
-
Hi [~pbacsko],
Thanks for working on this, this is a good finding!
One nit: FSConfigConverterTestCommons#FS_ALLOC_INVALID_PLACEMENTRULES is not
used, you can remove this constant.
Otherwise, patch looks good to me.
As per our offline discussion and since this is a minor thing, I am removing
this constant now and committing to trunk.
was (Author: snemeth):
Hi [~pbacsko],
Thanks for working on this, this is a good finding!
One nit: FSConfigConverterTestCommons#FS_ALLOC_INVALID_PLACEMENTRULES is not
used, you can remove this constant.
Otherwise, patch looks good to me.
> FS-CS converter: disable terminal placement rule checking
> -
>
> Key: YARN-10082
> URL: https://issues.apache.org/jira/browse/YARN-10082
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: YARN-10082-001.patch, YARN-10082-002.patch
>
>
> Before YARN-8967, {{QueuePlacementRule}} class had a method called
> {{isTerminal()}}. However, sometimes this method was hard-coded to return
> false, accepting such configurations as:
> {noformat}
>
>
>
>
>
>
>
> {noformat}
> It's because {{NestedUserQueue.isTerminal()}} always returns {{false}}.
> This changed after YARN-8967. Now, this configuration is not accepted because
> {{QueuePlacementPolicy.fromXml()}} calculates a list of terminal rules
> differently:
> https://github.com/apache/hadoop/blob/5257f50abb71905ef3068fd45541d00ce9e8f355/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementPolicy.java#L176-L183
> In order to migrate existing configuration that were created before
> YARN-8967, we need a new switch (at least in migration mode) in FS to turn
> off this validation, otherwise the tool will not be able to migrate these
> configs and the following exception will be thrown:
> {noformat}
> ~$ ./yarn fs2cs -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp
> WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
> 20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Output
> directory for yarn-site.xml and capacity-scheduler.xml is: /tmp
> 20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Conversion
> rules file is not defined, using default conversion config!
> 20/01/13 05:48:21 INFO converter.FSConfigToCSConfigConverter: Using
> explicitly defined fair-scheduler.xml
> WARNING: This feature is experimental and not intended for production use!
> 20/01/13 05:48:21 INFO conf.Configuration: resource-types.xml not found
> 20/01/13 05:48:21 INFO resource.ResourceUtils: Unable to find
> 'resource-types.xml'.
> 20/01/13 05:48:21 INFO security.YarnAuthorizationProvider:
> org.apache.hadoop.yarn.security.ConfiguredYarnAuthorizer is instantiated.
> 20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Minimum allocation =
>
> 20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Maximum allocation =
>
> 20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
> 20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
> 20/01/13 05:48:21 INFO fair.AllocationFileLoaderService: Loading allocation
> file file:/tmp/fair-scheduler.xml
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
> 20/01/13 05:48:22 INFO service.AbstractService: Service
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
> failed in state INITED
> java.io.IOException: Failed to initialize FairScheduler
> at
>
[jira] [Commented] (YARN-10082) FS-CS converter: disable terminal placement rule checking
[
https://issues.apache.org/jira/browse/YARN-10082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015869#comment-17015869
]
Szilard Nemeth commented on YARN-10082:
---
Hi [~pbacsko],
Thanks for working on this, this is a good finding!
One nit: FSConfigConverterTestCommons#FS_ALLOC_INVALID_PLACEMENTRULES is not
used, you can remove this constant.
Otherwise, patch looks good to me.
> FS-CS converter: disable terminal placement rule checking
> -
>
> Key: YARN-10082
> URL: https://issues.apache.org/jira/browse/YARN-10082
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Critical
> Attachments: YARN-10082-001.patch, YARN-10082-002.patch
>
>
> Before YARN-8967, {{QueuePlacementRule}} class had a method called
> {{isTerminal()}}. However, sometimes this method was hard-coded to return
> false, accepting such configurations as:
> {noformat}
>
>
>
>
>
>
>
> {noformat}
> It's because {{NestedUserQueue.isTerminal()}} always returns {{false}}.
> This changed after YARN-8967. Now, this configuration is not accepted because
> {{QueuePlacementPolicy.fromXml()}} calculates a list of terminal rules
> differently:
> https://github.com/apache/hadoop/blob/5257f50abb71905ef3068fd45541d00ce9e8f355/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueuePlacementPolicy.java#L176-L183
> In order to migrate existing configuration that were created before
> YARN-8967, we need a new switch (at least in migration mode) in FS to turn
> off this validation, otherwise the tool will not be able to migrate these
> configs and the following exception will be thrown:
> {noformat}
> ~$ ./yarn fs2cs -y /tmp/yarn-site.xml -f /tmp/fair-scheduler.xml -o /tmp
> WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
> 20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Output
> directory for yarn-site.xml and capacity-scheduler.xml is: /tmp
> 20/01/13 05:48:20 INFO converter.FSConfigToCSConfigConverter: Conversion
> rules file is not defined, using default conversion config!
> 20/01/13 05:48:21 INFO converter.FSConfigToCSConfigConverter: Using
> explicitly defined fair-scheduler.xml
> WARNING: This feature is experimental and not intended for production use!
> 20/01/13 05:48:21 INFO conf.Configuration: resource-types.xml not found
> 20/01/13 05:48:21 INFO resource.ResourceUtils: Unable to find
> 'resource-types.xml'.
> 20/01/13 05:48:21 INFO security.YarnAuthorizationProvider:
> org.apache.hadoop.yarn.security.ConfiguredYarnAuthorizer is instantiated.
> 20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Minimum allocation =
>
> 20/01/13 05:48:21 INFO scheduler.AbstractYarnScheduler: Maximum allocation =
>
> 20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
> 20/01/13 05:48:21 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
> 20/01/13 05:48:21 INFO fair.AllocationFileLoaderService: Loading allocation
> file file:/tmp/fair-scheduler.xml
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.SpecifiedPlacementRule
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.UserPlacementRule
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
> 20/01/13 05:48:22 INFO placement.PlacementFactory: Creating PlacementRule
> implementation: class
> org.apache.hadoop.yarn.server.resourcemanager.placement.DefaultPlacementRule
> 20/01/13 05:48:22 INFO service.AbstractService: Service
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler
> failed in state INITED
> java.io.IOException: Failed to initialize FairScheduler
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.initScheduler(FairScheduler.java:1438)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.serviceInit(FairScheduler.java:1479)
> at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> at
>
[jira] [Commented] (YARN-9892) Capacity scheduler: support DRF ordering policy on queue level
[
https://issues.apache.org/jira/browse/YARN-9892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015865#comment-17015865
]
Peter Bacsko commented on YARN-9892:
I uploaded patch v3 to fix checkstyle, whitespace issues and did a minimal code
cleanup. Let's wait for Jenkins.
> Capacity scheduler: support DRF ordering policy on queue level
> --
>
> Key: YARN-9892
> URL: https://issues.apache.org/jira/browse/YARN-9892
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-9892-003.patch, YARN-9892.001.patch,
> YARN-9892.002.patch
>
>
> Capacity scheduler does not support DRF (Dominant Resource Fairness) ordering
> policy on queue level. Only "fifo" and "fair" are accepted for
> {{yarn.scheduler.capacity..ordering-policy}}.
> DRF can only be used globally if
> {{yarn.scheduler.capacity.resource-calculator}} is set to
> DominantResourceCalculator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9892) Capacity scheduler: support DRF ordering policy on queue level
[
https://issues.apache.org/jira/browse/YARN-9892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-9892:
---
Attachment: YARN-9892-003.patch
> Capacity scheduler: support DRF ordering policy on queue level
> --
>
> Key: YARN-9892
> URL: https://issues.apache.org/jira/browse/YARN-9892
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-9892-003.patch, YARN-9892.001.patch,
> YARN-9892.002.patch
>
>
> Capacity scheduler does not support DRF (Dominant Resource Fairness) ordering
> policy on queue level. Only "fifo" and "fair" are accepted for
> {{yarn.scheduler.capacity..ordering-policy}}.
> DRF can only be used globally if
> {{yarn.scheduler.capacity.resource-calculator}} is set to
> DominantResourceCalculator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9512) [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath ClassCastException: class jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class java.net.URLC
[
https://issues.apache.org/jira/browse/YARN-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015834#comment-17015834
]
Szilard Nemeth commented on YARN-9512:
--
Thanks [~aajisaka],
Will check your PR if I have some time
> [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath ClassCastException:
> class jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader
> ---
>
> Key: YARN-9512
> URL: https://issues.apache.org/jira/browse/YARN-9512
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Siyao Meng
>Assignee: Akira Ajisaka
>Priority: Major
>
> Found in maven JDK 11 unit test run. Compiled on JDK 8:
> {code}
> [ERROR]
> testCustomizedAuxServiceClassPath(org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices)
> Time elapsed: 0.019 s <<< ERROR!java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices$ServiceC.getMetaData(TestAuxServices.java:197)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:315)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices.testCustomizedAuxServiceClassPath(TestAuxServices.java:344)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10083) Provide utility to ask whether an application is in final status
[
https://issues.apache.org/jira/browse/YARN-10083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015830#comment-17015830
]
Hadoop QA commented on YARN-10083:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
17s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
20s{color} | {color:green} root: The patch generated 0 new + 132 unchanged - 3
fixed = 132 total (was 135) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 9m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
9s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
47s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
49s{color} | {color:green} hadoop-yarn-server-applicationhistoryservice in the
patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 7s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 33m 18s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
17s{color} | {color:green} hadoop-yarn-server-timeline-pluginstorage in the
patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 2s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 37s{color}
| {color:red} hadoop-dynamometer-infra in the patch failed. {color}
[jira] [Issue Comment Deleted] (YARN-10081) Exception message from ClientRMProxy#getRMAddress is misleading
[
https://issues.apache.org/jira/browse/YARN-10081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravuri Sushma sree updated YARN-10081:
--
Comment: was deleted
(was: Hi [~adam.antal] ,
Good Catch, correcting the error Message will give better insight of the Error
.I have added a patch correcting the same.)
> Exception message from ClientRMProxy#getRMAddress is misleading
> ---
>
> Key: YARN-10081
> URL: https://issues.apache.org/jira/browse/YARN-10081
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Priority: Trivial
> Attachments: YARN-10081.001.patch
>
>
> In {{ClientRMProxy#getRMAddress}} in the else branch we have the following
> piece of code.
> {code:java}
> } else {
> String message = "Unsupported protocol found when creating the proxy " +
> "connection to ResourceManager: " +
> ((protocol != null) ? protocol.getClass().getName() : "null");
> LOG.error(message);
> throw new IllegalStateException(message);
> }
> {code}
> This is wrong, because the protocol variable is of type "Class", so
> {{Class.getClass()}} will be always {{Object}}. It should be
> {{protocol.getName()}}.
> An example of the error message if {{RMProxy}} is misused, and this exception
> is thrown:
> {noformat}
> java.lang.IllegalStateException: Unsupported protocol found when creating the
> proxy connection to ResourceManager: java.lang.Class
> at
> org.apache.hadoop.yarn.client.ClientRMProxy.getRMAddress(ClientRMProxy.java:109)
> at
> org.apache.hadoop.yarn.client.RMProxy.newProxyInstance(RMProxy.java:133)
> ...
> {noformat}
> where obviously not a {{Object.class}} was provided to this function as
> protocol parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-10081) Exception message from ClientRMProxy#getRMAddress is misleading
[
https://issues.apache.org/jira/browse/YARN-10081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravuri Sushma sree updated YARN-10081:
--
Comment: was deleted
(was: Hi [~adam.antal] , Good Catch
I have attached a patch correcting the error message.)
> Exception message from ClientRMProxy#getRMAddress is misleading
> ---
>
> Key: YARN-10081
> URL: https://issues.apache.org/jira/browse/YARN-10081
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Priority: Trivial
> Attachments: YARN-10081.001.patch
>
>
> In {{ClientRMProxy#getRMAddress}} in the else branch we have the following
> piece of code.
> {code:java}
> } else {
> String message = "Unsupported protocol found when creating the proxy " +
> "connection to ResourceManager: " +
> ((protocol != null) ? protocol.getClass().getName() : "null");
> LOG.error(message);
> throw new IllegalStateException(message);
> }
> {code}
> This is wrong, because the protocol variable is of type "Class", so
> {{Class.getClass()}} will be always {{Object}}. It should be
> {{protocol.getName()}}.
> An example of the error message if {{RMProxy}} is misused, and this exception
> is thrown:
> {noformat}
> java.lang.IllegalStateException: Unsupported protocol found when creating the
> proxy connection to ResourceManager: java.lang.Class
> at
> org.apache.hadoop.yarn.client.ClientRMProxy.getRMAddress(ClientRMProxy.java:109)
> at
> org.apache.hadoop.yarn.client.RMProxy.newProxyInstance(RMProxy.java:133)
> ...
> {noformat}
> where obviously not a {{Object.class}} was provided to this function as
> protocol parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10081) Exception message from ClientRMProxy#getRMAddress is misleading
[
https://issues.apache.org/jira/browse/YARN-10081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015827#comment-17015827
]
Ravuri Sushma sree commented on YARN-10081:
---
Hi [~adam.antal] ,
Good Catch, correcting this Error Message will give better insight . I have
corrected the same and added a patch.
> Exception message from ClientRMProxy#getRMAddress is misleading
> ---
>
> Key: YARN-10081
> URL: https://issues.apache.org/jira/browse/YARN-10081
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Priority: Trivial
> Attachments: YARN-10081.001.patch
>
>
> In {{ClientRMProxy#getRMAddress}} in the else branch we have the following
> piece of code.
> {code:java}
> } else {
> String message = "Unsupported protocol found when creating the proxy " +
> "connection to ResourceManager: " +
> ((protocol != null) ? protocol.getClass().getName() : "null");
> LOG.error(message);
> throw new IllegalStateException(message);
> }
> {code}
> This is wrong, because the protocol variable is of type "Class", so
> {{Class.getClass()}} will be always {{Object}}. It should be
> {{protocol.getName()}}.
> An example of the error message if {{RMProxy}} is misused, and this exception
> is thrown:
> {noformat}
> java.lang.IllegalStateException: Unsupported protocol found when creating the
> proxy connection to ResourceManager: java.lang.Class
> at
> org.apache.hadoop.yarn.client.ClientRMProxy.getRMAddress(ClientRMProxy.java:109)
> at
> org.apache.hadoop.yarn.client.RMProxy.newProxyInstance(RMProxy.java:133)
> ...
> {noformat}
> where obviously not a {{Object.class}} was provided to this function as
> protocol parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10081) Exception message from ClientRMProxy#getRMAddress is misleading
[
https://issues.apache.org/jira/browse/YARN-10081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015825#comment-17015825
]
Ravuri Sushma sree commented on YARN-10081:
---
Hi [~adam.antal] ,
Good Catch, correcting the error Message will give better insight of the Error
.I have added a patch correcting the same.
> Exception message from ClientRMProxy#getRMAddress is misleading
> ---
>
> Key: YARN-10081
> URL: https://issues.apache.org/jira/browse/YARN-10081
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Priority: Trivial
> Attachments: YARN-10081.001.patch
>
>
> In {{ClientRMProxy#getRMAddress}} in the else branch we have the following
> piece of code.
> {code:java}
> } else {
> String message = "Unsupported protocol found when creating the proxy " +
> "connection to ResourceManager: " +
> ((protocol != null) ? protocol.getClass().getName() : "null");
> LOG.error(message);
> throw new IllegalStateException(message);
> }
> {code}
> This is wrong, because the protocol variable is of type "Class", so
> {{Class.getClass()}} will be always {{Object}}. It should be
> {{protocol.getName()}}.
> An example of the error message if {{RMProxy}} is misused, and this exception
> is thrown:
> {noformat}
> java.lang.IllegalStateException: Unsupported protocol found when creating the
> proxy connection to ResourceManager: java.lang.Class
> at
> org.apache.hadoop.yarn.client.ClientRMProxy.getRMAddress(ClientRMProxy.java:109)
> at
> org.apache.hadoop.yarn.client.RMProxy.newProxyInstance(RMProxy.java:133)
> ...
> {noformat}
> where obviously not a {{Object.class}} was provided to this function as
> protocol parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10081) Exception message from ClientRMProxy#getRMAddress is misleading
[
https://issues.apache.org/jira/browse/YARN-10081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015819#comment-17015819
]
Ravuri Sushma sree commented on YARN-10081:
---
Hi [~adam.antal] , Good Catch
I have attached a patch correcting the error message.
> Exception message from ClientRMProxy#getRMAddress is misleading
> ---
>
> Key: YARN-10081
> URL: https://issues.apache.org/jira/browse/YARN-10081
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Priority: Trivial
> Attachments: YARN-10081.001.patch
>
>
> In {{ClientRMProxy#getRMAddress}} in the else branch we have the following
> piece of code.
> {code:java}
> } else {
> String message = "Unsupported protocol found when creating the proxy " +
> "connection to ResourceManager: " +
> ((protocol != null) ? protocol.getClass().getName() : "null");
> LOG.error(message);
> throw new IllegalStateException(message);
> }
> {code}
> This is wrong, because the protocol variable is of type "Class", so
> {{Class.getClass()}} will be always {{Object}}. It should be
> {{protocol.getName()}}.
> An example of the error message if {{RMProxy}} is misused, and this exception
> is thrown:
> {noformat}
> java.lang.IllegalStateException: Unsupported protocol found when creating the
> proxy connection to ResourceManager: java.lang.Class
> at
> org.apache.hadoop.yarn.client.ClientRMProxy.getRMAddress(ClientRMProxy.java:109)
> at
> org.apache.hadoop.yarn.client.RMProxy.newProxyInstance(RMProxy.java:133)
> ...
> {noformat}
> where obviously not a {{Object.class}} was provided to this function as
> protocol parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10081) Exception message from ClientRMProxy#getRMAddress is misleading
[
https://issues.apache.org/jira/browse/YARN-10081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravuri Sushma sree updated YARN-10081:
--
Attachment: YARN-10081.001.patch
> Exception message from ClientRMProxy#getRMAddress is misleading
> ---
>
> Key: YARN-10081
> URL: https://issues.apache.org/jira/browse/YARN-10081
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Priority: Trivial
> Attachments: YARN-10081.001.patch
>
>
> In {{ClientRMProxy#getRMAddress}} in the else branch we have the following
> piece of code.
> {code:java}
> } else {
> String message = "Unsupported protocol found when creating the proxy " +
> "connection to ResourceManager: " +
> ((protocol != null) ? protocol.getClass().getName() : "null");
> LOG.error(message);
> throw new IllegalStateException(message);
> }
> {code}
> This is wrong, because the protocol variable is of type "Class", so
> {{Class.getClass()}} will be always {{Object}}. It should be
> {{protocol.getName()}}.
> An example of the error message if {{RMProxy}} is misused, and this exception
> is thrown:
> {noformat}
> java.lang.IllegalStateException: Unsupported protocol found when creating the
> proxy connection to ResourceManager: java.lang.Class
> at
> org.apache.hadoop.yarn.client.ClientRMProxy.getRMAddress(ClientRMProxy.java:109)
> at
> org.apache.hadoop.yarn.client.RMProxy.newProxyInstance(RMProxy.java:133)
> ...
> {noformat}
> where obviously not a {{Object.class}} was provided to this function as
> protocol parameter.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9170) Name Node Format Exception showing
[ https://issues.apache.org/jira/browse/YARN-9170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015797#comment-17015797 ] Ravuri Sushma sree commented on YARN-9170: -- Hi [~RasmiRanjan] , Can you please provide the exception you encountered while formatting the NameNode > Name Node Format Exception showing > -- > > Key: YARN-9170 > URL: https://issues.apache.org/jira/browse/YARN-9170 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 2.6.0 >Reporter: RasmiRanjan Biswal >Priority: Critical > Fix For: 2.6.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10028) Integrate the new abstract log servlet to the JobHistory server
[ https://issues.apache.org/jira/browse/YARN-10028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10028: -- Fix Version/s: 3.2.2 > Integrate the new abstract log servlet to the JobHistory server > --- > > Key: YARN-10028 > URL: https://issues.apache.org/jira/browse/YARN-10028 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0, 3.2.2 > > Attachments: YARN-10028.001.patch, YARN-10028.002.patch, > YARN-10028.003.patch, YARN-10028.branch-3.2.001.patch > > > Currently JHS has already incorporates a log servlet, but it in incapable of > serving REST calls. We can integrate the new common log servlet to the JHS in > order to have a REST interface. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10028) Integrate the new abstract log servlet to the JobHistory server
[ https://issues.apache.org/jira/browse/YARN-10028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015795#comment-17015795 ] Szilard Nemeth commented on YARN-10028: --- Thanks [~adam.antal], committed branch-3.2 patch as well. > Integrate the new abstract log servlet to the JobHistory server > --- > > Key: YARN-10028 > URL: https://issues.apache.org/jira/browse/YARN-10028 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-10028.001.patch, YARN-10028.002.patch, > YARN-10028.003.patch, YARN-10028.branch-3.2.001.patch > > > Currently JHS has already incorporates a log servlet, but it in incapable of > serving REST calls. We can integrate the new common log servlet to the JHS in > order to have a REST interface. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9512) [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath ClassCastException: class jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class java.net.URLC
[
https://issues.apache.org/jira/browse/YARN-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015785#comment-17015785
]
Akira Ajisaka commented on YARN-9512:
-
Thank you [~snemeth]. Created a PR to fix this issue.
> [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath ClassCastException:
> class jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader
> ---
>
> Key: YARN-9512
> URL: https://issues.apache.org/jira/browse/YARN-9512
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Siyao Meng
>Assignee: Akira Ajisaka
>Priority: Major
>
> Found in maven JDK 11 unit test run. Compiled on JDK 8:
> {code}
> [ERROR]
> testCustomizedAuxServiceClassPath(org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices)
> Time elapsed: 0.019 s <<< ERROR!java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices$ServiceC.getMetaData(TestAuxServices.java:197)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:315)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices.testCustomizedAuxServiceClassPath(TestAuxServices.java:344)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9831) NMTokenSecretManagerInRM#createNMToken blocks ApplicationMasterService allocate flow
[ https://issues.apache.org/jira/browse/YARN-9831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilwa S T reassigned YARN-9831: --- Assignee: Bilwa S T > NMTokenSecretManagerInRM#createNMToken blocks ApplicationMasterService > allocate flow > > > Key: YARN-9831 > URL: https://issues.apache.org/jira/browse/YARN-9831 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin Chundatt >Assignee: Bilwa S T >Priority: Critical > > Currently attempt's NMToken cannot be generated independently. > Each attempts allocate flow blocks each other. We should improve the same -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9872) DecommissioningNodesWatcher#update blocks the heartbeat processing
[
https://issues.apache.org/jira/browse/YARN-9872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015749#comment-17015749
]
Hadoop QA commented on YARN-9872:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} YARN-9872 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-9872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12990961/YARN-9872.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/25392/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> DecommissioningNodesWatcher#update blocks the heartbeat processing
> --
>
> Key: YARN-9872
> URL: https://issues.apache.org/jira/browse/YARN-9872
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin Chundatt
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-9872.001.patch
>
>
> ResourceTrackerService handlers gettting blocked due to the synchronisation
> at DecommissioningNodesWatcher#update
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9872) DecommissioningNodesWatcher#update blocks the heartbeat processing
[ https://issues.apache.org/jira/browse/YARN-9872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilwa S T updated YARN-9872: Attachment: YARN-9872.001.patch > DecommissioningNodesWatcher#update blocks the heartbeat processing > -- > > Key: YARN-9872 > URL: https://issues.apache.org/jira/browse/YARN-9872 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin Chundatt >Assignee: Bilwa S T >Priority: Major > Attachments: YARN-9872.001.patch > > > ResourceTrackerService handlers gettting blocked due to the synchronisation > at DecommissioningNodesWatcher#update -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9970) Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping
[ https://issues.apache.org/jira/browse/YARN-9970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015742#comment-17015742 ] Hudson commented on YARN-9970: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #17864 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17864/]) YARN-9970. Refactor (snemeth: rev 7c5cecc3b3c00886a5bc39a9a8cad6ca1088b095) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueMappings.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoCreatedQueueBase.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerQueueMappingFactory.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/QueueMapping.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestPlacementManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java > Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping > - > > Key: YARN-9970 > URL: https://issues.apache.org/jira/browse/YARN-9970 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9970.001.patch, YARN-9970.002.patch, > YARN-9970.003.patch, YARN-9970.004.patch, YARN-9970.005.patch, > YARN-9970.006.patch, YARN-9970.007.patch, YARN-9970.008.patch, > YARN-9970.009.patch > > > Scope of this Jira is to refactor > TestUserGroupMappingPlacementRule#verifyQueueMapping and QueueMapping class > as discussed in > https://issues.apache.org/jira/browse/YARN-9865?focusedCommentId=16971482=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16971482 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9970) Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping
[ https://issues.apache.org/jira/browse/YARN-9970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015738#comment-17015738 ] Szilard Nemeth edited comment on YARN-9970 at 1/15/20 8:58 AM: --- Hi [~maniraj...@gmail.com], Latest patch LGTM, committed to trunk. Thanks [~pbacsko] for the reviews. [~maniraj...@gmail.com] Can you upload a backport patch to branch-3.2? Ofc only if it's not a big pain to backport. was (Author: snemeth): Hi [~maniraj...@gmail.com], Latest patch LGTM, committed to trunk. Thanks [~pbacsko] for the reviews. [~maniraj...@gmail.com] Can you upload a backport patch to branch-3.2? > Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping > - > > Key: YARN-9970 > URL: https://issues.apache.org/jira/browse/YARN-9970 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9970.001.patch, YARN-9970.002.patch, > YARN-9970.003.patch, YARN-9970.004.patch, YARN-9970.005.patch, > YARN-9970.006.patch, YARN-9970.007.patch, YARN-9970.008.patch, > YARN-9970.009.patch > > > Scope of this Jira is to refactor > TestUserGroupMappingPlacementRule#verifyQueueMapping and QueueMapping class > as discussed in > https://issues.apache.org/jira/browse/YARN-9865?focusedCommentId=16971482=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16971482 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9970) Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping
[ https://issues.apache.org/jira/browse/YARN-9970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-9970: - Fix Version/s: 3.3.0 > Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping > - > > Key: YARN-9970 > URL: https://issues.apache.org/jira/browse/YARN-9970 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9970.001.patch, YARN-9970.002.patch, > YARN-9970.003.patch, YARN-9970.004.patch, YARN-9970.005.patch, > YARN-9970.006.patch, YARN-9970.007.patch, YARN-9970.008.patch, > YARN-9970.009.patch > > > Scope of this Jira is to refactor > TestUserGroupMappingPlacementRule#verifyQueueMapping and QueueMapping class > as discussed in > https://issues.apache.org/jira/browse/YARN-9865?focusedCommentId=16971482=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16971482 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9970) Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping
[ https://issues.apache.org/jira/browse/YARN-9970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015738#comment-17015738 ] Szilard Nemeth commented on YARN-9970: -- Hi [~maniraj...@gmail.com], Latest patch LGTM, committed to trunk. Thanks [~pbacsko] for the reviews. [~maniraj...@gmail.com] Can you upload a backport patch to branch-3.2? > Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping > - > > Key: YARN-9970 > URL: https://issues.apache.org/jira/browse/YARN-9970 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Manikandan R >Assignee: Manikandan R >Priority: Major > Attachments: YARN-9970.001.patch, YARN-9970.002.patch, > YARN-9970.003.patch, YARN-9970.004.patch, YARN-9970.005.patch, > YARN-9970.006.patch, YARN-9970.007.patch, YARN-9970.008.patch, > YARN-9970.009.patch > > > Scope of this Jira is to refactor > TestUserGroupMappingPlacementRule#verifyQueueMapping and QueueMapping class > as discussed in > https://issues.apache.org/jira/browse/YARN-9865?focusedCommentId=16971482=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16971482 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10083) Provide utility to ask whether an application is in final status
[
https://issues.apache.org/jira/browse/YARN-10083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015727#comment-17015727
]
Adam Antal commented on YARN-10083:
---
The "unable to create native thread" failures are unrelated to this patch -
this is some jenkins issue, also ASF license warning is of a . Reuploaded patch
v2.
> Provide utility to ask whether an application is in final status
>
>
> Key: YARN-10083
> URL: https://issues.apache.org/jira/browse/YARN-10083
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Adam Antal
>Assignee: Adam Antal
>Priority: Minor
> Attachments: YARN-10083.001.patch, YARN-10083.002.patch,
> YARN-10083.002.patch
>
>
> This code part is severely duplicated across the Hadoop repo:
> {code:java}
> public static boolean isApplicationFinalState(YarnApplicationState
> appState) {
> return appState == YarnApplicationState.FINISHED
> || appState == YarnApplicationState.FAILED
> || appState == YarnApplicationState.KILLED;
> }
> {code}
> This functionality is used heavily by the log aggregation as well, so we may
> do some sanitizing here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9512) [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath ClassCastException: class jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class java.net.URLC
[
https://issues.apache.org/jira/browse/YARN-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015709#comment-17015709
]
Szilard Nemeth commented on YARN-9512:
--
Hi [~aajisaka],
It was assigned to me, planned to check it months ago but did not have time to
work on it.
Feeel free to take this over
> [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath ClassCastException:
> class jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader
> ---
>
> Key: YARN-9512
> URL: https://issues.apache.org/jira/browse/YARN-9512
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Siyao Meng
>Assignee: Akira Ajisaka
>Priority: Major
>
> Found in maven JDK 11 unit test run. Compiled on JDK 8:
> {code}
> [ERROR]
> testCustomizedAuxServiceClassPath(org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices)
> Time elapsed: 0.019 s <<< ERROR!java.lang.ClassCastException: class
> jdk.internal.loader.ClassLoaders$AppClassLoader cannot be cast to class
> java.net.URLClassLoader (jdk.internal.loader.ClassLoaders$AppClassLoader and
> java.net.URLClassLoader are in module java.base of loader 'bootstrap')
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices$ServiceC.getMetaData(TestAuxServices.java:197)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceStart(AuxServices.java:315)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.TestAuxServices.testCustomizedAuxServiceClassPath(TestAuxServices.java:344)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
