[jira] [Commented] (YARN-8732) Add unit tests of min/max allocation for custom resource types in FairScheduler
[ https://issues.apache.org/jira/browse/YARN-8732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637819#comment-16637819 ] Szilard Nemeth commented on YARN-8732: -- Hi [~haibochen]! Unit test failures are not related and I think we can ignore the checkstyle issues in this case. > Add unit tests of min/max allocation for custom resource types in > FairScheduler > --- > > Key: YARN-8732 > URL: https://issues.apache.org/jira/browse/YARN-8732 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Labels: unittest > Attachments: YARN-8732.001.patch, YARN-8732.002.patch, > YARN-8732.003.patch, YARN-8732.004.patch, YARN-8732.005.patch, > YARN-8732.006.patch, YARN-8732.007.patch > > > Create testcase like this, but for FS: > org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService#testValidateRequestCapacityAgainstMinMaxAllocationFor3rdResourceTypes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637795#comment-16637795
]
Rahul Anand commented on YARN-6900:
---
[~elgoiri] [~subru]
I would like to know how one can configure the znode so as to specify the queue
policies. I mean how can I specify the router, amrmpolicy, associated weights
etc in the znode and in which format. I am unable to find this anywhere.
> ZooKeeper based implementation of the FederationStateStore
> --
>
> Key: YARN-6900
> URL: https://issues.apache.org/jira/browse/YARN-6900
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Íñigo Goiri
>Priority: Major
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6900-002.patch, YARN-6900-003.patch,
> YARN-6900-004.patch, YARN-6900-005.patch, YARN-6900-006.patch,
> YARN-6900-007.patch, YARN-6900-008.patch, YARN-6900-009.patch,
> YARN-6900-010.patch, YARN-6900-011.patch, YARN-6900-YARN-2915-000.patch,
> YARN-6900-YARN-2915-001.patch
>
>
> YARN-5408 defines the unified {{FederationStateStore}} API. Currently we only
> support SQL based stores, this JIRA tracks adding a ZooKeeper based
> implementation for simplifying deployment as it's already popularly used for
> {{RMStateStore}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8750) Refactor TestQueueMetrics
[
https://issues.apache.org/jira/browse/YARN-8750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637686#comment-16637686
]
Hadoop QA commented on YARN-8750:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 11 unchanged - 71 fixed = 13 total (was 82)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 72m
3s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}119m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8750 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942333/YARN-8750.006.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux b6d9047905d7 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1dc0adf |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22051/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22051/testReport/ |
| Max. process+thread count | 913 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (YARN-6055) ContainersMonitorImpl need be adjusted when NM resource changed.
[
https://issues.apache.org/jira/browse/YARN-6055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637712#comment-16637712
]
Íñigo Goiri commented on YARN-6055:
---
The comment from [~raviprak] was:
{quote}
Should we also modify the limits set in
[ContainersMonitorImpl.maxPmemAllottedForContainers|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java#L76]
etc. also?
{quote}
I agree on this, {{NodeStatusUpdaterImpl#updateNMResource}} should modify the
{{ContainersMonitorImpl}} values.
It should be fairly easy to check it from TestRMAdminService.
I could give it a try.
> ContainersMonitorImpl need be adjusted when NM resource changed.
>
>
> Key: YARN-6055
> URL: https://issues.apache.org/jira/browse/YARN-6055
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: graceful, nodemanager, scheduler
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Major
>
> Per Ravi's comments in YARN-4832, we need to check some limits in
> containerMonitorImpl to make sure it get updated also when Resource updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8843) updateNodeResource does not support units for memory
[
https://issues.apache.org/jira/browse/YARN-8843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637708#comment-16637708
]
Hadoop QA commented on YARN-8843:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 7s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 46s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
2s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 26m
37s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
56s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8843 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942337/YARN-8843.000.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux f3e7469fea0a 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1dc0adf |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22052/testReport/ |
| Max. process+thread count | 682 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (YARN-8777) Container Executor C binary change to execute interactive docker command
[
https://issues.apache.org/jira/browse/YARN-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637697#comment-16637697
]
Hadoop QA commented on YARN-8777:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
28m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 52s{color} |
{color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 39 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 57s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 4s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 53s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.TestNMProxy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8777 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942341/YARN-8777.005.patch |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit |
| uname | Linux 1164b8f045a6 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1dc0adf |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| cc |
https://builds.apache.org/job/PreCommit-YARN-Build/22053/artifact/out/diff-compile-cc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/22053/artifact/out/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22053/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22053/testReport/ |
| Max. process+thread count | 411 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22053/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Container Executor C binary
[jira] [Commented] (YARN-8840) Add missing cleanupSSLConfig() call for TestTimelineClient test
[
https://issues.apache.org/jira/browse/YARN-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637673#comment-16637673
]
Hudson commented on YARN-8840:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15111 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/15111/])
YARN-8840. Add missing cleanupSSLConfig() call for TestTimelineClient
(aajisaka: rev 39b35036ba47064149003046a7b59feb01575d1e)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestTimelineClient.java
> Add missing cleanupSSLConfig() call for TestTimelineClient test
> ---
>
> Key: YARN-8840
> URL: https://issues.apache.org/jira/browse/YARN-8840
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, timelineclient
>Reporter: Aki Tanaka
>Assignee: Aki Tanaka
>Priority: Minor
> Attachments: YARN-8840.001.patch, YARN-8840.002.patch,
> YARN-8840.003.patch
>
>
> Tests that setup SSLConfigs can leave conf-files lingering unless they are
> cleaned up via {{KeyStoreTestUtil.cleanupSSLConfig}} call. TestTimelineClient
> test is missing this call.
> If the cleanup method is not called explicitly, a modified ssl-client.xml is
> left in {{test-classes}}, might affect subsequent test cases.
>
> There was a similar report in HDFS-11042, but looks that we need to fix
> TestTimelineClient test too.
>
> {code:java}
> $ mvn test -Dtest=TestTimelineClient
> $ find .|grep ssl-client.xml$
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-classes/ssl-client.xml
> $ cat
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-classes/ssl-client.xml
>
> ssl.client.truststore.reload.interval1000falseprogrammatically
> ssl.client.truststore.location/Users/tanaka/work/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-dir/trustKS.jksfalseprogrammatically
> ssl.client.keystore.keypasswordclientPfalseprogrammatically
> ssl.client.keystore.location/Users/tanaka/work/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-dir/clientKS.jksfalseprogrammatically
> ssl.client.truststore.passwordtrustPfalseprogrammatically
> ssl.client.keystore.passwordclientPfalseprogrammatically
> {code}
>
> After applying this patch, the ssl-client.xml is not generated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8840) Add missing cleanupSSLConfig() call for TestTimelineClient test
[
https://issues.apache.org/jira/browse/YARN-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reassigned YARN-8840:
---
Assignee: Aki Tanaka
> Add missing cleanupSSLConfig() call for TestTimelineClient test
> ---
>
> Key: YARN-8840
> URL: https://issues.apache.org/jira/browse/YARN-8840
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, timelineclient
>Reporter: Aki Tanaka
>Assignee: Aki Tanaka
>Priority: Minor
> Attachments: YARN-8840.001.patch, YARN-8840.002.patch,
> YARN-8840.003.patch
>
>
> Tests that setup SSLConfigs can leave conf-files lingering unless they are
> cleaned up via {{KeyStoreTestUtil.cleanupSSLConfig}} call. TestTimelineClient
> test is missing this call.
> If the cleanup method is not called explicitly, a modified ssl-client.xml is
> left in {{test-classes}}, might affect subsequent test cases.
>
> There was a similar report in HDFS-11042, but looks that we need to fix
> TestTimelineClient test too.
>
> {code:java}
> $ mvn test -Dtest=TestTimelineClient
> $ find .|grep ssl-client.xml$
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-classes/ssl-client.xml
> $ cat
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-classes/ssl-client.xml
>
> ssl.client.truststore.reload.interval1000falseprogrammatically
> ssl.client.truststore.location/Users/tanaka/work/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-dir/trustKS.jksfalseprogrammatically
> ssl.client.keystore.keypasswordclientPfalseprogrammatically
> ssl.client.keystore.location/Users/tanaka/work/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-dir/clientKS.jksfalseprogrammatically
> ssl.client.truststore.passwordtrustPfalseprogrammatically
> ssl.client.keystore.passwordclientPfalseprogrammatically
> {code}
>
> After applying this patch, the ssl-client.xml is not generated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8840) Add missing cleanupSSLConfig() call for TestTimelineClient test
[
https://issues.apache.org/jira/browse/YARN-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637660#comment-16637660
]
Akira Ajisaka commented on YARN-8840:
-
+1
> Add missing cleanupSSLConfig() call for TestTimelineClient test
> ---
>
> Key: YARN-8840
> URL: https://issues.apache.org/jira/browse/YARN-8840
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, timelineclient
>Reporter: Aki Tanaka
>Priority: Minor
> Attachments: YARN-8840.001.patch, YARN-8840.002.patch,
> YARN-8840.003.patch
>
>
> Tests that setup SSLConfigs can leave conf-files lingering unless they are
> cleaned up via {{KeyStoreTestUtil.cleanupSSLConfig}} call. TestTimelineClient
> test is missing this call.
> If the cleanup method is not called explicitly, a modified ssl-client.xml is
> left in {{test-classes}}, might affect subsequent test cases.
>
> There was a similar report in HDFS-11042, but looks that we need to fix
> TestTimelineClient test too.
>
> {code:java}
> $ mvn test -Dtest=TestTimelineClient
> $ find .|grep ssl-client.xml$
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-classes/ssl-client.xml
> $ cat
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-classes/ssl-client.xml
>
> ssl.client.truststore.reload.interval1000falseprogrammatically
> ssl.client.truststore.location/Users/tanaka/work/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-dir/trustKS.jksfalseprogrammatically
> ssl.client.keystore.keypasswordclientPfalseprogrammatically
> ssl.client.keystore.location/Users/tanaka/work/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-dir/clientKS.jksfalseprogrammatically
> ssl.client.truststore.passwordtrustPfalseprogrammatically
> ssl.client.keystore.passwordclientPfalseprogrammatically
> {code}
>
> After applying this patch, the ssl-client.xml is not generated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8732) Add unit tests of min/max allocation for custom resource types in FairScheduler
[
https://issues.apache.org/jira/browse/YARN-8732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637646#comment-16637646
]
Hadoop QA commented on YARN-8732:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 3 new + 0 unchanged - 19 fixed = 3 total (was 19) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 40s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 1s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}121m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueManagementDynamicEditPolicy
|
| | hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8732 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942318/YARN-8732.007.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 02f619c7b6bd 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1dc0adf |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22046/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
[jira] [Commented] (YARN-8777) Container Executor C binary change to execute interactive docker command
[ https://issues.apache.org/jira/browse/YARN-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637645#comment-16637645 ] Eric Yang commented on YARN-8777: - Patch 005 rebase code to current trunk. > Container Executor C binary change to execute interactive docker command > > > Key: YARN-8777 > URL: https://issues.apache.org/jira/browse/YARN-8777 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zian Chen >Assignee: Eric Yang >Priority: Major > Labels: Docker > Attachments: YARN-8777.001.patch, YARN-8777.002.patch, > YARN-8777.003.patch, YARN-8777.004.patch, YARN-8777.005.patch > > > Since Container Executor provides Container execution using the native > container-executor binary, we also need to make changes to accept new > “dockerExec” method to invoke the corresponding native function to execute > docker exec command to the running container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8777) Container Executor C binary change to execute interactive docker command
[ https://issues.apache.org/jira/browse/YARN-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8777: Attachment: YARN-8777.005.patch > Container Executor C binary change to execute interactive docker command > > > Key: YARN-8777 > URL: https://issues.apache.org/jira/browse/YARN-8777 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zian Chen >Assignee: Eric Yang >Priority: Major > Labels: Docker > Attachments: YARN-8777.001.patch, YARN-8777.002.patch, > YARN-8777.003.patch, YARN-8777.004.patch, YARN-8777.005.patch > > > Since Container Executor provides Container execution using the native > container-executor binary, we also need to make changes to accept new > “dockerExec” method to invoke the corresponding native function to execute > docker exec command to the running container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8732) Add unit tests of min/max allocation for custom resource types in FairScheduler
[ https://issues.apache.org/jira/browse/YARN-8732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637642#comment-16637642 ] Haibo Chen commented on YARN-8732: -- +1 on the latest patch pending Jenkins. > Add unit tests of min/max allocation for custom resource types in > FairScheduler > --- > > Key: YARN-8732 > URL: https://issues.apache.org/jira/browse/YARN-8732 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Labels: unittest > Attachments: YARN-8732.001.patch, YARN-8732.002.patch, > YARN-8732.003.patch, YARN-8732.004.patch, YARN-8732.005.patch, > YARN-8732.006.patch, YARN-8732.007.patch > > > Create testcase like this, but for FS: > org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService#testValidateRequestCapacityAgainstMinMaxAllocationFor3rdResourceTypes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8843) updateNodeResource does not support units for memory
[
https://issues.apache.org/jira/browse/YARN-8843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637629#comment-16637629
]
Íñigo Goiri commented on YARN-8843:
---
I think this was introduced in YARN-6761.
[~vvasudev], [~templedf], do you guys mind taking a look.
In addition to the test for updateNodeResource, I think we should test
somewhere close to {{Resource}} or {{ResourceInformation}}.
> updateNodeResource does not support units for memory
>
>
> Key: YARN-8843
> URL: https://issues.apache.org/jira/browse/YARN-8843
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Minor
> Attachments: YARN-8843.000.patch
>
>
> When doing -updateNodeResource memory-mb=1Gi, it assigns 1MB.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8843) updateNodeResource does not support units for memory
[ https://issues.apache.org/jira/browse/YARN-8843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-8843: -- Attachment: YARN-8843.000.patch > updateNodeResource does not support units for memory > > > Key: YARN-8843 > URL: https://issues.apache.org/jira/browse/YARN-8843 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Attachments: YARN-8843.000.patch > > > When doing -updateNodeResource memory-mb=1Gi, it assigns 1MB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8843) updateNodeResource does not support units for memory
Íñigo Goiri created YARN-8843: - Summary: updateNodeResource does not support units for memory Key: YARN-8843 URL: https://issues.apache.org/jira/browse/YARN-8843 Project: Hadoop YARN Issue Type: Bug Reporter: Íñigo Goiri Assignee: Íñigo Goiri When doing -updateNodeResource memory-mb=1Gi, it assigns 1MB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7747) YARN UI is broken in the minicluster
[
https://issues.apache.org/jira/browse/YARN-7747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637609#comment-16637609
]
Gera Shegalov commented on YARN-7747:
-
[~ste...@apache.org] we definitely need tests to prevent this kind of
regression in the future. We could make sure that all web/http address keys are
properly reflected in MiniYARNCLuster#getConfig implementation and then probe
all of them through easy-to-validate REST api. RM URI should respond to the
RM-specific REST, and so on and so forth
> YARN UI is broken in the minicluster
> -
>
> Key: YARN-7747
> URL: https://issues.apache.org/jira/browse/YARN-7747
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Major
> Attachments: YARN-7747.001.patch, YARN-7747.002.patch
>
>
> YARN web apps use non-injected instances of GuiceFilter, i.e. instances
> created by Jetty as opposed by Guice itself. This triggers the [call
> path|https://github.com/google/guice/blob/master/extensions/servlet/src/com/google/inject/servlet/GuiceFilter.java#L251]
> where the static field {{pipeline}} is used instead of the instance field
> {{injectedPipeline}}. However, besides GuiceFilter instances created by
> Jetty, each Guice module generates them as well. On the injection call path
> this static variable is updated by each instance. Thus if there are multiple
> modules as it happens to be the case in the minicluster the one loaded last
> ends up defining the filter pipeline for all Jetty instances. In the
> minicluster case this is the nodemanager UI
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8750) Refactor TestQueueMetrics
[
https://issues.apache.org/jira/browse/YARN-8750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637599#comment-16637599
]
Szilard Nemeth commented on YARN-8750:
--
Hi [~haibochen]!
Thanks for another round of review.
Yeah, I was having this idea as well when formulating the first patch.
I updated the code accordingly, every metrics source (queue / user /
parentQueue / parentUser) now has its own checker object, so readers easily can
see the delta.
> Refactor TestQueueMetrics
> -
>
> Key: YARN-8750
> URL: https://issues.apache.org/jira/browse/YARN-8750
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-8750.001.patch, YARN-8750.002.patch,
> YARN-8750.003.patch, YARN-8750.004.patch, YARN-8750.005.patch,
> YARN-8750.006.patch
>
>
> {{TestQueueMetrics#checkApps}} and {{TestQueueMetrics#checkResources}} have 8
> and 14 parameters, respectively.
> It is very hard to read the testcases that are using these methods.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8750) Refactor TestQueueMetrics
[
https://issues.apache.org/jira/browse/YARN-8750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-8750:
-
Attachment: YARN-8750.006.patch
> Refactor TestQueueMetrics
> -
>
> Key: YARN-8750
> URL: https://issues.apache.org/jira/browse/YARN-8750
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-8750.001.patch, YARN-8750.002.patch,
> YARN-8750.003.patch, YARN-8750.004.patch, YARN-8750.005.patch,
> YARN-8750.006.patch
>
>
> {{TestQueueMetrics#checkApps}} and {{TestQueueMetrics#checkResources}} have 8
> and 14 parameters, respectively.
> It is very hard to read the testcases that are using these methods.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7747) YARN UI is broken in the minicluster
[
https://issues.apache.org/jira/browse/YARN-7747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637587#comment-16637587
]
Hadoop QA commented on YARN-7747:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-7747 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7747 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12914476/YARN-7747.002.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22050/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> YARN UI is broken in the minicluster
> -
>
> Key: YARN-7747
> URL: https://issues.apache.org/jira/browse/YARN-7747
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Major
> Attachments: YARN-7747.001.patch, YARN-7747.002.patch
>
>
> YARN web apps use non-injected instances of GuiceFilter, i.e. instances
> created by Jetty as opposed by Guice itself. This triggers the [call
> path|https://github.com/google/guice/blob/master/extensions/servlet/src/com/google/inject/servlet/GuiceFilter.java#L251]
> where the static field {{pipeline}} is used instead of the instance field
> {{injectedPipeline}}. However, besides GuiceFilter instances created by
> Jetty, each Guice module generates them as well. On the injection call path
> this static variable is updated by each instance. Thus if there are multiple
> modules as it happens to be the case in the minicluster the one loaded last
> ends up defining the filter pipeline for all Jetty instances. In the
> minicluster case this is the nodemanager UI
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7747) YARN UI is broken in the minicluster
[
https://issues.apache.org/jira/browse/YARN-7747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637567#comment-16637567
]
Steve Loughran commented on YARN-7747:
--
* is there a way to get that UI URL back? If it's not there already?
* are there tests which look @ this, e.g. issue a GET or two?
> YARN UI is broken in the minicluster
> -
>
> Key: YARN-7747
> URL: https://issues.apache.org/jira/browse/YARN-7747
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Major
> Attachments: YARN-7747.001.patch, YARN-7747.002.patch
>
>
> YARN web apps use non-injected instances of GuiceFilter, i.e. instances
> created by Jetty as opposed by Guice itself. This triggers the [call
> path|https://github.com/google/guice/blob/master/extensions/servlet/src/com/google/inject/servlet/GuiceFilter.java#L251]
> where the static field {{pipeline}} is used instead of the instance field
> {{injectedPipeline}}. However, besides GuiceFilter instances created by
> Jetty, each Guice module generates them as well. On the injection call path
> this static variable is updated by each instance. Thus if there are multiple
> modules as it happens to be the case in the minicluster the one loaded last
> ends up defining the filter pipeline for all Jetty instances. In the
> minicluster case this is the nodemanager UI
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4254) ApplicationAttempt stuck for ever due to UnknowHostexception
[ https://issues.apache.org/jira/browse/YARN-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637565#comment-16637565 ] Jason Lowe commented on YARN-4254: -- Sorry for the delay, was out of the office for a bit and very busy otherwise. Did a quick glance, will try to review more thoroughly later this week. DEFAUTL instead of DEFAULT Unused imports flagged by checkstyle should be fixed. The creation of the InetSocketAddress and InetAddress is unnecessary unless checkIpHostnameRegistration is true. > ApplicationAttempt stuck for ever due to UnknowHostexception > > > Key: YARN-4254 > URL: https://issues.apache.org/jira/browse/YARN-4254 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt >Priority: Major > Attachments: 0001-YARN-4254.patch, Logs.txt, Test.patch, > YARN-4254.002.patch > > > Scenario > === > 1. RM HA and 5 NMs available in cluster and are working fine > 2. Add one more NM to the same cluster but RM /etc/hosts not updated. > 3. Submit application to the same cluster > If Am get allocated to the newly added NM the *application attempt will get > stuck for ever*.User will not get to know why the same happened. > Impact > 1.RM logs gets overloaded with exception > 2.Application gets stuck for ever. > Handling suggestion YARN-261 allows for Fail application attempt . > If we fail the same next attempt could get assigned to another NM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8777) Container Executor C binary change to execute interactive docker command
[
https://issues.apache.org/jira/browse/YARN-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637561#comment-16637561
]
Hadoop QA commented on YARN-8777:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-8777 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-8777 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942316/YARN-8777.004.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22049/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Container Executor C binary change to execute interactive docker command
>
>
> Key: YARN-8777
> URL: https://issues.apache.org/jira/browse/YARN-8777
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zian Chen
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
> Attachments: YARN-8777.001.patch, YARN-8777.002.patch,
> YARN-8777.003.patch, YARN-8777.004.patch
>
>
> Since Container Executor provides Container execution using the native
> container-executor binary, we also need to make changes to accept new
> “dockerExec” method to invoke the corresponding native function to execute
> docker exec command to the running container.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8644) Add more test coverage for RMAppImpl.FinalTransition
[ https://issues.apache.org/jira/browse/YARN-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-8644: - Attachment: YARN-8644.009.patch > Add more test coverage for RMAppImpl.FinalTransition > > > Key: YARN-8644 > URL: https://issues.apache.org/jira/browse/YARN-8644 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Attachments: YARN-8644.001.patch, YARN-8644.002.patch, > YARN-8644.003.patch, YARN-8644.004.patch, YARN-8644.005.patch, > YARN-8644.006.patch, YARN-8644.007.patch, YARN-8644.008.patch, > YARN-8644.009.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8644) Add more test coverage for RMAppImpl.FinalTransition
[ https://issues.apache.org/jira/browse/YARN-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637476#comment-16637476 ] Szilard Nemeth commented on YARN-8644: -- patch009 fixes checkstyle issues. Unit test failures were unrelated. > Add more test coverage for RMAppImpl.FinalTransition > > > Key: YARN-8644 > URL: https://issues.apache.org/jira/browse/YARN-8644 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Attachments: YARN-8644.001.patch, YARN-8644.002.patch, > YARN-8644.003.patch, YARN-8644.004.patch, YARN-8644.005.patch, > YARN-8644.006.patch, YARN-8644.007.patch, YARN-8644.008.patch, > YARN-8644.009.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8732) Add unit tests of min/max allocation for custom resource types in FairScheduler
[ https://issues.apache.org/jira/browse/YARN-8732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637466#comment-16637466 ] Szilard Nemeth commented on YARN-8732: -- Thanks [~haibochen] for suggestions. Renamed the class and fixed the checkstyle issues with patch007 > Add unit tests of min/max allocation for custom resource types in > FairScheduler > --- > > Key: YARN-8732 > URL: https://issues.apache.org/jira/browse/YARN-8732 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Labels: unittest > Attachments: YARN-8732.001.patch, YARN-8732.002.patch, > YARN-8732.003.patch, YARN-8732.004.patch, YARN-8732.005.patch, > YARN-8732.006.patch, YARN-8732.007.patch > > > Create testcase like this, but for FS: > org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService#testValidateRequestCapacityAgainstMinMaxAllocationFor3rdResourceTypes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8732) Add unit tests of min/max allocation for custom resource types in FairScheduler
[ https://issues.apache.org/jira/browse/YARN-8732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-8732: - Attachment: YARN-8732.007.patch > Add unit tests of min/max allocation for custom resource types in > FairScheduler > --- > > Key: YARN-8732 > URL: https://issues.apache.org/jira/browse/YARN-8732 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Labels: unittest > Attachments: YARN-8732.001.patch, YARN-8732.002.patch, > YARN-8732.003.patch, YARN-8732.004.patch, YARN-8732.005.patch, > YARN-8732.006.patch, YARN-8732.007.patch > > > Create testcase like this, but for FS: > org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService#testValidateRequestCapacityAgainstMinMaxAllocationFor3rdResourceTypes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8777) Container Executor C binary change to execute interactive docker command
[ https://issues.apache.org/jira/browse/YARN-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637447#comment-16637447 ] Eric Yang commented on YARN-8777: - Patch 004 added pseudo terminal support for running with docker exec -it flag. > Container Executor C binary change to execute interactive docker command > > > Key: YARN-8777 > URL: https://issues.apache.org/jira/browse/YARN-8777 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zian Chen >Assignee: Eric Yang >Priority: Major > Labels: Docker > Attachments: YARN-8777.001.patch, YARN-8777.002.patch, > YARN-8777.003.patch, YARN-8777.004.patch > > > Since Container Executor provides Container execution using the native > container-executor binary, we also need to make changes to accept new > “dockerExec” method to invoke the corresponding native function to execute > docker exec command to the running container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8777) Container Executor C binary change to execute interactive docker command
[ https://issues.apache.org/jira/browse/YARN-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8777: Attachment: YARN-8777.004.patch > Container Executor C binary change to execute interactive docker command > > > Key: YARN-8777 > URL: https://issues.apache.org/jira/browse/YARN-8777 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zian Chen >Assignee: Eric Yang >Priority: Major > Labels: Docker > Attachments: YARN-8777.001.patch, YARN-8777.002.patch, > YARN-8777.003.patch, YARN-8777.004.patch > > > Since Container Executor provides Container execution using the native > container-executor binary, we also need to make changes to accept new > “dockerExec” method to invoke the corresponding native function to execute > docker exec command to the running container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8808) Use aggregate container utilization instead of node utilization to determine resources available for oversubscription
[ https://issues.apache.org/jira/browse/YARN-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637437#comment-16637437 ] Haibo Chen commented on YARN-8808: -- Thanks [~asuresh], for the reviews! > Use aggregate container utilization instead of node utilization to determine > resources available for oversubscription > - > > Key: YARN-8808 > URL: https://issues.apache.org/jira/browse/YARN-8808 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: YARN-1011 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8088-YARN-1011.01.patch, > YARN-8808-YARN-1011.00.patch, YARN-8808-YARN-1011.02.patch, > YARN-8808-YARN-1011.03.patch > > > Resource oversubscription should be bound to the amount of the resources that > can be allocated to containers, hence the allocation threshold should be with > respect to aggregate container utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8822) Nvidia-docker v2 support
[ https://issues.apache.org/jira/browse/YARN-8822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated YARN-8822: -- Labels: Docker (was: ) > Nvidia-docker v2 support > > > Key: YARN-8822 > URL: https://issues.apache.org/jira/browse/YARN-8822 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Priority: Major > Labels: Docker > > To run a GPU container with Docker, we have nvdia-docker v1 support already > but is deprecated per > [here|https://github.com/NVIDIA/nvidia-docker/wiki/About-version-2.0]. We > should support nvdia-docker v2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6338) Typos in Docker docs: contains => containers
[ https://issues.apache.org/jira/browse/YARN-6338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated YARN-6338: -- Labels: Docker docs (was: docs) > Typos in Docker docs: contains => containers > > > Key: YARN-6338 > URL: https://issues.apache.org/jira/browse/YARN-6338 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: Zoltan Siegl >Priority: Minor > Labels: Docker, docs > Fix For: 3.2.0 > > Attachments: YARN-6338.001.patch > > > "allowed to request privileged contains" should be "allowed to request > privileged containers" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7644) NM gets backed up deleting docker containers
[
https://issues.apache.org/jira/browse/YARN-7644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637378#comment-16637378
]
Chandni Singh commented on YARN-7644:
-
Fixed some of the checkstyle warnings in patch 2.
The ones for indentation in ContainersLauncher follow the indentation of the
existing statements under case
{code:java}
/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainersLauncher.java:156:
ContainerLaunch existingLaunch = running.remove(containerId);: 'block'
child has incorrect indentation level 8, expected level should be 6.
[Indentation]
{code}
{quote}I think ContainerCleanup class belong to
org.apache.hadoop.yarn.server.nodemanager.containermanager.deletion.task
instead of in
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher for reusing
existing package structure.
{quote}
[~eyang] I think the
{{org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher}} is
better for ContainerCleanup. This package contains all the tasks- launch,
relaunch which are created by {{ContainersLauncher}} and are not used outside
this package.
> NM gets backed up deleting docker containers
>
>
> Key: YARN-7644
> URL: https://issues.apache.org/jira/browse/YARN-7644
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Eric Badger
>Assignee: Chandni Singh
>Priority: Major
> Labels: Docker
> Attachments: YARN-7644.001.patch, YARN-7644.002.patch
>
>
> We are sending a {{docker stop}} to the docker container with a timeout of 10
> seconds when we shut down a container. If the container does not stop after
> 10 seconds then we force kill it. However, the {{docker stop}} command is a
> blocking call. So in cases where lots of containers don't go down with the
> initial SIGTERM, we have to wait 10+ seconds for the {{docker stop}} to
> return. This ties up the ContainerLaunch handler and so these kill events
> back up. It also appears to be backing up new container launches as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8840) Add missing cleanupSSLConfig() call for TestTimelineClient test
[
https://issues.apache.org/jira/browse/YARN-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637375#comment-16637375
]
Hadoop QA commented on YARN-8840:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
20s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8840 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942297/YARN-8840.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux ff8bbcba60e3 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7051bd7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22045/testReport/ |
| Max. process+thread count | 412 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22045/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add missing cleanupSSLConfig() call for TestTimelineClient test
> ---
>
>
[jira] [Updated] (YARN-7644) NM gets backed up deleting docker containers
[
https://issues.apache.org/jira/browse/YARN-7644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandni Singh updated YARN-7644:
Attachment: YARN-7644.002.patch
> NM gets backed up deleting docker containers
>
>
> Key: YARN-7644
> URL: https://issues.apache.org/jira/browse/YARN-7644
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Eric Badger
>Assignee: Chandni Singh
>Priority: Major
> Labels: Docker
> Attachments: YARN-7644.001.patch, YARN-7644.002.patch
>
>
> We are sending a {{docker stop}} to the docker container with a timeout of 10
> seconds when we shut down a container. If the container does not stop after
> 10 seconds then we force kill it. However, the {{docker stop}} command is a
> blocking call. So in cases where lots of containers don't go down with the
> initial SIGTERM, we have to wait 10+ seconds for the {{docker stop}} to
> return. This ties up the ContainerLaunch handler and so these kill events
> back up. It also appears to be backing up new container launches as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8808) Use aggregate container utilization instead of node utilization to determine resources available for oversubscription
[ https://issues.apache.org/jira/browse/YARN-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637342#comment-16637342 ] Arun Suresh commented on YARN-8808: --- Thanks [~haibochen]. Committed to YARN-1011 > Use aggregate container utilization instead of node utilization to determine > resources available for oversubscription > - > > Key: YARN-8808 > URL: https://issues.apache.org/jira/browse/YARN-8808 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: YARN-1011 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8088-YARN-1011.01.patch, > YARN-8808-YARN-1011.00.patch, YARN-8808-YARN-1011.02.patch, > YARN-8808-YARN-1011.03.patch > > > Resource oversubscription should be bound to the amount of the resources that > can be allocated to containers, hence the allocation threshold should be with > respect to aggregate container utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8808) Use aggregate container utilization instead of node utilization to determine resources available for oversubscription
[ https://issues.apache.org/jira/browse/YARN-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637335#comment-16637335 ] Arun Suresh commented on YARN-8808: --- Nope, +1 Will commit shortly.. > Use aggregate container utilization instead of node utilization to determine > resources available for oversubscription > - > > Key: YARN-8808 > URL: https://issues.apache.org/jira/browse/YARN-8808 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: YARN-1011 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8088-YARN-1011.01.patch, > YARN-8808-YARN-1011.00.patch, YARN-8808-YARN-1011.02.patch, > YARN-8808-YARN-1011.03.patch > > > Resource oversubscription should be bound to the amount of the resources that > can be allocated to containers, hence the allocation threshold should be with > respect to aggregate container utilization. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8840) Add missing cleanupSSLConfig() call for TestTimelineClient test
[
https://issues.apache.org/jira/browse/YARN-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637265#comment-16637265
]
Aki Tanaka commented on YARN-8840:
--
Fixed styles issues. Regarding the unit test failure, I think this is not
related to my patch. I confirmed the error without applying my patch.
[^YARN-8840.003.patch]
> Add missing cleanupSSLConfig() call for TestTimelineClient test
> ---
>
> Key: YARN-8840
> URL: https://issues.apache.org/jira/browse/YARN-8840
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, timelineclient
>Reporter: Aki Tanaka
>Priority: Minor
> Attachments: YARN-8840.001.patch, YARN-8840.002.patch,
> YARN-8840.003.patch
>
>
> Tests that setup SSLConfigs can leave conf-files lingering unless they are
> cleaned up via {{KeyStoreTestUtil.cleanupSSLConfig}} call. TestTimelineClient
> test is missing this call.
> If the cleanup method is not called explicitly, a modified ssl-client.xml is
> left in {{test-classes}}, might affect subsequent test cases.
>
> There was a similar report in HDFS-11042, but looks that we need to fix
> TestTimelineClient test too.
>
> {code:java}
> $ mvn test -Dtest=TestTimelineClient
> $ find .|grep ssl-client.xml$
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-classes/ssl-client.xml
> $ cat
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-classes/ssl-client.xml
>
> ssl.client.truststore.reload.interval1000falseprogrammatically
> ssl.client.truststore.location/Users/tanaka/work/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-dir/trustKS.jksfalseprogrammatically
> ssl.client.keystore.keypasswordclientPfalseprogrammatically
> ssl.client.keystore.location/Users/tanaka/work/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-dir/clientKS.jksfalseprogrammatically
> ssl.client.truststore.passwordtrustPfalseprogrammatically
> ssl.client.keystore.passwordclientPfalseprogrammatically
> {code}
>
> After applying this patch, the ssl-client.xml is not generated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8644) Add more test coverage for RMAppImpl.FinalTransition
[
https://issues.apache.org/jira/browse/YARN-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637267#comment-16637267
]
Hadoop QA commented on YARN-8644:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 4 new + 128 unchanged - 8 fixed = 132 total (was 136)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 15s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}124m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
| | hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8644 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942284/YARN-8644.008.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 28348971b724 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 7051bd7 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22044/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
[jira] [Updated] (YARN-8840) Add missing cleanupSSLConfig() call for TestTimelineClient test
[
https://issues.apache.org/jira/browse/YARN-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aki Tanaka updated YARN-8840:
-
Attachment: YARN-8840.003.patch
> Add missing cleanupSSLConfig() call for TestTimelineClient test
> ---
>
> Key: YARN-8840
> URL: https://issues.apache.org/jira/browse/YARN-8840
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, timelineclient
>Reporter: Aki Tanaka
>Priority: Minor
> Attachments: YARN-8840.001.patch, YARN-8840.002.patch,
> YARN-8840.003.patch
>
>
> Tests that setup SSLConfigs can leave conf-files lingering unless they are
> cleaned up via {{KeyStoreTestUtil.cleanupSSLConfig}} call. TestTimelineClient
> test is missing this call.
> If the cleanup method is not called explicitly, a modified ssl-client.xml is
> left in {{test-classes}}, might affect subsequent test cases.
>
> There was a similar report in HDFS-11042, but looks that we need to fix
> TestTimelineClient test too.
>
> {code:java}
> $ mvn test -Dtest=TestTimelineClient
> $ find .|grep ssl-client.xml$
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-classes/ssl-client.xml
> $ cat
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-classes/ssl-client.xml
>
> ssl.client.truststore.reload.interval1000falseprogrammatically
> ssl.client.truststore.location/Users/tanaka/work/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-dir/trustKS.jksfalseprogrammatically
> ssl.client.keystore.keypasswordclientPfalseprogrammatically
> ssl.client.keystore.location/Users/tanaka/work/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-dir/clientKS.jksfalseprogrammatically
> ssl.client.truststore.passwordtrustPfalseprogrammatically
> ssl.client.keystore.passwordclientPfalseprogrammatically
> {code}
>
> After applying this patch, the ssl-client.xml is not generated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8732) Add unit tests of min/max allocation for custom resource types in FairScheduler
[ https://issues.apache.org/jira/browse/YARN-8732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637253#comment-16637253 ] Haibo Chen commented on YARN-8732: -- Thanks [~snemeth] for the update! Can you address the outstanding checkstyle issues? Also, we can rename TestApplicationMasterInterceptor to TestApplicationMasterServiceInterceptor. Otherwise, patch looks good to me. > Add unit tests of min/max allocation for custom resource types in > FairScheduler > --- > > Key: YARN-8732 > URL: https://issues.apache.org/jira/browse/YARN-8732 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.2.0 >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Labels: unittest > Attachments: YARN-8732.001.patch, YARN-8732.002.patch, > YARN-8732.003.patch, YARN-8732.004.patch, YARN-8732.005.patch, > YARN-8732.006.patch > > > Create testcase like this, but for FS: > org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService#testValidateRequestCapacityAgainstMinMaxAllocationFor3rdResourceTypes -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8644) Add more test coverage for RMAppImpl.FinalTransition
[ https://issues.apache.org/jira/browse/YARN-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8644: - Summary: Add more test coverage for RMAppImpl.FinalTransition (was: Make RMAppImpl$FinalTransition more readable + add more test coverage) > Add more test coverage for RMAppImpl.FinalTransition > > > Key: YARN-8644 > URL: https://issues.apache.org/jira/browse/YARN-8644 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Attachments: YARN-8644.001.patch, YARN-8644.002.patch, > YARN-8644.003.patch, YARN-8644.004.patch, YARN-8644.005.patch, > YARN-8644.006.patch, YARN-8644.007.patch, YARN-8644.008.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8750) Refactor TestQueueMetrics
[
https://issues.apache.org/jira/browse/YARN-8750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637190#comment-16637190
]
Haibo Chen commented on YARN-8750:
--
Thank [~snemeth] for the update. The *MetricsChecker is delta based, by which I
mean we can create one from another and just specify what values have changed
as
AppMetricsChecker.createFromChecker(appMetricsChecker).gaugeInt().gaugeLong().
I can see in testSingleQueueWithUserMetrics() and
testTwoLevelWithUserMetrics(), the dependency is messed up. We should create a
AppMetricsChecker instance for queueSource and another instance for
parentQueueSource.
> Refactor TestQueueMetrics
> -
>
> Key: YARN-8750
> URL: https://issues.apache.org/jira/browse/YARN-8750
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-8750.001.patch, YARN-8750.002.patch,
> YARN-8750.003.patch, YARN-8750.004.patch, YARN-8750.005.patch
>
>
> {{TestQueueMetrics#checkApps}} and {{TestQueueMetrics#checkResources}} have 8
> and 14 parameters, respectively.
> It is very hard to read the testcases that are using these methods.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8777) Container Executor C binary change to execute interactive docker command
[ https://issues.apache.org/jira/browse/YARN-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637151#comment-16637151 ] Eric Yang commented on YARN-8777: - [~Zian Chen] Docker exec -it requires a tty to be attached for program to produce text based GUI output. For example, running top, or handling terminal resize event. Therefore, some environment variables, such as TERM, and control characters are managed externally to the exec program. The current patch doesn't implement pseudo terminal setup, this can prevent docker exec -it to fail if the Java side doesn't implement pseudo terminal. I need to revise this patch to implement pseudo terminal setup in C side, it would be much easier than doing pseudo terminal in the Java. > Container Executor C binary change to execute interactive docker command > > > Key: YARN-8777 > URL: https://issues.apache.org/jira/browse/YARN-8777 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zian Chen >Assignee: Eric Yang >Priority: Major > Labels: Docker > Attachments: YARN-8777.001.patch, YARN-8777.002.patch, > YARN-8777.003.patch > > > Since Container Executor provides Container execution using the native > container-executor binary, we also need to make changes to accept new > “dockerExec” method to invoke the corresponding native function to execute > docker exec command to the running container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7225) Add queue and partition info to RM audit log
[
https://issues.apache.org/jira/browse/YARN-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637142#comment-16637142
]
Eric Payne commented on YARN-7225:
--
After more manual testing on 001, I discovered the following issues:
- In the capacity scheduler, the {{AM Released Container}} log entry contains
an empty {{NODELABEL=}} KVP when there is no node label.
- In the fair scheduler, the {{AM Released Container}} log entry does not
container the {{QUEUENAME}} KVP.
- In the fair scheduler, if a queue is created at runtime, the {{Submit
Application Request}} log entry contains {{QUEUENAME=Default}} instead of the
name of the new queue.
> Add queue and partition info to RM audit log
>
>
> Key: YARN-7225
> URL: https://issues.apache.org/jira/browse/YARN-7225
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Jonathan Hung
>Assignee: Eric Payne
>Priority: Major
> Attachments: YARN-7225.001.patch
>
>
> Right now RM audit log has fields such as user, ip, resource, etc. Having
> queue and partition is useful for resource tracking.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7644) NM gets backed up deleting docker containers
[
https://issues.apache.org/jira/browse/YARN-7644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637117#comment-16637117
]
Eric Yang commented on YARN-7644:
-
[~csingh] The mvn install on root seems like a release branch error. We can
trigger test again after trunk has been fixed. I think ContainerCleanup class
belong to
org.apache.hadoop.yarn.server.nodemanager.containermanager.deletion.task
instead of in
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher for reusing
existing package structure.
Some checkstyle can be clean up.
[~ebadger] [~jlowe] Do you have any concern in the async task for reapContainer?
> NM gets backed up deleting docker containers
>
>
> Key: YARN-7644
> URL: https://issues.apache.org/jira/browse/YARN-7644
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Eric Badger
>Assignee: Chandni Singh
>Priority: Major
> Labels: Docker
> Attachments: YARN-7644.001.patch
>
>
> We are sending a {{docker stop}} to the docker container with a timeout of 10
> seconds when we shut down a container. If the container does not stop after
> 10 seconds then we force kill it. However, the {{docker stop}} command is a
> blocking call. So in cases where lots of containers don't go down with the
> initial SIGTERM, we have to wait 10+ seconds for the {{docker stop}} to
> return. This ties up the ContainerLaunch handler and so these kill events
> back up. It also appears to be backing up new container launches as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8763) Add WebSocket logic to the Node Manager web server to establish servlet
[
https://issues.apache.org/jira/browse/YARN-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637100#comment-16637100
]
Eric Yang edited comment on YARN-8763 at 10/3/18 3:47 PM:
--
[~Zian Chen] Thank you for the patch. A few minor problems:
{code}
+@WebServlet(urlPatterns="/container/$containerId")
{code}
I think it should be written as:
{code}
+@WebServlet(urlPatterns="/container/*")
{code}
As far as I can tell, $containerId is not used in the code, nor the correct way
to express WebServlet annotation expression.
ContainerShellClientSocket needs to be renamed to
ContainerShellClientSocketTest for testing purpose only. We need to write a
real version of ContainerShellClientSocket in src/main in YARN-8778.
Check style and findbugs errors needs to be cleaned up. The rest of the patch
looks good.
TestNNProxy unit test failure is not related to this patch.
was (Author: eyang):
[~Zian Chen] Thank you for the patch. A few minor problems:
{code}
+@WebServlet(urlPatterns="/container/$containerId")
{code}
I think it should be written as:
{code}
+@WebServlet(urlPatterns="/container/*")
{code}
As far as I can tell, $containerId is not used in the code, nor the correct way
to express WebServlet annotation expression.
ContainerShellClientSocket needs to be renamed to
ContainerShellClientSocketTest for testing purpose only. We need to write a
real version of ContainerShellClientSocket in src/main in YARN-8778.
Check style and findbugs errors needs to be cleaned up. The rest of the patch
looks good.
> Add WebSocket logic to the Node Manager web server to establish servlet
> ---
>
> Key: YARN-8763
> URL: https://issues.apache.org/jira/browse/YARN-8763
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zian Chen
>Assignee: Zian Chen
>Priority: Major
> Labels: Docker
> Attachments: YARN-8763-001.patch, YARN-8763.002.patch,
> YARN-8763.003.patch
>
>
> The reason we want to use WebSocket servlet to serve the backend instead of
> establishing the connection through HTTP is that WebSocket solves a few
> issues with HTTP which needed for our scenario,
> # In HTTP, the request is always initiated by the client and the response is
> processed by the server — making HTTP a unidirectional protocol, while web
> socket provides the Bi-directional protocol which means either client/server
> can send a message to the other party.
> # Full-duplex communication — client and server can talk to each other
> independently at the same time
> # Single TCP connection — After upgrading the HTTP connection in the
> beginning, client and server communicate over that same TCP connection
> throughout the lifecycle of WebSocket connection
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8763) Add WebSocket logic to the Node Manager web server to establish servlet
[
https://issues.apache.org/jira/browse/YARN-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637100#comment-16637100
]
Eric Yang commented on YARN-8763:
-
[~Zian Chen] Thank you for the patch. A few minor problems:
{code}
+@WebServlet(urlPatterns="/container/$containerId")
{code}
I think it should be written as:
{code}
+@WebServlet(urlPatterns="/container/*")
{code}
As far as I can tell, $containerId is not used in the code, nor the correct way
to express WebServlet annotation expression.
ContainerShellClientSocket needs to be renamed to
ContainerShellClientSocketTest for testing purpose only. We need to write a
real version of ContainerShellClientSocket in src/main in YARN-8778.
Check style and findbugs errors needs to be cleaned up. The rest of the patch
looks good.
> Add WebSocket logic to the Node Manager web server to establish servlet
> ---
>
> Key: YARN-8763
> URL: https://issues.apache.org/jira/browse/YARN-8763
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zian Chen
>Assignee: Zian Chen
>Priority: Major
> Labels: Docker
> Attachments: YARN-8763-001.patch, YARN-8763.002.patch,
> YARN-8763.003.patch
>
>
> The reason we want to use WebSocket servlet to serve the backend instead of
> establishing the connection through HTTP is that WebSocket solves a few
> issues with HTTP which needed for our scenario,
> # In HTTP, the request is always initiated by the client and the response is
> processed by the server — making HTTP a unidirectional protocol, while web
> socket provides the Bi-directional protocol which means either client/server
> can send a message to the other party.
> # Full-duplex communication — client and server can talk to each other
> independently at the same time
> # Single TCP connection — After upgrading the HTTP connection in the
> beginning, client and server communicate over that same TCP connection
> throughout the lifecycle of WebSocket connection
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6416) SIGNAL_CMD argument number is wrong
[
https://issues.apache.org/jira/browse/YARN-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637076#comment-16637076
]
Vidura Bhathiya Mudalige commented on YARN-6416:
[~miklos.szeg...@cloudera.com]
Miklos Szegedi, can you please assign this JIRA to me?
> SIGNAL_CMD argument number is wrong
> ---
>
> Key: YARN-6416
> URL: https://issues.apache.org/jira/browse/YARN-6416
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Priority: Minor
> Labels: newbie
>
> Yarn application signal command has two arguments, so the number below should
> be 2 I think.
> {code}
> opts.getOption(SIGNAL_CMD).setArgs(3);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6416) SIGNAL_CMD argument number is wrong
[
https://issues.apache.org/jira/browse/YARN-6416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637073#comment-16637073
]
ASF GitHub Bot commented on YARN-6416:
--
GitHub user vbmudalige opened a pull request:
https://github.com/apache/hadoop/pull/423
YARN-6416. SIGNAL_CMD argument number is wrong
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vbmudalige/hadoop YARN-6416
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hadoop/pull/423.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #423
commit b2532663babcf4ef4560fb8067b922e61638679e
Author: Vidura Mudalige
Date: 2018-10-03T15:13:44Z
YARN-6416. SIGNAL_CMD argument number is wrong
> SIGNAL_CMD argument number is wrong
> ---
>
> Key: YARN-6416
> URL: https://issues.apache.org/jira/browse/YARN-6416
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Priority: Minor
> Labels: newbie
>
> Yarn application signal command has two arguments, so the number below should
> be 2 I think.
> {code}
> opts.getOption(SIGNAL_CMD).setArgs(3);
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8644) Make RMAppImpl$FinalTransition more readable + add more test coverage
[ https://issues.apache.org/jira/browse/YARN-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16637055#comment-16637055 ] Szilard Nemeth commented on YARN-8644: -- Hi [~rohithsharma]! Yes, the changes in RmAppImpl are only for better readability (refactor). This is what I had put to the title of this jira plus adding more test coverage to the final transition. Hope this is clear. Thanks! > Make RMAppImpl$FinalTransition more readable + add more test coverage > - > > Key: YARN-8644 > URL: https://issues.apache.org/jira/browse/YARN-8644 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Attachments: YARN-8644.001.patch, YARN-8644.002.patch, > YARN-8644.003.patch, YARN-8644.004.patch, YARN-8644.005.patch, > YARN-8644.006.patch, YARN-8644.007.patch, YARN-8644.008.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8644) Make RMAppImpl$FinalTransition more readable + add more test coverage
[ https://issues.apache.org/jira/browse/YARN-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-8644: - Attachment: YARN-8644.008.patch > Make RMAppImpl$FinalTransition more readable + add more test coverage > - > > Key: YARN-8644 > URL: https://issues.apache.org/jira/browse/YARN-8644 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Attachments: YARN-8644.001.patch, YARN-8644.002.patch, > YARN-8644.003.patch, YARN-8644.004.patch, YARN-8644.005.patch, > YARN-8644.006.patch, YARN-8644.007.patch, YARN-8644.008.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6091) the AppMaster register failed when use Docker on LinuxContainer
[ https://issues.apache.org/jira/browse/YARN-6091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe resolved YARN-6091. -- Resolution: Implemented Fix Version/s: 3.1.1 3.2.0 Closing this as fixed by YARN-7654. > the AppMaster register failed when use Docker on LinuxContainer > > > Key: YARN-6091 > URL: https://issues.apache.org/jira/browse/YARN-6091 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, yarn >Affects Versions: 2.8.1 > Environment: CentOS >Reporter: zhengchenyu >Assignee: Eric Badger >Priority: Critical > Labels: Docker > Fix For: 3.2.0, 3.1.1 > > Attachments: YARN-6091.001.patch, YARN-6091.002.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > In some servers, When I use Docker on LinuxContainer, I found the aciton that > AppMaster register to Resourcemanager failed. But didn't happen in other > servers. > I found the pclose (in container-executor.c) return different value in > different server, even though the process which is launched by popen is > running normally. Some server return 0, and others return 13. > Because yarn regard the application as failed application when pclose return > nonzero, and yarn will remove the AMRMToken, then the AppMaster register > failed because Resourcemanager have removed this applicaiton's token. > In container-executor.c, the judgement condition is whether the return code > is zero. But man the pclose, the document tells that "pclose return -1" > represent wrong. So I change the judgement condition, then slove this > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6091) the AppMaster register failed when use Docker on LinuxContainer
[ https://issues.apache.org/jira/browse/YARN-6091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636996#comment-16636996 ] Eric Badger commented on YARN-6091: --- Given the fact that Docker support is pretty low on branch-2.x, I'm inclined to close this as won't fix and link YARN-7654 > the AppMaster register failed when use Docker on LinuxContainer > > > Key: YARN-6091 > URL: https://issues.apache.org/jira/browse/YARN-6091 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, yarn >Affects Versions: 2.8.1 > Environment: CentOS >Reporter: zhengchenyu >Assignee: Eric Badger >Priority: Critical > Labels: Docker > Attachments: YARN-6091.001.patch, YARN-6091.002.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > In some servers, When I use Docker on LinuxContainer, I found the aciton that > AppMaster register to Resourcemanager failed. But didn't happen in other > servers. > I found the pclose (in container-executor.c) return different value in > different server, even though the process which is launched by popen is > running normally. Some server return 0, and others return 13. > Because yarn regard the application as failed application when pclose return > nonzero, and yarn will remove the AMRMToken, then the AppMaster register > failed because Resourcemanager have removed this applicaiton's token. > In container-executor.c, the judgement condition is whether the return code > is zero. But man the pclose, the document tells that "pclose return -1" > represent wrong. So I change the judgement condition, then slove this > problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8750) Refactor TestQueueMetrics
[
https://issues.apache.org/jira/browse/YARN-8750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636708#comment-16636708
]
Hadoop QA commented on YARN-8750:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 11 unchanged - 71 fixed = 12 total (was 82)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 82m
33s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8750 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942235/YARN-8750.005.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6b582434e2cc 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2626f46 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22042/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22042/testReport/ |
| Max. process+thread count | 900 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (YARN-8644) Make RMAppImpl$FinalTransition more readable + add more test coverage
[
https://issues.apache.org/jira/browse/YARN-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636700#comment-16636700
]
Hadoop QA commented on YARN-8644:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 5 new + 128 unchanged - 8 fixed = 133 total (was 136)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 4 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 24s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 3s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueManagementDynamicEditPolicy
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8644 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942233/YARN-8644.007.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 0bbabc12f858 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2626f46 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22040/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| whitespace |
[jira] [Commented] (YARN-8732) Add unit tests of min/max allocation for custom resource types in FairScheduler
[
https://issues.apache.org/jira/browse/YARN-8732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636698#comment-16636698
]
Hadoop QA commented on YARN-8732:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 8 new + 0 unchanged - 19 fixed = 8 total (was 19) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 76m
26s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}124m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8732 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942237/YARN-8732.006.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 749f55e2bec8 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10
07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2626f46 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22038/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22038/testReport/ |
| Max. process+thread count | 931 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
[jira] [Commented] (YARN-8763) Add WebSocket logic to the Node Manager web server to establish servlet
[
https://issues.apache.org/jira/browse/YARN-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636687#comment-16636687
]
Hadoop QA commented on YARN-8763:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 3s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-client-modules/hadoop-client-runtime
hadoop-client-modules/hadoop-client-minicluster {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 57s{color} | {color:orange} root: The patch generated 28 new + 67 unchanged
- 0 fixed = 95 total (was 67) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-client-modules/hadoop-client-runtime
hadoop-client-modules/hadoop-client-minicluster {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
11s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
26s{color} | {color:green} hadoop-project in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m 21s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hadoop-client-runtime in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hadoop-client-minicluster in the patch passed.
[jira] [Commented] (YARN-8840) Add missing cleanupSSLConfig() call for TestTimelineClient test
[
https://issues.apache.org/jira/browse/YARN-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636617#comment-16636617
]
Hadoop QA commented on YARN-8840:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: The patch generated 2 new +
25 unchanged - 0 fixed = 27 total (was 25) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 44s{color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.client.api.impl.TestTimelineClientV2Impl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8840 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942242/YARN-8840.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 63fcfc511dfb 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2626f46 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22039/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22039/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22039/testReport/ |
[jira] [Commented] (YARN-8644) Make RMAppImpl$FinalTransition more readable + add more test coverage
[ https://issues.apache.org/jira/browse/YARN-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636612#comment-16636612 ] Rohith Sharma K S commented on YARN-8644: - Thanks [~snemeth] for the patch.. I couldn't get intention of this patch. Basically to add more test coverage ? If so, why RMAppImpl is changed? Is change in RMAppImpl is only refactoring code? > Make RMAppImpl$FinalTransition more readable + add more test coverage > - > > Key: YARN-8644 > URL: https://issues.apache.org/jira/browse/YARN-8644 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Attachments: YARN-8644.001.patch, YARN-8644.002.patch, > YARN-8644.003.patch, YARN-8644.004.patch, YARN-8644.005.patch, > YARN-8644.006.patch, YARN-8644.007.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8842) Update QueueMetrics with custom resource values
[
https://issues.apache.org/jira/browse/YARN-8842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636559#comment-16636559
]
Hadoop QA commented on YARN-8842:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} YARN-8842 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-8842 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942244/YARN-8842.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22043/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Update QueueMetrics with custom resource values
>
>
> Key: YARN-8842
> URL: https://issues.apache.org/jira/browse/YARN-8842
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-8842.001.patch
>
>
> This is the 2nd dependent jira of YARN-8059.
> As updating the metrics is an independent step from handling preemption, this
> jira only deals with the queue metrics update of custom resources.
> The following metrics should be updated:
> * allocated resources
> * available resources
> * pending resources
> * reserved resources
> * aggregate seconds preempted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8763) Add WebSocket logic to the Node Manager web server to establish servlet
[ https://issues.apache.org/jira/browse/YARN-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636543#comment-16636543 ] Zian Chen commented on YARN-8763: - Hi [~eyang], could you help review patch 003? It address comments as we discussed above. Thanks > Add WebSocket logic to the Node Manager web server to establish servlet > --- > > Key: YARN-8763 > URL: https://issues.apache.org/jira/browse/YARN-8763 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zian Chen >Assignee: Zian Chen >Priority: Major > Labels: Docker > Attachments: YARN-8763-001.patch, YARN-8763.002.patch, > YARN-8763.003.patch > > > The reason we want to use WebSocket servlet to serve the backend instead of > establishing the connection through HTTP is that WebSocket solves a few > issues with HTTP which needed for our scenario, > # In HTTP, the request is always initiated by the client and the response is > processed by the server — making HTTP a unidirectional protocol, while web > socket provides the Bi-directional protocol which means either client/server > can send a message to the other party. > # Full-duplex communication — client and server can talk to each other > independently at the same time > # Single TCP connection — After upgrading the HTTP connection in the > beginning, client and server communicate over that same TCP connection > throughout the lifecycle of WebSocket connection -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8763) Add WebSocket logic to the Node Manager web server to establish servlet
[ https://issues.apache.org/jira/browse/YARN-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zian Chen updated YARN-8763: Attachment: YARN-8763.003.patch > Add WebSocket logic to the Node Manager web server to establish servlet > --- > > Key: YARN-8763 > URL: https://issues.apache.org/jira/browse/YARN-8763 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zian Chen >Assignee: Zian Chen >Priority: Major > Labels: Docker > Attachments: YARN-8763-001.patch, YARN-8763.002.patch, > YARN-8763.003.patch > > > The reason we want to use WebSocket servlet to serve the backend instead of > establishing the connection through HTTP is that WebSocket solves a few > issues with HTTP which needed for our scenario, > # In HTTP, the request is always initiated by the client and the response is > processed by the server — making HTTP a unidirectional protocol, while web > socket provides the Bi-directional protocol which means either client/server > can send a message to the other party. > # Full-duplex communication — client and server can talk to each other > independently at the same time > # Single TCP connection — After upgrading the HTTP connection in the > beginning, client and server communicate over that same TCP connection > throughout the lifecycle of WebSocket connection -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8834) Provide Java client for fetching entities from TimelineReader
[ https://issues.apache.org/jira/browse/YARN-8834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636534#comment-16636534 ] Rohith Sharma K S commented on YARN-8834: - Never mind.. let go with current approach. bq. I am not in favor of taking url path as input parameter as that can be abused and client need to know about all the endpoints. Not the URL taking as parameter.. URL is internal to client.. getEntity(String entityType, String entityId, String fields, String filter) > Provide Java client for fetching entities from TimelineReader > - > > Key: YARN-8834 > URL: https://issues.apache.org/jira/browse/YARN-8834 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Abhishek Modi >Priority: Critical > Attachments: YARN-8834.001.patch, YARN-8834.002.patch > > > While reviewing YARN-8303, we felt that it is necessary to provide > TimelineReaderClient which wraps all the REST calls in it so that user can > just provide EntityType and EntityId along with filters.Currently fetching > entities from TimelineReader is only via REST call or somebody need to write > java client get entities. > It is good to provide TimelineReaderClient which fetch entities from > TimelineReaderServer. This will be more useful. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8788) mvn package -Pyarn-ui fails on JDK9
[
https://issues.apache.org/jira/browse/YARN-8788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636516#comment-16636516
]
ASF GitHub Bot commented on YARN-8788:
--
Github user vbmudalige commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/421#discussion_r222198499
--- Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/pom.xml ---
@@ -202,6 +202,13 @@
ro.isdc.wro4j
wro4j-maven-plugin
1.7.9
+
+
+ org.mockito
+ mockito-core
+ 2.18.0
+
+
--- End diff --
@aajisaka, I made the changes. Please review.
> mvn package -Pyarn-ui fails on JDK9
> ---
>
> Key: YARN-8788
> URL: https://issues.apache.org/jira/browse/YARN-8788
> Project: Hadoop YARN
> Issue Type: Bug
> Environment: Java 9.0.4, CentOS 7.5
>Reporter: Akira Ajisaka
>Assignee: Vidura Bhathiya Mudalige
>Priority: Major
> Labels: newbie
>
> {{mvn package -Pdist,native,yarn-ui -Dtar -DskipTests}} failed on trunk.
> {noformat}
> [ERROR] Failed to execute goal ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run
> (default) on project hadoop-yarn-ui: Execution default of goal
> ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run failed: An API incompatibility was
> encountered while executing ro.isdc.wro4j:wro4j-maven-plugin:1.7.9:run:
> java.lang.ExceptionInInitializerError: null
> [ERROR] -
> [ERROR] realm =plugin>ro.isdc.wro4j:wro4j-maven-plugin:1.7.9
> [ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
> [ERROR] urls[0] =
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-maven-plugin/1.7.9/wro4j-maven-plugin-1.7.9.jar
> [ERROR] urls[1] =
> file:/home/aajisaka/.m2/repository/ro/isdc/wro4j/wro4j-core/1.7.9/wro4j-core-1.7.9.jar
> [ERROR] urls[2] =
> file:/home/aajisaka/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar
> (snip)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8834) Provide Java client for fetching entities from TimelineReader
[
https://issues.apache.org/jira/browse/YARN-8834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636511#comment-16636511
]
Abhishek Modi commented on YARN-8834:
-
[~rohithsharma] thanks for reviewing it. I will attach a patch with the fix.
{quote}I am thinking should reader client more abstract to entities only? I
mean just to have getEntity() and getEntities() with much more options to
provide query params. Thoughts?
{quote}
Instead of exposing generic API like this, should I make existing APIs generic
enough to take params and filters as parameters. I am not in favor of taking
url path as input parameter as that can be abused and client need to know about
all the endpoints. In future, we can add more apis as and when required.
> Provide Java client for fetching entities from TimelineReader
> -
>
> Key: YARN-8834
> URL: https://issues.apache.org/jira/browse/YARN-8834
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Critical
> Attachments: YARN-8834.001.patch, YARN-8834.002.patch
>
>
> While reviewing YARN-8303, we felt that it is necessary to provide
> TimelineReaderClient which wraps all the REST calls in it so that user can
> just provide EntityType and EntityId along with filters.Currently fetching
> entities from TimelineReader is only via REST call or somebody need to write
> java client get entities.
> It is good to provide TimelineReaderClient which fetch entities from
> TimelineReaderServer. This will be more useful.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8834) Provide Java client for fetching entities from TimelineReader
[
https://issues.apache.org/jira/browse/YARN-8834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636505#comment-16636505
]
Rohith Sharma K S commented on YARN-8834:
-
Thanks for the patch.. Few comments
# ClusterId is super important for any REST queries. So take client
configuration cluster-id in serviceInit and use that as default. Else we may
need to provide option to provide cluster-id in API. In short, instead of
{{"apps/" + appId.toString()}}, this can be {{"cluster-id/apps/" +
appId.toString();}}
# I think retrieving specific fields should be controlled by API params. So,
params.add("fields", "ALL"); could be taken as param? Or default fields could
be INFO because retrieving ALL would consume more memory especially in list of
entities. IIRC, for constructing XXXReport, only INFO field is sufficient.
# For constructing URL, could we use static JOINER?
# getApplicationEntity has LOG with info method. Could it be removed?
Leaving apart all my above comments, I am thinking should reader client more
abstract to entities only? I mean just to have getEntity() and getEntities()
with much more options to provide query params. Thoughts?
> Provide Java client for fetching entities from TimelineReader
> -
>
> Key: YARN-8834
> URL: https://issues.apache.org/jira/browse/YARN-8834
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Abhishek Modi
>Priority: Critical
> Attachments: YARN-8834.001.patch, YARN-8834.002.patch
>
>
> While reviewing YARN-8303, we felt that it is necessary to provide
> TimelineReaderClient which wraps all the REST calls in it so that user can
> just provide EntityType and EntityId along with filters.Currently fetching
> entities from TimelineReader is only via REST call or somebody need to write
> java client get entities.
> It is good to provide TimelineReaderClient which fetch entities from
> TimelineReaderServer. This will be more useful.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8059) Resource type is ignored when FS decide to preempt
[
https://issues.apache.org/jira/browse/YARN-8059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636504#comment-16636504
]
Szilard Nemeth commented on YARN-8059:
--
Hi [~wilfreds]!
Thank you for your suggestion, it totally makes sense.
I separated the patch accordingly: created the jira (YARN-8842) that adds the
custom resource values to the QueueMetrics.
The latest patch for this jira now only contains the preemption related logic +
tests.
So the order to review and merge patches:
1. YARN-8750: past several rounds of review, will be committed soon.
2. YARN-8842
3. YARN-8059 (this jira)
Please let me know if this makes sense, thanks!
> Resource type is ignored when FS decide to preempt
> --
>
> Key: YARN-8059
> URL: https://issues.apache.org/jira/browse/YARN-8059
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0
>Reporter: Yufei Gu
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-8059.001.patch, YARN-8059.002.patch
>
>
> Method Fairscheduler#shouldAttemptPreemption doesn't consider resources other
> than vcore and memory. We may need to rethink it in the resource type
> scenario. cc [~miklos.szeg...@cloudera.com], [~wilfreds] and [~snemeth].
> {code}
> if (context.isPreemptionEnabled()) {
> return (context.getPreemptionUtilizationThreshold() < Math.max(
> (float) rootMetrics.getAllocatedMB() /
> getClusterResource().getMemorySize(),
> (float) rootMetrics.getAllocatedVirtualCores() /
> getClusterResource().getVirtualCores()));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8842) Update QueueMetrics with custom resource values
[ https://issues.apache.org/jira/browse/YARN-8842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-8842: - Attachment: YARN-8842.001.patch > Update QueueMetrics with custom resource values > > > Key: YARN-8842 > URL: https://issues.apache.org/jira/browse/YARN-8842 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-8842.001.patch > > > This is the 2nd dependent jira of YARN-8059. > As updating the metrics is an independent step from handling preemption, this > jira only deals with the queue metrics update of custom resources. > The following metrics should be updated: > * allocated resources > * available resources > * pending resources > * reserved resources > * aggregate seconds preempted -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8059) Resource type is ignored when FS decide to preempt
[
https://issues.apache.org/jira/browse/YARN-8059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-8059:
-
Attachment: YARN-8059.002.patch
> Resource type is ignored when FS decide to preempt
> --
>
> Key: YARN-8059
> URL: https://issues.apache.org/jira/browse/YARN-8059
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler
>Affects Versions: 3.0.0
>Reporter: Yufei Gu
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-8059.001.patch, YARN-8059.002.patch
>
>
> Method Fairscheduler#shouldAttemptPreemption doesn't consider resources other
> than vcore and memory. We may need to rethink it in the resource type
> scenario. cc [~miklos.szeg...@cloudera.com], [~wilfreds] and [~snemeth].
> {code}
> if (context.isPreemptionEnabled()) {
> return (context.getPreemptionUtilizationThreshold() < Math.max(
> (float) rootMetrics.getAllocatedMB() /
> getClusterResource().getMemorySize(),
> (float) rootMetrics.getAllocatedVirtualCores() /
> getClusterResource().getVirtualCores()));
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8840) Add missing cleanupSSLConfig() call for TestTimelineClient test
[
https://issues.apache.org/jira/browse/YARN-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636501#comment-16636501
]
Aki Tanaka commented on YARN-8840:
--
Thank you for looking and your comment! I moved the cleanup method in @After
class.
[^YARN-8840.002.patch]
> Add missing cleanupSSLConfig() call for TestTimelineClient test
> ---
>
> Key: YARN-8840
> URL: https://issues.apache.org/jira/browse/YARN-8840
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, timelineclient
>Reporter: Aki Tanaka
>Priority: Minor
> Attachments: YARN-8840.001.patch, YARN-8840.002.patch
>
>
> Tests that setup SSLConfigs can leave conf-files lingering unless they are
> cleaned up via {{KeyStoreTestUtil.cleanupSSLConfig}} call. TestTimelineClient
> test is missing this call.
> If the cleanup method is not called explicitly, a modified ssl-client.xml is
> left in {{test-classes}}, might affect subsequent test cases.
>
> There was a similar report in HDFS-11042, but looks that we need to fix
> TestTimelineClient test too.
>
> {code:java}
> $ mvn test -Dtest=TestTimelineClient
> $ find .|grep ssl-client.xml$
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-classes/ssl-client.xml
> $ cat
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-classes/ssl-client.xml
>
> ssl.client.truststore.reload.interval1000falseprogrammatically
> ssl.client.truststore.location/Users/tanaka/work/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-dir/trustKS.jksfalseprogrammatically
> ssl.client.keystore.keypasswordclientPfalseprogrammatically
> ssl.client.keystore.location/Users/tanaka/work/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-dir/clientKS.jksfalseprogrammatically
> ssl.client.truststore.passwordtrustPfalseprogrammatically
> ssl.client.keystore.passwordclientPfalseprogrammatically
> {code}
>
> After applying this patch, the ssl-client.xml is not generated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8840) Add missing cleanupSSLConfig() call for TestTimelineClient test
[
https://issues.apache.org/jira/browse/YARN-8840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aki Tanaka updated YARN-8840:
-
Attachment: YARN-8840.002.patch
> Add missing cleanupSSLConfig() call for TestTimelineClient test
> ---
>
> Key: YARN-8840
> URL: https://issues.apache.org/jira/browse/YARN-8840
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, timelineclient
>Reporter: Aki Tanaka
>Priority: Minor
> Attachments: YARN-8840.001.patch, YARN-8840.002.patch
>
>
> Tests that setup SSLConfigs can leave conf-files lingering unless they are
> cleaned up via {{KeyStoreTestUtil.cleanupSSLConfig}} call. TestTimelineClient
> test is missing this call.
> If the cleanup method is not called explicitly, a modified ssl-client.xml is
> left in {{test-classes}}, might affect subsequent test cases.
>
> There was a similar report in HDFS-11042, but looks that we need to fix
> TestTimelineClient test too.
>
> {code:java}
> $ mvn test -Dtest=TestTimelineClient
> $ find .|grep ssl-client.xml$
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-classes/ssl-client.xml
> $ cat
> ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-classes/ssl-client.xml
>
> ssl.client.truststore.reload.interval1000falseprogrammatically
> ssl.client.truststore.location/Users/tanaka/work/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-dir/trustKS.jksfalseprogrammatically
> ssl.client.keystore.keypasswordclientPfalseprogrammatically
> ssl.client.keystore.location/Users/tanaka/work/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/target/test-dir/clientKS.jksfalseprogrammatically
> ssl.client.truststore.passwordtrustPfalseprogrammatically
> ssl.client.keystore.passwordclientPfalseprogrammatically
> {code}
>
> After applying this patch, the ssl-client.xml is not generated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8842) Update QueueMetrics with custom resource values
Szilard Nemeth created YARN-8842: Summary: Update QueueMetrics with custom resource values Key: YARN-8842 URL: https://issues.apache.org/jira/browse/YARN-8842 Project: Hadoop YARN Issue Type: Improvement Reporter: Szilard Nemeth Assignee: Szilard Nemeth This is the 2nd dependent jira of YARN-8059. As updating the metrics is an independent step from handling preemption, this jira only deals with the queue metrics update of custom resources. The following metrics should be updated: * allocated resources * available resources * pending resources * reserved resources * aggregate seconds preempted -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
