[jira] [Commented] (YARN-6406) Remove SchedulerRequestKeys when no more pending ResourceRequest
[
https://issues.apache.org/jira/browse/YARN-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969771#comment-15969771
]
Hadoop QA commented on YARN-6406:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
42s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 7 new + 589 unchanged - 7 fixed = 596 total (was 596)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
8s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
13s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 46s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}119m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_121 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestRMRestart |
| JDK v1.7.0_121 Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue |
[jira] [Commented] (YARN-6406) Remove SchedulerRequestKeys when no more pending ResourceRequest
[
https://issues.apache.org/jira/browse/YARN-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969769#comment-15969769
]
Hadoop QA commented on YARN-6406:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
52s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
29s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
35s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 7 new + 589 unchanged - 7 fixed = 596 total (was 596)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
10s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
10s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 45s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}118m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_121 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestRMRestart |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | YARN-6406 |
| JIRA Patch URL |
[jira] [Commented] (YARN-6406) Remove SchedulerRequestKeys when no more pending ResourceRequest
[ https://issues.apache.org/jira/browse/YARN-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969670#comment-15969670 ] Hadoop QA commented on YARN-6406: - (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-YARN-Build/15648/console in case of problems. > Remove SchedulerRequestKeys when no more pending ResourceRequest > > > Key: YARN-6406 > URL: https://issues.apache.org/jira/browse/YARN-6406 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha2 >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6406.001.patch, YARN-6406.002.patch, > YARN-6406-branch-2.001.patch, YARN-6406-branch-2.002.patch > > > YARN-5540 introduced some optimizations to remove satisfied SchedulerKeys > from the AppScheduleingInfo. It looks like after YARN-6040, > ScedulerRequestKeys are removed only if the Application sends a 0 > numContainers requests. While earlier, the outstanding schedulerKeys were > also remove as soon as a container is allocated as well. > An additional optimization we were hoping to include is to remove the > ResourceRequests itself once the numContainers == 0, since we see in our > clusters that the RM heap space consumption increases drastically due to a > large number of ResourceRequests with 0 num containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6406) Remove SchedulerRequestKeys when no more pending ResourceRequest
[ https://issues.apache.org/jira/browse/YARN-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969664#comment-15969664 ] Hadoop QA commented on YARN-6406: - (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-YARN-Build/15647/console in case of problems. > Remove SchedulerRequestKeys when no more pending ResourceRequest > > > Key: YARN-6406 > URL: https://issues.apache.org/jira/browse/YARN-6406 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha2 >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6406.001.patch, YARN-6406.002.patch, > YARN-6406-branch-2.001.patch, YARN-6406-branch-2.002.patch > > > YARN-5540 introduced some optimizations to remove satisfied SchedulerKeys > from the AppScheduleingInfo. It looks like after YARN-6040, > ScedulerRequestKeys are removed only if the Application sends a 0 > numContainers requests. While earlier, the outstanding schedulerKeys were > also remove as soon as a container is allocated as well. > An additional optimization we were hoping to include is to remove the > ResourceRequests itself once the numContainers == 0, since we see in our > clusters that the RM heap space consumption increases drastically due to a > large number of ResourceRequests with 0 num containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6406) Remove SchedulerRequestKeys when no more pending ResourceRequest
[ https://issues.apache.org/jira/browse/YARN-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6406: -- Attachment: YARN-6406-branch-2.002.patch Reattaching patch with HADOOP-14311 fix to see if Jenkins will work > Remove SchedulerRequestKeys when no more pending ResourceRequest > > > Key: YARN-6406 > URL: https://issues.apache.org/jira/browse/YARN-6406 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha2 >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6406.001.patch, YARN-6406.002.patch, > YARN-6406-branch-2.001.patch, YARN-6406-branch-2.002.patch > > > YARN-5540 introduced some optimizations to remove satisfied SchedulerKeys > from the AppScheduleingInfo. It looks like after YARN-6040, > ScedulerRequestKeys are removed only if the Application sends a 0 > numContainers requests. While earlier, the outstanding schedulerKeys were > also remove as soon as a container is allocated as well. > An additional optimization we were hoping to include is to remove the > ResourceRequests itself once the numContainers == 0, since we see in our > clusters that the RM heap space consumption increases drastically due to a > large number of ResourceRequests with 0 num containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6483) Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned by the Resource Manager as a response to the Application Master heartbeat
[ https://issues.apache.org/jira/browse/YARN-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Rodríguez Hortalá updated YARN-6483: - Attachment: (was: YARN-6483.patch) > Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes > returned by the Resource Manager as a response to the Application Master > heartbeat > > > Key: YARN-6483 > URL: https://issues.apache.org/jira/browse/YARN-6483 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.7.3 >Reporter: Juan Rodríguez Hortalá > > The DECOMMISSIONING node state is currently used as part of the graceful > decommissioning mechanism to give time for tasks to complete in a node that > is scheduled for decommission, and for reducer tasks to read the shuffle > blocks in that node. Also, YARN effectively blacklists nodes in > DECOMMISSIONING state by assigning them a capacity of 0, to prevent > additional containers to be launched in those nodes, so no more shuffle > blocks are written to the node. This blacklisting is not effective for > applications like Spark, because a Spark executor running in a YARN container > will keep receiving more tasks after the corresponding node has been > blacklisted at the YARN level. We would like to propose a modification of the > YARN heartbeat mechanism so nodes transitioning to DECOMMISSIONING are added > to the list of updated nodes returned by the Resource Manager as a response > to the Application Master heartbeat. This way a Spark application master > would be able to blacklist a DECOMMISSIONING at the Spark level. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6480) Timeout is too aggressive for TestAMRestart.testPreemptedAMRestartOnRMRestart
[ https://issues.apache.org/jira/browse/YARN-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969649#comment-15969649 ] Hudson commented on YARN-6480: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11587 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11587/]) YARN-6480. Timeout is too aggressive for (jlowe: rev 416880550214e58f2284a045ad0c96ba4aa78ea8) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java > Timeout is too aggressive for TestAMRestart.testPreemptedAMRestartOnRMRestart > - > > Key: YARN-6480 > URL: https://issues.apache.org/jira/browse/YARN-6480 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3 > > Attachments: YARN-6480.001.patch > > > Timeout is set to 20 seconds, but the test runs regularly at 15 seconds on my > machine. Any load and it could timeout. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6438) Code can be improved in ContainersMonitorImpl.java
[
https://issues.apache.org/jira/browse/YARN-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969631#comment-15969631
]
Hadoop QA commented on YARN-6438:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
3s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | YARN-6438 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863509/YARN-6438.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 681d2cc9b780 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a41f8dd |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15646/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15646/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Code can be improved in ContainersMonitorImpl.java
> --
>
> Key: YARN-6438
> URL: https://issues.apache.org/jira/browse/YARN-6438
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>
[jira] [Commented] (YARN-6433) Only accessible cgroup mount directories should be selected for a controller
[
https://issues.apache.org/jira/browse/YARN-6433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969625#comment-15969625
]
Hudson commented on YARN-6433:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11586 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11586/])
YARN-6433. Only accessible cgroup mount directories should be selected (kasha:
rev 8a1d7480f73906d8e0342690ec6c6b008d6de21b)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java
> Only accessible cgroup mount directories should be selected for a controller
>
>
> Key: YARN-6433
> URL: https://issues.apache.org/jira/browse/YARN-6433
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-alpha3
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6433.000.patch, YARN-6433.001.patch,
> YARN-6433.002.patch
>
>
> I have a Ubuntu16 box that returns the following error with pre-mounted
> cgroups on the latest trunk:
> {code}
> 2017-04-03 19:42:18,511 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsHandlerImpl:
> Cgroups not accessible /run/lxcfs/controllers/cpu,cpuacct
> {code}
> The version is:
> {code}
> $ uname -a
> Linux mybox 4.4.0-24-generic #43-Ubuntu SMP Wed Jun 8 19:27:37 UTC 2016
> x86_64 x86_64 x86_64 GNU/Linux
> {code}
> The following cpu cgroup filesystems are mounted:
> {code}
> $ grep cpuacct /etc/mtab
> cgroup /sys/fs/cgroup/cpu,cpuacct cgroup
> rw,nosuid,nodev,noexec,relatime,cpu,cpuacct,nsroot=/ 0 0
> cpu,cpuacct /run/lxcfs/controllers/cpu,cpuacct cgroup
> rw,relatime,cpu,cpuacct,nsroot=/ 0 0
> {code}
> /sys/fs/cgroup is accessible to my yarn user, so it should be selected
> instead of /run/lxcfs/controllers
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6480) Timeout is too aggressive for TestAMRestart.testPreemptedAMRestartOnRMRestart
[ https://issues.apache.org/jira/browse/YARN-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969614#comment-15969614 ] Jason Lowe commented on YARN-6480: -- +1 lgtm. Committing this. > Timeout is too aggressive for TestAMRestart.testPreemptedAMRestartOnRMRestart > - > > Key: YARN-6480 > URL: https://issues.apache.org/jira/browse/YARN-6480 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: YARN-6480.001.patch > > > Timeout is set to 20 seconds, but the test runs regularly at 15 seconds on my > machine. Any load and it could timeout. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6433) Only accessible cgroup mount directories should be selected for a controller
[
https://issues.apache.org/jira/browse/YARN-6433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969588#comment-15969588
]
Karthik Kambatla commented on YARN-6433:
+1
> Only accessible cgroup mount directories should be selected for a controller
>
>
> Key: YARN-6433
> URL: https://issues.apache.org/jira/browse/YARN-6433
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-alpha3
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6433.000.patch, YARN-6433.001.patch,
> YARN-6433.002.patch
>
>
> I have a Ubuntu16 box that returns the following error with pre-mounted
> cgroups on the latest trunk:
> {code}
> 2017-04-03 19:42:18,511 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsHandlerImpl:
> Cgroups not accessible /run/lxcfs/controllers/cpu,cpuacct
> {code}
> The version is:
> {code}
> $ uname -a
> Linux mybox 4.4.0-24-generic #43-Ubuntu SMP Wed Jun 8 19:27:37 UTC 2016
> x86_64 x86_64 x86_64 GNU/Linux
> {code}
> The following cpu cgroup filesystems are mounted:
> {code}
> $ grep cpuacct /etc/mtab
> cgroup /sys/fs/cgroup/cpu,cpuacct cgroup
> rw,nosuid,nodev,noexec,relatime,cpu,cpuacct,nsroot=/ 0 0
> cpu,cpuacct /run/lxcfs/controllers/cpu,cpuacct cgroup
> rw,relatime,cpu,cpuacct,nsroot=/ 0 0
> {code}
> /sys/fs/cgroup is accessible to my yarn user, so it should be selected
> instead of /run/lxcfs/controllers
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6438) Code can be improved in ContainersMonitorImpl.java
[ https://issues.apache.org/jira/browse/YARN-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-6438: - Attachment: YARN-6438.003.patch > Code can be improved in ContainersMonitorImpl.java > -- > > Key: YARN-6438 > URL: https://issues.apache.org/jira/browse/YARN-6438 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Labels: newbie > Attachments: YARN-6438.000.patch, YARN-6438.001.patch, > YARN-6438.002.patch, YARN-6438.003.patch > > > I noticed two code blocks that can be improved in ContainersMonitorImpl.java. > cpuUsagePercentPerCoreByAllContainers and cpuUsageTotalCoresByAllContainers > track the same value and CHANGE_MONITORING_CONTAINER_RESOURCE is checked > twice along with two calls to changeContainerResource. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6438) Code can be improved in ContainersMonitorImpl.java
[ https://issues.apache.org/jira/browse/YARN-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969569#comment-15969569 ] Miklos Szegedi commented on YARN-6438: -- Good point [~templedf]. I meant to keep both, because the first one describes cpuUsagePercentPerCore in both places. > Code can be improved in ContainersMonitorImpl.java > -- > > Key: YARN-6438 > URL: https://issues.apache.org/jira/browse/YARN-6438 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Labels: newbie > Attachments: YARN-6438.000.patch, YARN-6438.001.patch, > YARN-6438.002.patch > > > I noticed two code blocks that can be improved in ContainersMonitorImpl.java. > cpuUsagePercentPerCoreByAllContainers and cpuUsageTotalCoresByAllContainers > track the same value and CHANGE_MONITORING_CONTAINER_RESOURCE is checked > twice along with two calls to changeContainerResource. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6432) FairScheduler: Reserve preempted resources for corresponding applications
[
https://issues.apache.org/jira/browse/YARN-6432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969559#comment-15969559
]
Hadoop QA commented on YARN-6432:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
8s{color} | {color:red} Docker failed to build yetus/hadoop:b59b8b7. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6432 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863505/YARN-6432.branch-2.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15645/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FairScheduler: Reserve preempted resources for corresponding applications
> -
>
> Key: YARN-6432
> URL: https://issues.apache.org/jira/browse/YARN-6432
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6432.000.patch, YARN-6432.001.patch,
> YARN-6432.002.patch, YARN-6432.branch-2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6432) FairScheduler: Reserve preempted resources for corresponding applications
[ https://issues.apache.org/jira/browse/YARN-6432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-6432: - Attachment: YARN-6432.branch-2.patch Thank you for checking this in. Attached the branch-2 patch. > FairScheduler: Reserve preempted resources for corresponding applications > - > > Key: YARN-6432 > URL: https://issues.apache.org/jira/browse/YARN-6432 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-6432.000.patch, YARN-6432.001.patch, > YARN-6432.002.patch, YARN-6432.branch-2.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6344) Add parameter for rack locality delay in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969514#comment-15969514
]
Konstantinos Karanasos commented on YARN-6344:
--
Hi [~Huangkx6810]. I just uploaded a patch that can be applied in branch-2.8. I
built and ran some tests locally and it looks good.
You cannot apply this new patch directly to 2.7 though. The major change
between 2.7 and 2.8 is that you will need to apply the changes I did to
{{RegularContainerAllocator}} directly to the {{LeafQueue}} instead. Try to see
what I mean and if you need help with 2.7, let me know.
> Add parameter for rack locality delay in CapacityScheduler
> --
>
> Key: YARN-6344
> URL: https://issues.apache.org/jira/browse/YARN-6344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6344.001.patch, YARN-6344.002.patch,
> YARN-6344.003.patch, YARN-6344.004.patch, YARN-6344-branch-2.8.patch
>
>
> When relaxing locality from node to rack, the {{node-locality-parameter}} is
> used: when scheduling opportunities for a scheduler key are more than the
> value of this parameter, we relax locality and try to assign the container to
> a node in the corresponding rack.
> On the other hand, when relaxing locality to off-switch (i.e., assign the
> container anywhere in the cluster), we are using a {{localityWaitFactor}},
> which is computed based on the number of outstanding requests for a specific
> scheduler key, which is divided by the size of the cluster.
> In case of applications that request containers in big batches (e.g.,
> traditional MR jobs), and for relatively small clusters, the
> localityWaitFactor does not affect relaxing locality much.
> However, in case of applications that request containers in small batches,
> this load factor takes a very small value, which leads to assigning
> off-switch containers too soon. This situation is even more pronounced in big
> clusters.
> For example, if an application requests only one container per request, the
> locality will be relaxed after a single missed scheduling opportunity.
> The purpose of this JIRA is to rethink the way we are relaxing locality for
> off-switch assignments.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6344) Add parameter for rack locality delay in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantinos Karanasos updated YARN-6344:
-
Attachment: YARN-6344-branch-2.8.patch
Attaching patch that can be applied to branch-2.8.
> Add parameter for rack locality delay in CapacityScheduler
> --
>
> Key: YARN-6344
> URL: https://issues.apache.org/jira/browse/YARN-6344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6344.001.patch, YARN-6344.002.patch,
> YARN-6344.003.patch, YARN-6344.004.patch, YARN-6344-branch-2.8.patch
>
>
> When relaxing locality from node to rack, the {{node-locality-parameter}} is
> used: when scheduling opportunities for a scheduler key are more than the
> value of this parameter, we relax locality and try to assign the container to
> a node in the corresponding rack.
> On the other hand, when relaxing locality to off-switch (i.e., assign the
> container anywhere in the cluster), we are using a {{localityWaitFactor}},
> which is computed based on the number of outstanding requests for a specific
> scheduler key, which is divided by the size of the cluster.
> In case of applications that request containers in big batches (e.g.,
> traditional MR jobs), and for relatively small clusters, the
> localityWaitFactor does not affect relaxing locality much.
> However, in case of applications that request containers in small batches,
> this load factor takes a very small value, which leads to assigning
> off-switch containers too soon. This situation is even more pronounced in big
> clusters.
> For example, if an application requests only one container per request, the
> locality will be relaxed after a single missed scheduling opportunity.
> The purpose of this JIRA is to rethink the way we are relaxing locality for
> off-switch assignments.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6363) Extending SLS: Synthetic Load Generator
[
https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969452#comment-15969452
]
Hadoop QA commented on YARN-6363:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
21s{color} | {color:red} hadoop-sls in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
46s{color} | {color:red} hadoop-tools in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 46s{color}
| {color:red} hadoop-tools in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} hadoop-tools: The patch generated 68 new + 138
unchanged - 25 fixed = 206 total (was 163) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
21s{color} | {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 3m
4s{color} | {color:green} The patch generated 0 new + 74 unchanged - 1 fixed =
74 total (was 75) {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
12s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
7s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
20s{color} | {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
26s{color} | {color:green} hadoop-rumen in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 20s{color}
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | YARN-6363 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863495/YARN-6363.v13.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml shellcheck shelldocs |
| uname | Linux 8f41a28bcaf4 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-3053) [Security] Review and implement authentication in ATS v.2
[ https://issues.apache.org/jira/browse/YARN-3053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969443#comment-15969443 ] Robert Kanter commented on YARN-3053: - [~varun_saxena], thanks for posting the updated doc. It sounds like Approach 2 would be the best, right? It's much simpler and more straightforward than Approach 1, which requires passing delegation tokens around everywhere. The only downside to Approach 2 called out in the doc is that there's a potential ID clash, but it didn't seem like a big issue. Why did you decide to go with Approach 1? > [Security] Review and implement authentication in ATS v.2 > - > > Key: YARN-3053 > URL: https://issues.apache.org/jira/browse/YARN-3053 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Varun Saxena > Labels: YARN-5355, yarn-5355-merge-blocker > Attachments: ATSv2Authentication(draft).pdf, > ATSv2Authentication.v01.pdf > > > Per design in YARN-2928, we want to evaluate and review the system for > security, and ensure proper security in the system. > This includes proper authentication, token management, access control, and > any other relevant security aspects. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6363) Extending SLS: Synthetic Load Generator
[
https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969434#comment-15969434
]
Hadoop QA commented on YARN-6363:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
15s{color} | {color:red} hadoop-sls in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
24s{color} | {color:red} hadoop-tools in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 24s{color}
| {color:red} hadoop-tools in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} hadoop-tools: The patch generated 68 new + 136
unchanged - 25 fixed = 204 total (was 161) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
17s{color} | {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 2m
37s{color} | {color:green} The patch generated 0 new + 74 unchanged - 1 fixed =
74 total (was 75) {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
11s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
21s{color} | {color:green} hadoop-rumen in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 17s{color}
| {color:red} hadoop-sls in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | YARN-6363 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863492/YARN-6363.v12.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml shellcheck shelldocs |
| uname | Linux 59c592dfc92c 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a41f8dd |
| Default Java |
[jira] [Commented] (YARN-6433) Only accessible cgroup mount directories should be selected for a controller
[
https://issues.apache.org/jira/browse/YARN-6433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969425#comment-15969425
]
Hadoop QA commented on YARN-6433:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
3s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | YARN-6433 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863491/YARN-6433.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 11bec6d0d666 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a41f8dd |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15642/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15642/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Only accessible cgroup mount directories should be selected for a controller
>
>
> Key: YARN-6433
> URL: https://issues.apache.org/jira/browse/YARN-6433
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-alpha3
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6433.000.patch,
[jira] [Commented] (YARN-3666) Federation Intercepting and propagating AM-RM communications
[ https://issues.apache.org/jira/browse/YARN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969407#comment-15969407 ] Subru Krishnan commented on YARN-3666: -- Thanks [~botong] for reporting the test failure. I have created YARN-6485 to track it. > Federation Intercepting and propagating AM-RM communications > > > Key: YARN-3666 > URL: https://issues.apache.org/jira/browse/YARN-3666 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Kishore Chaliparambil >Assignee: Botong Huang > Attachments: YARN-3666-YARN-2915.v1.patch, > YARN-3666-YARN-2915.v2.patch, YARN-3666-YARN-2915.v3.patch, > YARN-3666-YARN-2915.v4.patch, YARN-3666-YARN-2915.v5.patch > > > In order, to support transparent "spanning" of jobs across sub-clusters, all > AM-RM communications are proxied (via YARN-2884). > This JIRA tracks federation-specific mechanisms that decide how to > "split/broadcast" requests to the RMs and "merge" answers to > the AM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6485) [Regression] TestFederationRMStateStoreService is failing with null pointer exception
Subru Krishnan created YARN-6485: Summary: [Regression] TestFederationRMStateStoreService is failing with null pointer exception Key: YARN-6485 URL: https://issues.apache.org/jira/browse/YARN-6485 Project: Hadoop YARN Issue Type: Sub-task Reporter: Subru Krishnan Assignee: Subru Krishnan TestFederationRMStateStoreService is failing with null pointer exception. This looks to be a regression caused by YARN-5602 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6484) [Documentation] Documenting the YARN Federation feature
Subru Krishnan created YARN-6484: Summary: [Documentation] Documenting the YARN Federation feature Key: YARN-6484 URL: https://issues.apache.org/jira/browse/YARN-6484 Project: Hadoop YARN Issue Type: Sub-task Reporter: Subru Krishnan We should document the high level design and configuration to enable YARN Federation -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6363) Extending SLS: Synthetic Load Generator
[ https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-6363: --- Attachment: YARN-6363.v13.patch Documentation. > Extending SLS: Synthetic Load Generator > --- > > Key: YARN-6363 > URL: https://issues.apache.org/jira/browse/YARN-6363 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6363 overview.pdf, YARN-6363.v0.patch, > YARN-6363.v10.patch, YARN-6363.v11.patch, YARN-6363.v12.patch, > YARN-6363.v13.patch, YARN-6363.v1.patch, YARN-6363.v2.patch, > YARN-6363.v3.patch, YARN-6363.v4.patch, YARN-6363.v5.patch, > YARN-6363.v6.patch, YARN-6363.v7.patch, YARN-6363.v9.patch > > > This JIRA tracks the introduction of a synthetic load generator in the SLS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6472) Possible Java sandbox improvements
[
https://issues.apache.org/jira/browse/YARN-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969372#comment-15969372
]
Miklos Szegedi commented on YARN-6472:
--
Thank you, [~gphillips]. Is there any reason why nm-sandbox-policies is not a
subdirectory of nmPrivate?
> Possible Java sandbox improvements
> --
>
> Key: YARN-6472
> URL: https://issues.apache.org/jira/browse/YARN-6472
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Greg Phillips
> Attachments: YARN-6472.001.patch, YARN-6472.002.patch
>
>
> I set the sandbox to enforcing mode. Unfortunately I was able to break out of
> the sandbox running native code with the following command:
> {code}
> cmd = "$JAVA_HOME/bin/java %s -Xmx825955249
> org.apache.hadoop.yarn.applications.helloworld.HelloWorld `touch
> ../../helloworld`" + \
> " 1>/AppMaster.stdout 2>/AppMaster.stderr"
> $ ls .../nm-local-dir/usercache/root/appcache/
> helloworld
> {code}
> Also, if I am not using sandboxes, could we create the nm-sandbox-policies
> directory (empty) lazily?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6363) Extending SLS: Synthetic Load Generator
[ https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-6363: --- Attachment: YARN-6363.v12.patch Working towards yetus happiness and partial docs. > Extending SLS: Synthetic Load Generator > --- > > Key: YARN-6363 > URL: https://issues.apache.org/jira/browse/YARN-6363 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6363 overview.pdf, YARN-6363.v0.patch, > YARN-6363.v10.patch, YARN-6363.v11.patch, YARN-6363.v12.patch, > YARN-6363.v1.patch, YARN-6363.v2.patch, YARN-6363.v3.patch, > YARN-6363.v4.patch, YARN-6363.v5.patch, YARN-6363.v6.patch, > YARN-6363.v7.patch, YARN-6363.v9.patch > > > This JIRA tracks the introduction of a synthetic load generator in the SLS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6433) Only accessible cgroup mount directories should be selected for a controller
[
https://issues.apache.org/jira/browse/YARN-6433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Miklos Szegedi updated YARN-6433:
-
Attachment: YARN-6433.002.patch
> Only accessible cgroup mount directories should be selected for a controller
>
>
> Key: YARN-6433
> URL: https://issues.apache.org/jira/browse/YARN-6433
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-alpha3
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
> Attachments: YARN-6433.000.patch, YARN-6433.001.patch,
> YARN-6433.002.patch
>
>
> I have a Ubuntu16 box that returns the following error with pre-mounted
> cgroups on the latest trunk:
> {code}
> 2017-04-03 19:42:18,511 WARN
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsHandlerImpl:
> Cgroups not accessible /run/lxcfs/controllers/cpu,cpuacct
> {code}
> The version is:
> {code}
> $ uname -a
> Linux mybox 4.4.0-24-generic #43-Ubuntu SMP Wed Jun 8 19:27:37 UTC 2016
> x86_64 x86_64 x86_64 GNU/Linux
> {code}
> The following cpu cgroup filesystems are mounted:
> {code}
> $ grep cpuacct /etc/mtab
> cgroup /sys/fs/cgroup/cpu,cpuacct cgroup
> rw,nosuid,nodev,noexec,relatime,cpu,cpuacct,nsroot=/ 0 0
> cpu,cpuacct /run/lxcfs/controllers/cpu,cpuacct cgroup
> rw,relatime,cpu,cpuacct,nsroot=/ 0 0
> {code}
> /sys/fs/cgroup is accessible to my yarn user, so it should be selected
> instead of /run/lxcfs/controllers
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5331) Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968207#comment-15968207 ] Sean Po edited comment on YARN-5331 at 4/14/17 6:08 PM: Thanks for the patch [~ajsangeetha], it looks good to me. +1 YARN-5816 tracks TestDelegationTokenRenewer.testCancelWithMultipleAppSubmissions failure YARN-5548 tracks TestRMRestart.testFinishedAppRemovalAfterRMRestart failure was (Author: seanpo03): Thanks for the patch [~ajsangeetha], it looks good to me. +1 > Extend RLESparseResourceAllocation with period for supporting recurring > reservations in YARN ReservationSystem > -- > > Key: YARN-5331 > URL: https://issues.apache.org/jira/browse/YARN-5331 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Sangeetha Abdu Jyothi > Labels: oct16-medium > Attachments: YARN-5331.001.patch, YARN-5331.002.patch, > YARN-5331.003.patch, YARN-5331.004.patch, YARN-5331.005.patch, > YARN-5331.006.patch, YARN-5331.007.patch, YARN-5331.008.patch > > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to add a > PeriodicRLESparseResourceAllocation. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-3666) Federation Intercepting and propagating AM-RM communications
[ https://issues.apache.org/jira/browse/YARN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969335#comment-15969335 ] Botong Huang edited comment on YARN-3666 at 4/14/17 5:57 PM: - Hi [~giovanni.fumarola] and [~subru], the following test is failing after Yarn-5602 5eacf0cdb8bac8, can you guys take a look please? Thanks! testFederationStateStoreService(org.apache.hadoop.yarn.server.resourcemanager.federation.TestFederationRMStateStoreService) Time elapsed: 1.98 sec <<< ERROR! java.lang.NullPointerException: null at org.apache.hadoop.yarn.server.resourcemanager.federation.TestFederationRMStateStoreService.testFederationStateStoreService(TestFederationRMStateStoreService.java:93) The other failure from testCancelWithMultipleAppSubmissions is already tracked under YARN-5816. was (Author: botong): [~giovanni.fumarola] and [~subru], the following test is failing after Yarn-5602 5eacf0cdb8bac8, can you guys take a look please? Thanks! testFederationStateStoreService(org.apache.hadoop.yarn.server.resourcemanager.federation.TestFederationRMStateStoreService) Time elapsed: 1.98 sec <<< ERROR! java.lang.NullPointerException: null at org.apache.hadoop.yarn.server.resourcemanager.federation.TestFederationRMStateStoreService.testFederationStateStoreService(TestFederationRMStateStoreService.java:93) > Federation Intercepting and propagating AM-RM communications > > > Key: YARN-3666 > URL: https://issues.apache.org/jira/browse/YARN-3666 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Kishore Chaliparambil >Assignee: Botong Huang > Attachments: YARN-3666-YARN-2915.v1.patch, > YARN-3666-YARN-2915.v2.patch, YARN-3666-YARN-2915.v3.patch, > YARN-3666-YARN-2915.v4.patch, YARN-3666-YARN-2915.v5.patch > > > In order, to support transparent "spanning" of jobs across sub-clusters, all > AM-RM communications are proxied (via YARN-2884). > This JIRA tracks federation-specific mechanisms that decide how to > "split/broadcast" requests to the RMs and "merge" answers to > the AM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3666) Federation Intercepting and propagating AM-RM communications
[ https://issues.apache.org/jira/browse/YARN-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969335#comment-15969335 ] Botong Huang commented on YARN-3666: [~giovanni.fumarola] and [~subru], the following test is failing after Yarn-5602 5eacf0cdb8bac8, can you guys take a look please? Thanks! testFederationStateStoreService(org.apache.hadoop.yarn.server.resourcemanager.federation.TestFederationRMStateStoreService) Time elapsed: 1.98 sec <<< ERROR! java.lang.NullPointerException: null at org.apache.hadoop.yarn.server.resourcemanager.federation.TestFederationRMStateStoreService.testFederationStateStoreService(TestFederationRMStateStoreService.java:93) > Federation Intercepting and propagating AM-RM communications > > > Key: YARN-3666 > URL: https://issues.apache.org/jira/browse/YARN-3666 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Kishore Chaliparambil >Assignee: Botong Huang > Attachments: YARN-3666-YARN-2915.v1.patch, > YARN-3666-YARN-2915.v2.patch, YARN-3666-YARN-2915.v3.patch, > YARN-3666-YARN-2915.v4.patch, YARN-3666-YARN-2915.v5.patch > > > In order, to support transparent "spanning" of jobs across sub-clusters, all > AM-RM communications are proxied (via YARN-2884). > This JIRA tracks federation-specific mechanisms that decide how to > "split/broadcast" requests to the RMs and "merge" answers to > the AM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6483) Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned by the Resource Manager as a response to the Application Master heartbeat
[
https://issues.apache.org/jira/browse/YARN-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969326#comment-15969326
]
Hadoop QA commented on YARN-6483:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 13s{color}
| {color:red} YARN-6483 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6483 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863488/YARN-6483.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15641/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes
> returned by the Resource Manager as a response to the Application Master
> heartbeat
>
>
> Key: YARN-6483
> URL: https://issues.apache.org/jira/browse/YARN-6483
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.7.3
>Reporter: Juan Rodríguez Hortalá
> Attachments: YARN-6483.patch
>
>
> The DECOMMISSIONING node state is currently used as part of the graceful
> decommissioning mechanism to give time for tasks to complete in a node that
> is scheduled for decommission, and for reducer tasks to read the shuffle
> blocks in that node. Also, YARN effectively blacklists nodes in
> DECOMMISSIONING state by assigning them a capacity of 0, to prevent
> additional containers to be launched in those nodes, so no more shuffle
> blocks are written to the node. This blacklisting is not effective for
> applications like Spark, because a Spark executor running in a YARN container
> will keep receiving more tasks after the corresponding node has been
> blacklisted at the YARN level. We would like to propose a modification of the
> YARN heartbeat mechanism so nodes transitioning to DECOMMISSIONING are added
> to the list of updated nodes returned by the Resource Manager as a response
> to the Application Master heartbeat. This way a Spark application master
> would be able to blacklist a DECOMMISSIONING at the Spark level.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6483) Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned by the Resource Manager as a response to the Application Master heartbeat
[ https://issues.apache.org/jira/browse/YARN-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Rodríguez Hortalá updated YARN-6483: - Attachment: YARN-6483.patch > Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes > returned by the Resource Manager as a response to the Application Master > heartbeat > > > Key: YARN-6483 > URL: https://issues.apache.org/jira/browse/YARN-6483 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.7.3 >Reporter: Juan Rodríguez Hortalá > Attachments: YARN-6483.patch > > > The DECOMMISSIONING node state is currently used as part of the graceful > decommissioning mechanism to give time for tasks to complete in a node that > is scheduled for decommission, and for reducer tasks to read the shuffle > blocks in that node. Also, YARN effectively blacklists nodes in > DECOMMISSIONING state by assigning them a capacity of 0, to prevent > additional containers to be launched in those nodes, so no more shuffle > blocks are written to the node. This blacklisting is not effective for > applications like Spark, because a Spark executor running in a YARN container > will keep receiving more tasks after the corresponding node has been > blacklisted at the YARN level. We would like to propose a modification of the > YARN heartbeat mechanism so nodes transitioning to DECOMMISSIONING are added > to the list of updated nodes returned by the Resource Manager as a response > to the Application Master heartbeat. This way a Spark application master > would be able to blacklist a DECOMMISSIONING at the Spark level. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6483) Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned by the Resource Manager as a response to the Application Master heartbeat
Juan Rodríguez Hortalá created YARN-6483: Summary: Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned by the Resource Manager as a response to the Application Master heartbeat Key: YARN-6483 URL: https://issues.apache.org/jira/browse/YARN-6483 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 2.7.3 Reporter: Juan Rodríguez Hortalá The DECOMMISSIONING node state is currently used as part of the graceful decommissioning mechanism to give time for tasks to complete in a node that is scheduled for decommission, and for reducer tasks to read the shuffle blocks in that node. Also, YARN effectively blacklists nodes in DECOMMISSIONING state by assigning them a capacity of 0, to prevent additional containers to be launched in those nodes, so no more shuffle blocks are written to the node. This blacklisting is not effective for applications like Spark, because a Spark executor running in a YARN container will keep receiving more tasks after the corresponding node has been blacklisted at the YARN level. We would like to propose a modification of the YARN heartbeat mechanism so nodes transitioning to DECOMMISSIONING are added to the list of updated nodes returned by the Resource Manager as a response to the Application Master heartbeat. This way a Spark application master would be able to blacklist a DECOMMISSIONING at the Spark level. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6363) Extending SLS: Synthetic Load Generator
[
https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969276#comment-15969276
]
Hadoop QA commented on YARN-6363:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 26s{color} | {color:orange} hadoop-tools: The patch generated 136 new + 139
unchanged - 23 fixed = 275 total (was 162) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 2m
28s{color} | {color:green} The patch generated 0 new + 74 unchanged - 1 fixed =
74 total (was 75) {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
9s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
20s{color} | {color:green} hadoop-rumen in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
56s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
21s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | YARN-6363 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12863481/YARN-6363.v11.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml shellcheck shelldocs |
| uname | Linux 9da83ec36735 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / a41f8dd |
| Default Java | 1.8.0_121 |
| shellcheck | v0.4.6 |
| findbugs |
[jira] [Commented] (YARN-6396) Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization to decrease load for name node
[ https://issues.apache.org/jira/browse/YARN-6396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969274#comment-15969274 ] Robert Kanter commented on YARN-6396: - That's a good point about each NM doing this check for each upload - I can see that becoming a problem on large busy clusters. > Call verifyAndCreateRemoteLogDir at service initialization instead of > application initialization to decrease load for name node > --- > > Key: YARN-6396 > URL: https://issues.apache.org/jira/browse/YARN-6396 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 3.0.0-alpha2 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: YARN-6396.000.patch > > > Call verifyAndCreateRemoteLogDir at service initialization instead of > application initialization to decrease load for name node. > Currently for every application at each Node, verifyAndCreateRemoteLogDir > will be called before doing log aggregation, This will be a non trivial > overhead for name node in a large cluster since verifyAndCreateRemoteLogDir > calls getFileStatus. Once the remote log directory is created successfully, > it is not necessary to call it again. It will be better to call > verifyAndCreateRemoteLogDir at LogAggregationService service initialization. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5892) Capacity Scheduler: Support user-specific minimum user limit percent
[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969256#comment-15969256
]
Sunil G commented on YARN-5892:
---
bq.Yes, the user weight of 0 case is a special one that we would like to
explore, but certainly not in this JIRA.
Yes, we could discuss more about the definition in separate jira. For now,
could we avoid taking 0 as a valid configuration.
bq.These methods then multiply the value of userLimitResource by the
appropriate user weight(s), which will result in the correct value of userLimit
for that specific user
Sure. Since we multiply again later, we can get the same userlimit back. A
value less than one as weight is going to be trickier but we are rounding off
to minimumAllocation. So i think its fine given patch have logs and UI.
bq.
{code}
495 if (activeUsersSet.contains(userName)) {
496 user.setUserResourceLimit(userSpecificUserLimit);
497 }
{code}
If *activeUsersSet* has userName, i think *user* cant be null. Correct?
> Capacity Scheduler: Support user-specific minimum user limit percent
>
>
> Key: YARN-5892
> URL: https://issues.apache.org/jira/browse/YARN-5892
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacityscheduler
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: Active users highlighted.jpg, YARN-5892.001.patch,
> YARN-5892.002.patch, YARN-5892.003.patch, YARN-5892.004.patch,
> YARN-5892.005.patch, YARN-5892.006.patch, YARN-5892.007.patch,
> YARN-5892.008.patch, YARN-5892.009.patch
>
>
> Currently, in the capacity scheduler, the {{minimum-user-limit-percent}}
> property is per queue. A cluster admin should be able to set the minimum user
> limit percent on a per-user basis within the queue.
> This functionality is needed so that when intra-queue preemption is enabled
> (YARN-4945 / YARN-2113), some users can be deemed as more important than
> other users, and resources from VIP users won't be as likely to be preempted.
> For example, if the {{getstuffdone}} queue has a MULP of 25 percent, but user
> {{jane}} is a power user of queue {{getstuffdone}} and needs to be guaranteed
> 75 percent, the properties for {{getstuffdone}} and {{jane}} would look like
> this:
> {code}
>
>
> yarn.scheduler.capacity.root.getstuffdone.minimum-user-limit-percent
> 25
>
>
>
> yarn.scheduler.capacity.root.getstuffdone.jane.minimum-user-limit-percent
> 75
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6363) Extending SLS: Synthetic Load Generator
[
https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969231#comment-15969231
]
Carlo Curino commented on YARN-6363:
I have fixed and tested the commandline verifying both new and old options, and
nodes/no-nodes as follows:
{code}
# MANUALLY prepare copying source and data-files (with something like):
# cp -r /hadoop-dist/target/hadoop-${COMMON_VERSION} $BASEDIR
# cp /hadoop-tools/hadoop-sls/src/test/resources/*.json
$BASEDIR/data
BASEDIR=/home/ccurino/hadoop-deployed
COMMON_VERSION=3.0.0-alpha3-SNAPSHOT
export
HADOOP_CONF_DIR=$BASEDIR/hadoop-${COMMON_VERSION}/share/hadoop/tools/sls/sample-conf/
export
YARN_CONF_DIR=$BASEDIR/hadoop-${COMMON_VERSION}/share/hadoop/tools/sls/sample-conf/
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
export
PATH=$PATH:${BASEDIR}/hadoop-${COMMON_VERSION}/bin:${BASEDIR}/hadoop-${COMMON_VERSION}/sbin:${BASEDIR}/hadoop-${COMMON_VERSION}/share/hadoop/tools/sls/bin
# NO NODES OLD OPTIONS
timeout -s 9 60s slsrun.sh
--input-rumen=$BASEDIR/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json
--output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --input-sls=$BASEDIR/data/inputsls.json
--output-dir=/tmp/sls-out2
# NO NODES NEW OPTIONS
timeout -s 9 60s slsrun.sh --tracetype=RUMEN
--tracelocation=$BASEDIR/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json
--output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --tracetype=SLS
--tracelocation=$BASEDIR/data/inputsls.json --output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --tracetype=SYNTH
--tracelocation=$BASEDIR/data/syn.json --output-dir=/tmp/sls-out2
# WITH NODES OLD OPTIONS
timeout -s 9 60s slsrun.sh
--input-rumen=$BASEDIR/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json
--nodes=$BASEDIR/data/nodes.json --output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --input-sls=$BASEDIR/data/inputsls.json
--nodes=$BASEDIR/data/nodes.json --output-dir=/tmp/sls-out2
# WITH NODES NEW OPTIONS
timeout -s 9 60s slsrun.sh --tracetype=RUMEN
--tracelocation=$BASEDIR/hadoop-3.0.0-alpha3-SNAPSHOT/share/hadoop/tools/sls/sample-data/2jobs2min-rumen-jh.json
--nodes=$BASEDIR/data/nodes.json --output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --tracetype=SLS
--tracelocation=$BASEDIR/data/inputsls.json --nodes=$BASEDIR/data/nodes.json
--output-dir=/tmp/sls-out2
timeout -s 9 60s slsrun.sh --tracetype=SYNTH
--tracelocation=$BASEDIR/data/syn.json --nodes=$BASEDIR/data/nodes.json
--output-dir=/tmp/sls-out2
{code}
I ran and checked all the configurations below, using default configs from
share/tools/sls/sample-conf, and the data from sample-data and
src/test/resources/*.json.
Manually inspecting the runs things look good, RM comes up with Webapp, nodes
are loaded as expected, and apps get to run for all but RUMEN configurations
(more below).
The new TestSLSRunner also introduces more checks (with both schedulers). In
future JIRAs we can further extend this to leverage YARN-6451 to have a deeper
introspection/check of these runs.
The RUMEN version seems to have a problem parsing the rumen example trace
(nodes are loaded only from --nodes, and jobs are never loaded).
Turns out this problem is also present in trunk (running the TestSLSRunner does
bring the RM up, but doesn't run any job nor loads nodes)---I am opening a
separate JIRA to track this existing bug YARN-6482.
(Per [~subru] offline comment, I am converting the attached overview document
in additions to the SLS documentation, will upload soon)
> Extending SLS: Synthetic Load Generator
> ---
>
> Key: YARN-6363
> URL: https://issues.apache.org/jira/browse/YARN-6363
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Attachments: YARN-6363 overview.pdf, YARN-6363.v0.patch,
> YARN-6363.v10.patch, YARN-6363.v11.patch, YARN-6363.v1.patch,
> YARN-6363.v2.patch, YARN-6363.v3.patch, YARN-6363.v4.patch,
> YARN-6363.v5.patch, YARN-6363.v6.patch, YARN-6363.v7.patch, YARN-6363.v9.patch
>
>
> This JIRA tracks the introduction of a synthetic load generator in the SLS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5881) Enable configuration of queue capacity in terms of absolute resources
[
https://issues.apache.org/jira/browse/YARN-5881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969225#comment-15969225
]
Sunil G commented on YARN-5881:
---
Thanks [~leftnoteasy].
Yes, we can have that linear approach as you mentioned. I chose a flatten
approach to avoid confusion of ordering within group.
So could we choose like below
{code}
yarn.scheduler.capacity.root.default.min-resource
[memory=10Gi,vcores=2]
{code}
We will try to keep units in sync with resource profile branch as we can reuse
unit conversion code and it will standard across.
Thoughts?
> Enable configuration of queue capacity in terms of absolute resources
> -
>
> Key: YARN-5881
> URL: https://issues.apache.org/jira/browse/YARN-5881
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Sean Po
>Assignee: Wangda Tan
> Attachments:
> YARN-5881.Support.Absolute.Min.Max.Resource.In.Capacity.Scheduler.design-doc.v1.pdf
>
>
> Currently, Yarn RM supports the configuration of queue capacity in terms of a
> proportion to cluster capacity. In the context of Yarn being used as a public
> cloud service, it makes more sense if queues can be configured absolutely.
> This will allow administrators to set usage limits more concretely and
> simplify customer expectations for cluster allocation.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6363) Extending SLS: Synthetic Load Generator
[ https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-6363: --- Attachment: YARN-6363.v11.patch > Extending SLS: Synthetic Load Generator > --- > > Key: YARN-6363 > URL: https://issues.apache.org/jira/browse/YARN-6363 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6363 overview.pdf, YARN-6363.v0.patch, > YARN-6363.v10.patch, YARN-6363.v11.patch, YARN-6363.v1.patch, > YARN-6363.v2.patch, YARN-6363.v3.patch, YARN-6363.v4.patch, > YARN-6363.v5.patch, YARN-6363.v6.patch, YARN-6363.v7.patch, YARN-6363.v9.patch > > > This JIRA tracks the introduction of a synthetic load generator in the SLS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5994) TestCapacityScheduler.testAMLimitUsage fails intermittently
[
https://issues.apache.org/jira/browse/YARN-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969087#comment-15969087
]
Eric Payne edited comment on YARN-5994 at 4/14/17 4:19 PM:
---
Thanks [~ebadger] for reporting and fixing this issue.
The only thing I found is that the patch does not build in branch-2 or
branch-2.8 because the anonymous inner class returned by
{{GenericTestUtils.waitFor}} in
{{TestCapacityScheduler#verifyAMLimitForLeafQueue}} uses both of the
{{nodeMemory}} and {{scheduler}} local variables. Since both branch-2 and
branch-2.8 are compiling with java 1.7, these variables need to be final.
The patch looks good to me. I give my +1. I will commit this soon if there are
no objections, and I will fix the {{final}} issues as part of the backport to
branch-2 and branch-2.8.
was (Author: eepayne):
Thanks [~ebadger] for reporting and fixing this issue.
The only thing I found is that the patch does not build in branch-2 or
branch-2.8 because the anonymous inner class returned by
{{GenericTestUtils.waitFor}} in
{{TestCapacityScheduler#verifyAMLimitForLeafQueue}} uses both of the
{{nodeMemory}} and {{scheduler}} local variables. Since both branch-2 and
branch-2.8 are compiling with java 2.7, these variables need to be final.
The patch looks good to me. I give my +1. I will commit this soon if there are
no objections, and I will fix the {{final}} issues as part of the backport to
branch-2 and branch-2.8.
> TestCapacityScheduler.testAMLimitUsage fails intermittently
> ---
>
> Key: YARN-5994
> URL: https://issues.apache.org/jira/browse/YARN-5994
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler-output.txt,
> YARN-5994.001.patch
>
>
> {noformat}
> java.lang.AssertionError: app shouldn't be null
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:169)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.submitApp(MockRM.java:577)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.submitApp(MockRM.java:488)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.submitApp(MockRM.java:395)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.verifyAMLimitForLeafQueue(TestCapacityScheduler.java:3389)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testAMLimitUsage(TestCapacityScheduler.java:3251)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5994) TestCapacityScheduler.testAMLimitUsage fails intermittently
[
https://issues.apache.org/jira/browse/YARN-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969222#comment-15969222
]
Hudson commented on YARN-5994:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11585 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11585/])
YARN-5994. TestCapacityScheduler.testAMLimitUsage fails intermittently.
(epayne: rev a41f8dd58e27d8835fbb64eeaba5d7416df0499d)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
> TestCapacityScheduler.testAMLimitUsage fails intermittently
> ---
>
> Key: YARN-5994
> URL: https://issues.apache.org/jira/browse/YARN-5994
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler-output.txt,
> YARN-5994.001.patch
>
>
> {noformat}
> java.lang.AssertionError: app shouldn't be null
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:169)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.submitApp(MockRM.java:577)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.submitApp(MockRM.java:488)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.submitApp(MockRM.java:395)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.verifyAMLimitForLeafQueue(TestCapacityScheduler.java:3389)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testAMLimitUsage(TestCapacityScheduler.java:3251)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6482) TestSLSRunner runs but doesn't executed jobs (.json parsing issue)
Carlo Curino created YARN-6482: -- Summary: TestSLSRunner runs but doesn't executed jobs (.json parsing issue) Key: YARN-6482 URL: https://issues.apache.org/jira/browse/YARN-6482 Project: Hadoop YARN Issue Type: Bug Reporter: Carlo Curino Priority: Minor The TestSLSRunner runs correctly brining up and RM, but the parsing of the rumen trace fails somehow silently, and no nodes nor jobs are loaded. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6396) Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization to decrease load for name node
[ https://issues.apache.org/jira/browse/YARN-6396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969197#comment-15969197 ] zhihai xu edited comment on YARN-6396 at 4/14/17 4:02 PM: -- Thanks for the review [~jianhe] and [~rkanter]! if some one deletes the remote log dir, all the old log will disappear. That will be a more serious issue, recreating the remote log dir won't save the old log data. This looks like a monitor problem, I think it will be better to do it in some tool outside the NM. It will be more efficient to do it at one place instead of on each NM, which could be many thousands in a large cluster. Yes, it's a trade off between validation and efficiency. Also restarting the NM will help recreate the remote log dir. was (Author: zxu): Thanks for the review [~jianhe] and [~rkanter]! if some one deletes the remote log dir, all the old log will disappear. That will be a more serious issue, recreating the remote log dir won't save the old log data. This looks like a monitor problem, I think it will be better to do it in some tool outside the NM. It will be more efficient to do it at one place instead of on each NM, which could be many thousands in a large cluster. Yes, it's a trade off between validation and efficiency. > Call verifyAndCreateRemoteLogDir at service initialization instead of > application initialization to decrease load for name node > --- > > Key: YARN-6396 > URL: https://issues.apache.org/jira/browse/YARN-6396 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 3.0.0-alpha2 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: YARN-6396.000.patch > > > Call verifyAndCreateRemoteLogDir at service initialization instead of > application initialization to decrease load for name node. > Currently for every application at each Node, verifyAndCreateRemoteLogDir > will be called before doing log aggregation, This will be a non trivial > overhead for name node in a large cluster since verifyAndCreateRemoteLogDir > calls getFileStatus. Once the remote log directory is created successfully, > it is not necessary to call it again. It will be better to call > verifyAndCreateRemoteLogDir at LogAggregationService service initialization. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6396) Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization to decrease load for name node
[ https://issues.apache.org/jira/browse/YARN-6396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969199#comment-15969199 ] Haibo Chen commented on YARN-6396: -- Robert and Jian's comments remind me of that we have seen customers delete the log directory every once in a while to clean up, which I think is a fair thing to do. Maybe we want to make it configurable (true by default) whether we want to call verifyAndCreateRemoteLogDir at LogAggregationService service initialization? > Call verifyAndCreateRemoteLogDir at service initialization instead of > application initialization to decrease load for name node > --- > > Key: YARN-6396 > URL: https://issues.apache.org/jira/browse/YARN-6396 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 3.0.0-alpha2 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: YARN-6396.000.patch > > > Call verifyAndCreateRemoteLogDir at service initialization instead of > application initialization to decrease load for name node. > Currently for every application at each Node, verifyAndCreateRemoteLogDir > will be called before doing log aggregation, This will be a non trivial > overhead for name node in a large cluster since verifyAndCreateRemoteLogDir > calls getFileStatus. Once the remote log directory is created successfully, > it is not necessary to call it again. It will be better to call > verifyAndCreateRemoteLogDir at LogAggregationService service initialization. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6396) Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization to decrease load for name node
[ https://issues.apache.org/jira/browse/YARN-6396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969197#comment-15969197 ] zhihai xu commented on YARN-6396: - Thanks for the review [~jianhe] and [~rkanter]! if some one deletes the remote log dir, all the old log will disappear. That will be a more serious issue, recreating the remote log dir won't save the old log data. This looks like a monitor problem, I think it will be better to do it in some tool outside the NM. It will be more efficient to do it at one place instead of on each NM, which could be many thousands in a large cluster. Yes, it's a trade off between validation and efficiency. > Call verifyAndCreateRemoteLogDir at service initialization instead of > application initialization to decrease load for name node > --- > > Key: YARN-6396 > URL: https://issues.apache.org/jira/browse/YARN-6396 > Project: Hadoop YARN > Issue Type: Improvement > Components: log-aggregation >Affects Versions: 3.0.0-alpha2 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Minor > Attachments: YARN-6396.000.patch > > > Call verifyAndCreateRemoteLogDir at service initialization instead of > application initialization to decrease load for name node. > Currently for every application at each Node, verifyAndCreateRemoteLogDir > will be called before doing log aggregation, This will be a non trivial > overhead for name node in a large cluster since verifyAndCreateRemoteLogDir > calls getFileStatus. Once the remote log directory is created successfully, > it is not necessary to call it again. It will be better to call > verifyAndCreateRemoteLogDir at LogAggregationService service initialization. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5994) TestCapacityScheduler.testAMLimitUsage fails intermittently
[
https://issues.apache.org/jira/browse/YARN-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969151#comment-15969151
]
Hadoop QA commented on YARN-5994:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 195 unchanged - 0 fixed = 196 total (was 195)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 39m
19s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:612578f |
| JIRA Issue | YARN-5994 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12848073/YARN-5994.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ff757257d637 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 0cab572 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/15639/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15639/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15639/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> TestCapacityScheduler.testAMLimitUsage fails intermittently
> ---
>
> Key: YARN-5994
>
[jira] [Commented] (YARN-2985) YARN should support to delete the aggregated logs for Non-MapReduce applications
[ https://issues.apache.org/jira/browse/YARN-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969088#comment-15969088 ] Jason Lowe commented on YARN-2985: -- Doing a config for branch-2 seems reasonable. bq. my understanding is that the timeline server is supposed to replace the JHS, even for deployments that only run MR jobs This is news to me. The timeline server has no UI, just REST APIs, so there minimally needs to be something that provides the javascript necessary for the client browser to render a MapReduce-aware UI. I haven't seen that in trunk, and without it the MapReduce JHS must still be running if there's going to be a MapReduce UI for completed jobs. Even without the timeline server completely replacing the MR JHS, the deletion service can still be moved in trunk without a config under the following conditions: * The timeline server is considered a critical server that always needs to be running (or we simply document that it must be used when log aggregation is enabled) * There's an equivalent way to refresh the config options like can be done with the deletion service in the MR JHS today. > YARN should support to delete the aggregated logs for Non-MapReduce > applications > > > Key: YARN-2985 > URL: https://issues.apache.org/jira/browse/YARN-2985 > Project: Hadoop YARN > Issue Type: New Feature > Components: log-aggregation, nodemanager >Affects Versions: 2.8.0 >Reporter: Xu Yang >Assignee: Steven Rand > Attachments: YARN-2985-branch-2-001.patch > > > Before Hadoop 2.6, the LogAggregationService is started in NodeManager. But > the AggregatedLogDeletionService is started in mapreduce`s JobHistoryServer. > Therefore, the Non-MapReduce application can aggregate their logs to HDFS, > but can not delete those logs. Need the NodeManager take over the function of > aggregated log deletion. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5994) TestCapacityScheduler.testAMLimitUsage fails intermittently
[
https://issues.apache.org/jira/browse/YARN-5994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969087#comment-15969087
]
Eric Payne commented on YARN-5994:
--
Thanks [~ebadger] for reporting and fixing this issue.
The only thing I found is that the patch does not build in branch-2 or
branch-2.8 because the anonymous inner class returned by
{{GenericTestUtils.waitFor}} in
{{TestCapacityScheduler#verifyAMLimitForLeafQueue}} uses both of the
{{nodeMemory}} and {{scheduler}} local variables. Since both branch-2 and
branch-2.8 are compiling with java 2.7, these variables need to be final.
The patch looks good to me. I give my +1. I will commit this soon if there are
no objections, and I will fix the {{final}} issues as part of the backport to
branch-2 and branch-2.8.
> TestCapacityScheduler.testAMLimitUsage fails intermittently
> ---
>
> Key: YARN-5994
> URL: https://issues.apache.org/jira/browse/YARN-5994
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler-output.txt,
> YARN-5994.001.patch
>
>
> {noformat}
> java.lang.AssertionError: app shouldn't be null
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:169)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.submitApp(MockRM.java:577)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.submitApp(MockRM.java:488)
> at
> org.apache.hadoop.yarn.server.resourcemanager.MockRM.submitApp(MockRM.java:395)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.verifyAMLimitForLeafQueue(TestCapacityScheduler.java:3389)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testAMLimitUsage(TestCapacityScheduler.java:3251)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-3839) Quit throwing NMNotYetReadyException
[ https://issues.apache.org/jira/browse/YARN-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reassigned YARN-3839: Assignee: Manikandan R > Quit throwing NMNotYetReadyException > > > Key: YARN-3839 > URL: https://issues.apache.org/jira/browse/YARN-3839 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Karthik Kambatla >Assignee: Manikandan R > > Quit throwing NMNotYetReadyException when NM has not yet registered with the > RM. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6462) Add yarn command to list all queues
[ https://issues.apache.org/jira/browse/YARN-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968883#comment-15968883 ] Shen Yinjie edited comment on YARN-6462 at 4/14/17 10:39 AM: - [~rohithsharma],[~sunilg] beg your review..:) was (Author: shenyinjie): [~rohithsharma],[~sunilg] bug your review..:) > Add yarn command to list all queues > --- > > Key: YARN-6462 > URL: https://issues.apache.org/jira/browse/YARN-6462 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Reporter: Shen Yinjie > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6462_1.patch, YARN-6462_2.patch > > > we need a yarn command to list all queues ,and there is this kind of command > for applications and nodemangers already... -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6462) Add yarn command to list all queues
[ https://issues.apache.org/jira/browse/YARN-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968883#comment-15968883 ] Shen Yinjie commented on YARN-6462: --- [~rohithsharma],[~sunilg] bug your review..:) > Add yarn command to list all queues > --- > > Key: YARN-6462 > URL: https://issues.apache.org/jira/browse/YARN-6462 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Reporter: Shen Yinjie > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6462_1.patch, YARN-6462_2.patch > > > we need a yarn command to list all queues ,and there is this kind of command > for applications and nodemangers already... -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6289) Fail to achieve data locality when runing MapReduce and Spark on HDFS
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968691#comment-15968691 ] Huangkaixuan edited comment on YARN-6289 at 4/14/17 6:33 AM: - Thanks, [~leftnoteasy]. It seems works for me. I noticed the patch has been merged into Hadoop 3.0 snapshot. Acutally, I am using Hadoop 2.7 and 2.8. Is there a way to port this patch to 2.7 and 2.8? was (Author: huangkx6810): Thanks, [~leftnoteasy]. It seems work for me. I noticed the patch has been merged into Hadoop 3.0 snapshot. Acutally, I am using Hadoop 2.7 and 2.8. Is there a way to port this patch to 2.7 and 2.8? > Fail to achieve data locality when runing MapReduce and Spark on HDFS > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-scheduling > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan > Attachments: Hadoop_Spark_Conf.zip, YARN-DataLocality.docx, > YARN-RackAwareness.docx > > > When running a simple wordcount experiment on YARN, I noticed that the task > failed to achieve data locality, even though there is no other job running on > the cluster at the same time. The experiment was done in a 7-node (1 master, > 6 data nodes/node managers) cluster and the input of the wordcount job (both > Spark and MapReduce) is a single-block file in HDFS which is two-way > replicated (replication factor = 2). I ran wordcount on YARN for 10 times. > The results show that only 30% of tasks can achieve data locality, which > seems like the result of a random placement of tasks. The experiment details > are in the attachment, and feel free to reproduce the experiments. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6289) Fail to achieve data locality when runing MapReduce and Spark on HDFS
[ https://issues.apache.org/jira/browse/YARN-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968691#comment-15968691 ] Huangkaixuan commented on YARN-6289: Thanks, [~leftnoteasy]. It seems work for me. I noticed the patch has been merged into Hadoop 3.0 snapshot. Acutally, I am using Hadoop 2.7 and 2.8. Is there a way to port this patch to 2.7 and 2.8? > Fail to achieve data locality when runing MapReduce and Spark on HDFS > - > > Key: YARN-6289 > URL: https://issues.apache.org/jira/browse/YARN-6289 > Project: Hadoop YARN > Issue Type: Bug > Components: distributed-scheduling > Environment: Hardware configuration > CPU: 2 x Intel(R) Xeon(R) E5-2620 v2 @ 2.10GHz /15M Cache 6-Core 12-Thread > Memory: 128GB Memory (16x8GB) 1600MHz > Disk: 600GBx2 3.5-inch with RAID-1 > Network bandwidth: 968Mb/s > Software configuration > Spark-1.6.2 Hadoop-2.7.1 >Reporter: Huangkaixuan > Attachments: Hadoop_Spark_Conf.zip, YARN-DataLocality.docx, > YARN-RackAwareness.docx > > > When running a simple wordcount experiment on YARN, I noticed that the task > failed to achieve data locality, even though there is no other job running on > the cluster at the same time. The experiment was done in a 7-node (1 master, > 6 data nodes/node managers) cluster and the input of the wordcount job (both > Spark and MapReduce) is a single-block file in HDFS which is two-way > replicated (replication factor = 2). I ran wordcount on YARN for 10 times. > The results show that only 30% of tasks can achieve data locality, which > seems like the result of a random placement of tasks. The experiment details > are in the attachment, and feel free to reproduce the experiments. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6344) Add parameter for rack locality delay in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968690#comment-15968690
]
Huangkaixuan commented on YARN-6344:
Thanks [~kkaranasos]
I noticed the patch has been merged into Hadoop 3.0 snapshot. Acutally, I am
using Hadoop 2.7 and 2.8. Is there a way to port this patch to 2.7 and 2.8?
> Add parameter for rack locality delay in CapacityScheduler
> --
>
> Key: YARN-6344
> URL: https://issues.apache.org/jira/browse/YARN-6344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-6344.001.patch, YARN-6344.002.patch,
> YARN-6344.003.patch, YARN-6344.004.patch
>
>
> When relaxing locality from node to rack, the {{node-locality-parameter}} is
> used: when scheduling opportunities for a scheduler key are more than the
> value of this parameter, we relax locality and try to assign the container to
> a node in the corresponding rack.
> On the other hand, when relaxing locality to off-switch (i.e., assign the
> container anywhere in the cluster), we are using a {{localityWaitFactor}},
> which is computed based on the number of outstanding requests for a specific
> scheduler key, which is divided by the size of the cluster.
> In case of applications that request containers in big batches (e.g.,
> traditional MR jobs), and for relatively small clusters, the
> localityWaitFactor does not affect relaxing locality much.
> However, in case of applications that request containers in small batches,
> this load factor takes a very small value, which leads to assigning
> off-switch containers too soon. This situation is even more pronounced in big
> clusters.
> For example, if an application requests only one container per request, the
> locality will be relaxed after a single missed scheduling opportunity.
> The purpose of this JIRA is to rethink the way we are relaxing locality for
> off-switch assignments.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
