[jira] [Updated] (YARN-7346) Fix compilation errors against hbase2 alpha release
[ https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-7346: Issue Type: Sub-task (was: Bug) Parent: YARN-7213 > Fix compilation errors against hbase2 alpha release > --- > > Key: YARN-7346 > URL: https://issues.apache.org/jira/browse/YARN-7346 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Vrushali C > > When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, > I got the following errors: > https://pastebin.com/Ms4jYEVB > This issue is to fix the compilation errors. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6927) Add support for individual resource types requests in MapReduce
[ https://issues.apache.org/jira/browse/YARN-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208845#comment-16208845 ] Sunil G commented on YARN-6927: --- Yes. Thats correct. +1 on this. > Add support for individual resource types requests in MapReduce > --- > > Key: YARN-6927 > URL: https://issues.apache.org/jira/browse/YARN-6927 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Daniel Templeton >Assignee: Gergo Repas > Attachments: YARN-6927.000.patch, YARN-6927.001.patch > > > YARN-6504 adds support for resource profiles in MapReduce jobs, but resource > profiles don't give users much flexibility in their resource requests. To > satisfy users' needs, MapReduce should also allow users to specify arbitrary > resource requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6927) Add support for individual resource types requests in MapReduce
[ https://issues.apache.org/jira/browse/YARN-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208840#comment-16208840 ] Daniel Templeton commented on YARN-6927: Sounds fine to me, [~leftnoteasy]. > Add support for individual resource types requests in MapReduce > --- > > Key: YARN-6927 > URL: https://issues.apache.org/jira/browse/YARN-6927 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Daniel Templeton >Assignee: Gergo Repas > Attachments: YARN-6927.000.patch, YARN-6927.001.patch > > > YARN-6504 adds support for resource profiles in MapReduce jobs, but resource > profiles don't give users much flexibility in their resource requests. To > satisfy users' needs, MapReduce should also allow users to specify arbitrary > resource requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7338) Support same origin policy for cross site scripting prevention.
[ https://issues.apache.org/jira/browse/YARN-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208839#comment-16208839 ] Wangda Tan commented on YARN-7338: -- Thanks [~sunil.gov...@gmail.com] for promptly working on this patch. [~eyang] do you have any suggestions to the attached patch? > Support same origin policy for cross site scripting prevention. > --- > > Key: YARN-7338 > URL: https://issues.apache.org/jira/browse/YARN-7338 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Vrushali C > Attachments: YARN-7338.001.patch > > > Opening jira as suggested b [~eyang] on the thread for merging YARN-3368 (new > web UI) to branch2 > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201610.mbox/%3ccad++ecmvvqnzqz9ynkvkcxaczdkg50yiofxktgk3mmms9sh...@mail.gmail.com%3E > -- > Ui2 does not seem to support same origin policy for cross site scripting > prevention. > The following parameters has no effect for /ui2: > hadoop.http.cross-origin.enabled = true > yarn.resourcemanager.webapp.cross-origin.enabled = true > This is because ui2 is designed as a separate web application. WebFilters > setup for existing resource manager doesn’t apply to the new web application. > Please open JIRA to track the security issue and resolve the problem prior to > backporting this to branch-2. > This would minimize the risk to open up security hole in branch-2. > -- -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7338) Support same origin policy for cross site scripting prevention.
[
https://issues.apache.org/jira/browse/YARN-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208819#comment-16208819
]
Hadoop QA commented on YARN-7338:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
30s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7338 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892728/YARN-7338.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 2e9893d95a83 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 2523e1c |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18003/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18003/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Support same origin policy for cross site scripting prevention.
> ---
>
>
[jira] [Commented] (YARN-6608) Backport all SLS improvements from trunk to branch-2
[ https://issues.apache.org/jira/browse/YARN-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208818#comment-16208818 ] Yufei Gu commented on YARN-6608: Great! Thanks for woking on this [~curino] and [~wangda]! > Backport all SLS improvements from trunk to branch-2 > > > Key: YARN-6608 > URL: https://issues.apache.org/jira/browse/YARN-6608 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.9.0 >Reporter: Carlo Curino >Assignee: Carlo Curino > Fix For: 2.9.0 > > Attachments: YARN-6608-branch-2.v0.patch, > YARN-6608-branch-2.v1.patch, YARN-6608-branch-2.v2.patch, > YARN-6608-branch-2.v3.patch, YARN-6608-branch-2.v4.patch, > YARN-6608-branch-2.v5.patch, YARN-6608-branch-2.v6.patch, > YARN-6608-branch-2.v7.patch, YARN-6608-branch-2.v8.patch, > YARN-6608-branch-2.v9.patch > > > The SLS has received lots of attention in trunk, but only some of it made it > back to branch-2. This patch is a "raw" fork-lift of the trunk development > from hadoop-tools/hadoop-sls. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7338) Support same origin policy for cross site scripting prevention.
[ https://issues.apache.org/jira/browse/YARN-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208789#comment-16208789 ] Vrushali C commented on YARN-7338: -- Thanks [~sunil.gov...@gmail.com] for the patch! [~eyang] what are your thoughts on the proposed patch? As you probably know, we are trying to back-port the UI to branch2 so that we can include it in the 2.9 release. The code freeze of that is this friday and I know it's holiday season in India right now, so really appreciate the prompt responses very much [~sunil.gov...@gmail.com]! > Support same origin policy for cross site scripting prevention. > --- > > Key: YARN-7338 > URL: https://issues.apache.org/jira/browse/YARN-7338 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Vrushali C > Attachments: YARN-7338.001.patch > > > Opening jira as suggested b [~eyang] on the thread for merging YARN-3368 (new > web UI) to branch2 > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201610.mbox/%3ccad++ecmvvqnzqz9ynkvkcxaczdkg50yiofxktgk3mmms9sh...@mail.gmail.com%3E > -- > Ui2 does not seem to support same origin policy for cross site scripting > prevention. > The following parameters has no effect for /ui2: > hadoop.http.cross-origin.enabled = true > yarn.resourcemanager.webapp.cross-origin.enabled = true > This is because ui2 is designed as a separate web application. WebFilters > setup for existing resource manager doesn’t apply to the new web application. > Please open JIRA to track the security issue and resolve the problem prior to > backporting this to branch-2. > This would minimize the risk to open up security hole in branch-2. > -- -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7338) Support same origin policy for cross site scripting prevention.
[
https://issues.apache.org/jira/browse/YARN-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208745#comment-16208745
]
Sunil G edited comment on YARN-7338 at 10/18/17 3:07 AM:
-
This patch has filters added to ui2.
Filters are as below
{noformat}
HW12715:sbin sunilgovindan$ curl -i http://localhost:8088/ui2/
HTTP/1.1 200 OK
Date: Wed, 18 Oct 2017 03:04:10 GMT
X-Frame-Options: SAMEORIGIN
Last-Modified: Tue, 17 Oct 2017 02:27:57 GMT
Content-Type: text/html
Accept-Ranges: bytes
Content-Length: 1691
YARN
HW12715:sbin sunilgovindan$
{noformat}
was (Author: sunilg):
This patch has filters added to ui2
> Support same origin policy for cross site scripting prevention.
> ---
>
> Key: YARN-7338
> URL: https://issues.apache.org/jira/browse/YARN-7338
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Vrushali C
> Attachments: YARN-7338.001.patch
>
>
> Opening jira as suggested b [~eyang] on the thread for merging YARN-3368 (new
> web UI) to branch2
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201610.mbox/%3ccad++ecmvvqnzqz9ynkvkcxaczdkg50yiofxktgk3mmms9sh...@mail.gmail.com%3E
> --
> Ui2 does not seem to support same origin policy for cross site scripting
> prevention.
> The following parameters has no effect for /ui2:
> hadoop.http.cross-origin.enabled = true
> yarn.resourcemanager.webapp.cross-origin.enabled = true
> This is because ui2 is designed as a separate web application. WebFilters
> setup for existing resource manager doesn’t apply to the new web application.
> Please open JIRA to track the security issue and resolve the problem prior to
> backporting this to branch-2.
> This would minimize the risk to open up security hole in branch-2.
> --
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7244) ShuffleHandler is not aware of disks that are added
[
https://issues.apache.org/jira/browse/YARN-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208749#comment-16208749
]
Hadoop QA commented on YARN-7244:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} root: The patch generated 0 new + 317 unchanged - 2
fixed = 317 total (was 319) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
42s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 10s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-mapreduce-client-shuffle in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}119m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7244 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892714/YARN-7244.013.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b20209a04aaa 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Updated] (YARN-7338) Support same origin policy for cross site scripting prevention.
[ https://issues.apache.org/jira/browse/YARN-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7338: -- Attachment: YARN-7338.001.patch This patch has filters added to ui2 > Support same origin policy for cross site scripting prevention. > --- > > Key: YARN-7338 > URL: https://issues.apache.org/jira/browse/YARN-7338 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Vrushali C > Attachments: YARN-7338.001.patch > > > Opening jira as suggested b [~eyang] on the thread for merging YARN-3368 (new > web UI) to branch2 > http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201610.mbox/%3ccad++ecmvvqnzqz9ynkvkcxaczdkg50yiofxktgk3mmms9sh...@mail.gmail.com%3E > -- > Ui2 does not seem to support same origin policy for cross site scripting > prevention. > The following parameters has no effect for /ui2: > hadoop.http.cross-origin.enabled = true > yarn.resourcemanager.webapp.cross-origin.enabled = true > This is because ui2 is designed as a separate web application. WebFilters > setup for existing resource manager doesn’t apply to the new web application. > Please open JIRA to track the security issue and resolve the problem prior to > backporting this to branch-2. > This would minimize the risk to open up security hole in branch-2. > -- -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7338) Support same origin policy for cross site scripting prevention.
[
https://issues.apache.org/jira/browse/YARN-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208739#comment-16208739
]
Sunil G commented on YARN-7338:
---
Thanks [~eyang] for sharing more points here.
{noformat}
Request URL:https://tags.tiqcdn.com/utag/bofa/main/prod/utag.js
Request Method:GET
Status Code:200
Remote Address:104.121.229.167:443
Referrer Policy:no-referrer-when-downgrade
{noformat}
Response Headers
{noformat}
accept-ranges:bytes
cache-control:max-age=300
content-encoding:gzip
content-length:27343
content-type:application/x-javascript
date:Wed, 18 Oct 2017 02:17:54 GMT
etag:"bc10f1dc838dfe4d03f3e9d5c204f760:1506620434"
expires:Wed, 18 Oct 2017 02:22:54 GMT
last-modified:Thu, 28 Sep 2017 17:40:34 GMT
server:Apache
status:200
vary:Accept-Encoding
{noformat}
When I referred various static contents from BOA or other sites, I saw mostly
responses like above. But the link which u shared has
{{Access-Control-Allow-Credentials:true}}. I am not sure why this is not there
for all pages. please correct me if I missed some other headers here.
I am tying to phrase the potential security threat model explained by you.
# Third party js libs used to compile some *.js* files of ui2 has harmful
contents.
# UI2 browser end point downloads whole contents from server to its cache.
# This content is hence in client end (NOT at server end). Could this impact
the REST response coming from NM or RM which is already protected by XFS and
other filters?
I am also in line with you where we have to protect contents before getting any
issue. I checked code and there is a way which this filters could be added.
However this detailed discussion will help folks to re-iterate on real reason
why static contents needs cors protection as few statics were not protected in
general.
> Support same origin policy for cross site scripting prevention.
> ---
>
> Key: YARN-7338
> URL: https://issues.apache.org/jira/browse/YARN-7338
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Vrushali C
>
> Opening jira as suggested b [~eyang] on the thread for merging YARN-3368 (new
> web UI) to branch2
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201610.mbox/%3ccad++ecmvvqnzqz9ynkvkcxaczdkg50yiofxktgk3mmms9sh...@mail.gmail.com%3E
> --
> Ui2 does not seem to support same origin policy for cross site scripting
> prevention.
> The following parameters has no effect for /ui2:
> hadoop.http.cross-origin.enabled = true
> yarn.resourcemanager.webapp.cross-origin.enabled = true
> This is because ui2 is designed as a separate web application. WebFilters
> setup for existing resource manager doesn’t apply to the new web application.
> Please open JIRA to track the security issue and resolve the problem prior to
> backporting this to branch-2.
> This would minimize the risk to open up security hole in branch-2.
> --
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6927) Add support for individual resource types requests in MapReduce
[ https://issues.apache.org/jira/browse/YARN-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208733#comment-16208733 ] Wangda Tan commented on YARN-6927: -- Since it might be hard to get rid of mapred-default.xml, I suggest to set an overwrite rule instead, new config should overwrite the old config (and print warning to console). Sounds like a plan? > Add support for individual resource types requests in MapReduce > --- > > Key: YARN-6927 > URL: https://issues.apache.org/jira/browse/YARN-6927 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Daniel Templeton >Assignee: Gergo Repas > Attachments: YARN-6927.000.patch, YARN-6927.001.patch > > > YARN-6504 adds support for resource profiles in MapReduce jobs, but resource > profiles don't give users much flexibility in their resource requests. To > satisfy users' needs, MapReduce should also allow users to specify arbitrary > resource requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
[ https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208709#comment-16208709 ] Vrushali C commented on YARN-7169: -- [~wangda] [~sunil.gov...@gmail.com] would appreciate your review / thoughts on this latest patch against branch-2. The code is in branch YARN-3368_branch2. > Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2) > -- > > Key: YARN-7169 > URL: https://issues.apache.org/jira/browse/YARN-7169 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C >Priority: Critical > Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png, > YARN-7169-YARN-3368_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0002.patch, > YARN-7169-YARN-5355_branch2.0003.patch, > YARN-7169-YARN-5355_branch2.0004.patch, > YARN-7169-YARN-5355_branch2.0004.patch, YARN-7169-branch-2.0001.patch, > ui_commits(1) > > > Jira to track the backport of the new yarn-ui onto branch2. Right now adding > into Timeline Service v2's branch2 which is YARN-5355_branch2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
[
https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208689#comment-16208689
]
Hadoop QA commented on YARN-7169:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m
54s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn .
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
56s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
6s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 56s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
47s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}145m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.http.TestHttpServer |
| | org.apache.hadoop.log.TestLogLevel |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:eaf5c66 |
| JIRA Issue | YARN-7169 |
|
[jira] [Commented] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
[
https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208688#comment-16208688
]
Hadoop QA commented on YARN-7169:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
12s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
25s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m
51s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
53s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 9m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
7s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
4s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 50s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
51s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.http.TestHttpServer |
| | org.apache.hadoop.log.TestLogLevel |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:eaf5c66 |
| JIRA Issue | YARN-7169 |
|
[jira] [Updated] (YARN-7244) ShuffleHandler is not aware of disks that are added
[ https://issues.apache.org/jira/browse/YARN-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated YARN-7244: -- Attachment: YARN-7244.013.patch The build failed with 2 separate and unrelated issues I believe. The second time the cache seems to be picking up the old package for AuxiliaryLocalPathHandler. Re-triggering by uploading the same patch again. Please let me know if I missed something. > ShuffleHandler is not aware of disks that are added > --- > > Key: YARN-7244 > URL: https://issues.apache.org/jira/browse/YARN-7244 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: YARN-7244.001.patch, YARN-7244.002.patch, > YARN-7244.003.patch, YARN-7244.004.patch, YARN-7244.005.patch, > YARN-7244.006.patch, YARN-7244.007.patch, YARN-7244.008.patch, > YARN-7244.009.patch, YARN-7244.010.patch, YARN-7244.011.patch, > YARN-7244.012.patch, YARN-7244.013.patch > > > The ShuffleHandler permanently remembers the list of "good" disks on NM > startup. If disks later are added to the node then map tasks will start using > them but the ShuffleHandler will not be aware of them. The end result is that > the data cannot be shuffled from the node leading to fetch failures and > re-runs of the map tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7346) Fix compilation errors against hbase2 alpha release
[
https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208639#comment-16208639
]
Ted Yu commented on YARN-7346:
--
2.0.0-alpha4 hasn't come out yet.
Please build / install hbase-2 locally.
I normally use the following command line parameters :
{code}
-Phadoop-3.0 -Dhadoop-three.version=3.0.0-beta1
-Dhadoop-two.version=3.0.0-beta1 -Djetty.version=9.3.19.v20170502
{code}
> Fix compilation errors against hbase2 alpha release
> ---
>
> Key: YARN-7346
> URL: https://issues.apache.org/jira/browse/YARN-7346
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Vrushali C
>
> When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3,
> I got the following errors:
> https://pastebin.com/Ms4jYEVB
> This issue is to fix the compilation errors.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7244) ShuffleHandler is not aware of disks that are added
[
https://issues.apache.org/jira/browse/YARN-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208630#comment-16208630
]
Hadoop QA commented on YARN-7244:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
18s{color} | {color:green} root: The patch generated 0 new + 317 unchanged - 2
fixed = 317 total (was 319) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
42s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 8s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 57s{color}
| {color:red} hadoop-mapreduce-client-shuffle in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7244 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892684/YARN-7244.012.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 879d912a414c 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-7351) High CPU usage issue in RegistryDNS
[
https://issues.apache.org/jira/browse/YARN-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208624#comment-16208624
]
Hadoop QA commented on YARN-7351:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} yarn-native-services Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
13s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} yarn-native-services passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} yarn-native-services passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry: The patch generated 1 new
+ 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 46s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
58s{color} | {color:green} hadoop-yarn-registry in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 57s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7351 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892707/YARN-7351.yarn-native-services.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 646b8995a672 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | yarn-native-services / 16ecb9c |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/18001/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-registry.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18001/testReport/ |
| modules | C:
[jira] [Commented] (YARN-6546) SLS is slow while loading 10k queues
[
https://issues.apache.org/jira/browse/YARN-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208619#comment-16208619
]
Yufei Gu commented on YARN-6546:
Thanks [~haibo.chen] for the review and commit. Thanks for the review,
[~miklos.szeg...@cloudera.com] and [~grant.sohn].
> SLS is slow while loading 10k queues
>
>
> Key: YARN-6546
> URL: https://issues.apache.org/jira/browse/YARN-6546
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler-load-simulator
>Affects Versions: 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 3.0.0
>
> Attachments: Desktop.png, YARN-6546.001.patch, YARN-6546.002.patch
>
>
> It takes a long time (more than 10 minutes) to load 10k queues in SLS. The
> problem should be in {{com.codahale.metrics.CsvReporter}} based on the result
> from profiler. SLS creates 14 .csv files for each leaf queue, and update them
> constantly during execution. It is not necessary to log information for
> inactive queues.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7346) Fix compilation errors against hbase2 alpha release
[ https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208611#comment-16208611 ] Haibo Chen commented on YARN-7346: -- [~tedyu], which maven repository has 2.0.0-alpha4-SNAPSHOT been published to? Hadoop fetches from https://repository.apache.org/content/repositories/snapshots, but maven seems unable to find hbase-common-2.0.0-alpha4-SNAPSHOT from there > Fix compilation errors against hbase2 alpha release > --- > > Key: YARN-7346 > URL: https://issues.apache.org/jira/browse/YARN-7346 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ted Yu >Assignee: Vrushali C > > When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, > I got the following errors: > https://pastebin.com/Ms4jYEVB > This issue is to fix the compilation errors. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4163) Audit getQueueInfo and getApplications calls
[
https://issues.apache.org/jira/browse/YARN-4163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208609#comment-16208609
]
Hadoop QA commented on YARN-4163:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 4 new + 62 unchanged - 3 fixed = 66 total (was 65) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 4s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}106m 17s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMAdminService |
| | hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
|
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore |
| |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-4163 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892688/YARN-4163.007.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 892163df1026 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7532339 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (YARN-6546) SLS is slow while loading 10k queues
[
https://issues.apache.org/jira/browse/YARN-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208593#comment-16208593
]
Hudson commented on YARN-6546:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13100 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13100/])
YARN-6546. SLS is slow while loading 10k queues. (Yufei Gu via Haibo
(haibochen: rev 46eb1033a86ca53e7b94202567aef8af2417bdf8)
* (edit)
hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/FifoSchedulerMetrics.java
* (edit)
hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java
* (edit)
hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
* (edit)
hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/FairSchedulerMetrics.java
* (edit)
hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerMetrics.java
* (edit)
hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/CapacitySchedulerMetrics.java
* (edit)
hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSFairScheduler.java
> SLS is slow while loading 10k queues
>
>
> Key: YARN-6546
> URL: https://issues.apache.org/jira/browse/YARN-6546
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler-load-simulator
>Affects Versions: 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 3.0.0
>
> Attachments: Desktop.png, YARN-6546.001.patch, YARN-6546.002.patch
>
>
> It takes a long time (more than 10 minutes) to load 10k queues in SLS. The
> problem should be in {{com.codahale.metrics.CsvReporter}} based on the result
> from profiler. SLS creates 14 .csv files for each leaf queue, and update them
> constantly during execution. It is not necessary to log information for
> inactive queues.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6620) Add support in NodeManager to isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208590#comment-16208590 ] Wangda Tan commented on YARN-6620: -- [~tangzhankun], I may not make it clear: what I meant is GPU should be a first-class resource instead of mandatory resource. To me the only mandatory resource for now is memory and vcores, in the future we might add network/disk as mandatory resource. The definition of mandatory resource: in order to run process, mandatory resource is must required. The definition of first class resource: Officially supported by YARN. For your questions. bq. 1. First-class resource should be parsed from resource-types.xml and node-resources.xml(or auto discover) instead of yarn-site.xml? To me, for all resources beyond memory/vcores (because of historical reason), they should be defined in resource-types.xml and node-resources.xml regardless if it is a mandatory or first-class. bq. 2. First-calss resource handler should register itself with the same resource name defined in xml files? To me this is true when resource isolation on NM side is required, all first-class resource should started with "yarn.io/" namespace. bq. 3. First-class resource should be shown in a separate user-defined column in web pages? I'm not sure about this, in the future we may add more and more first-class / mandatory resources, it might be too much if we add columns for every new resources we added. To me the ideal solution is user can select and filter columns in web UI (support this in new UI should be a trivial task). > Add support in NodeManager to isolate GPU devices by using CGroups > -- > > Key: YARN-6620 > URL: https://issues.apache.org/jira/browse/YARN-6620 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 3.1.0 > > Attachments: YARN-6620.001.patch, YARN-6620.002.patch, > YARN-6620.003.patch, YARN-6620.004.patch, YARN-6620.005.patch, > YARN-6620.006-WIP.patch, YARN-6620.007.patch, YARN-6620.008.patch, > YARN-6620.009.patch, YARN-6620.010.patch, YARN-6620.011.patch, > YARN-6620.012.patch, YARN-6620.013.patch, YARN-6620.014.patch, > YARN-6620.015.patch, YARN-6620.016.patch, YARN-6620.017.patch > > > This JIRA plan to add support of: > 1) GPU configuration for NodeManagers > 2) Isolation in CGroups. (Java side). > 3) NM restart and recovery allocated GPU devices -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7326) Some issues in RegistryDNS
[
https://issues.apache.org/jira/browse/YARN-7326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208588#comment-16208588
]
Jian He commented on YARN-7326:
---
I have found the issue why DNS eats up one core, opened YARN-7351 to fix it.
It's mainly because the implementation creates a new thread for every dns
queries.. After the patch, the cpu usage drops from 100% to 0.1% on my Mac
> Some issues in RegistryDNS
> --
>
> Key: YARN-7326
> URL: https://issues.apache.org/jira/browse/YARN-7326
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
>
> [~aw] helped to identify these issues:
> Now some general bad news, not related to this patch:
> Ran a few queries, but this one is a bit concerning:
> {code}
> root@ubuntu:/hadoop/logs# dig @localhost -p 54 .
> ;; Warning: query response not set
> ; <<>> DiG 9.10.3-P4-Ubuntu <<>> @localhost -p 54 .
> ; (2 servers found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOTAUTH, id: 47794
> ;; flags: rd ad; QUERY: 0, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
> ;; WARNING: recursion requested but not available
> ;; Query time: 0 msec
> ;; SERVER: 127.0.0.1#54(127.0.0.1)
> ;; WHEN: Thu Oct 12 16:04:54 PDT 2017
> ;; MSG SIZE rcvd: 12
> root@ubuntu:/hadoop/logs# dig @localhost -p 54 axfr .
> ;; Connection to ::1#54(::1) for . failed: connection refused.
> ;; communications error to 127.0.0.1#54: end of file
> root@ubuntu:/hadoop/logs#
> {code}
> It looks like it effectively fails when asked about a root zone, which is bad.
> It's also kind of interesting in what it does and doesn't log. Probably
> should be configured to rotate logs based on size not date.
> The real showstopper though: RegistryDNS basically eats a core. It is running
> with 100% cpu utilization with and without jsvc. On my laptop, this is
> triggering my fan.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7351) High CPU usage issue in RegistryDNS
[
https://issues.apache.org/jira/browse/YARN-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208580#comment-16208580
]
Jian He commented on YARN-7351:
---
The reason is because the implementation, it creates a new thread for every dns
query.
{code}
while (true) {
final SocketChannel socketChannel = serverSocketChannel.accept();
if (socketChannel != null) {
executor.submit(new Callable() {
@Override
public Boolean call() throws Exception {
nioTCPClient(socketChannel);
return true;
}
});
}
}
{code}
After the patch, the DNS process cpu usages drops from 100% to 0.1% on my Mac
> High CPU usage issue in RegistryDNS
> ---
>
> Key: YARN-7351
> URL: https://issues.apache.org/jira/browse/YARN-7351
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-7351.yarn-native-services.01.patch
>
>
> Thanks [~aw] for finding this issue.
> The current RegistryDNS implementation is always running on high CPU and
> pretty much eats one core.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7351) High CPU usage issue in RegistryDNS
[ https://issues.apache.org/jira/browse/YARN-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7351: -- Attachment: YARN-7351.yarn-native-services.01.patch > High CPU usage issue in RegistryDNS > --- > > Key: YARN-7351 > URL: https://issues.apache.org/jira/browse/YARN-7351 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7351.yarn-native-services.01.patch > > > Thanks [~aw] for finding this issue. > The current RegistryDNS implementation is always running on high CPU and > pretty much eats one core. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7345) Incorrect minor device numbers written to devices.deny file
[ https://issues.apache.org/jira/browse/YARN-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208576#comment-16208576 ] Wangda Tan commented on YARN-7345: -- Change looks good to me, thanks [~jhung]! Will commit later today. > Incorrect minor device numbers written to devices.deny file > --- > > Key: YARN-7345 > URL: https://issues.apache.org/jira/browse/YARN-7345 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-7345.001.patch > > > Currently the minor numbers written to devices.deny file is 0->(num devices > to block - 1). But blocked devices are not necessarily sequential starting > from 0. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7330) Add support to show GPU on UI/metrics
[ https://issues.apache.org/jira/browse/YARN-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208571#comment-16208571 ] Wangda Tan commented on YARN-7330: -- [~sunilg], my first UI patch after YARN-3368 merged to trunk ;), could you help with patch review? > Add support to show GPU on UI/metrics > - > > Key: YARN-7330 > URL: https://issues.apache.org/jira/browse/YARN-7330 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7330.0-wip.patch, screencapture-0-wip.png > > > We should be able to view GPU metrics from UI/REST API. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7330) Add support to show GPU on UI/metrics
[
https://issues.apache.org/jira/browse/YARN-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208573#comment-16208573
]
Hadoop QA commented on YARN-7330:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-7330 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7330 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892705/YARN-7330.0-wip.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18000/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add support to show GPU on UI/metrics
> -
>
> Key: YARN-7330
> URL: https://issues.apache.org/jira/browse/YARN-7330
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: YARN-7330.0-wip.patch, screencapture-0-wip.png
>
>
> We should be able to view GPU metrics from UI/REST API.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
[ https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208566#comment-16208566 ] Hadoop QA commented on YARN-7169: - (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-YARN-Build/17999/console in case of problems. > Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2) > -- > > Key: YARN-7169 > URL: https://issues.apache.org/jira/browse/YARN-7169 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C >Priority: Critical > Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png, > YARN-7169-YARN-3368_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0002.patch, > YARN-7169-YARN-5355_branch2.0003.patch, > YARN-7169-YARN-5355_branch2.0004.patch, > YARN-7169-YARN-5355_branch2.0004.patch, YARN-7169-branch-2.0001.patch, > ui_commits(1) > > > Jira to track the backport of the new yarn-ui onto branch2. Right now adding > into Timeline Service v2's branch2 which is YARN-5355_branch2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
[ https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208563#comment-16208563 ] Hadoop QA commented on YARN-7169: - (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-YARN-Build/17998/console in case of problems. > Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2) > -- > > Key: YARN-7169 > URL: https://issues.apache.org/jira/browse/YARN-7169 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C >Priority: Critical > Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png, > YARN-7169-YARN-3368_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0002.patch, > YARN-7169-YARN-5355_branch2.0003.patch, > YARN-7169-YARN-5355_branch2.0004.patch, > YARN-7169-YARN-5355_branch2.0004.patch, YARN-7169-branch-2.0001.patch, > ui_commits(1) > > > Jira to track the backport of the new yarn-ui onto branch2. Right now adding > into Timeline Service v2's branch2 which is YARN-5355_branch2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7330) Add support to show GPU on UI/metrics
[ https://issues.apache.org/jira/browse/YARN-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7330: - Attachment: YARN-7330.0-wip.patch screencapture-0-wip.png Attached ver.0 patch & screenshot, which include: 1) Updated UI (v2 only, and updated GPU metrics on per-node page) and REST API to get GPU information. 2) Screenshot of this patch. > Add support to show GPU on UI/metrics > - > > Key: YARN-7330 > URL: https://issues.apache.org/jira/browse/YARN-7330 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-7330.0-wip.patch, screencapture-0-wip.png > > > We should be able to view GPU metrics from UI/REST API. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7330) Add support to show GPU on UI/metrics
[ https://issues.apache.org/jira/browse/YARN-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7330: - Description: We should be able to view GPU metrics from UI/REST API. > Add support to show GPU on UI/metrics > - > > Key: YARN-7330 > URL: https://issues.apache.org/jira/browse/YARN-7330 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > > We should be able to view GPU metrics from UI/REST API. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
[ https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-7169: - Attachment: YARN-7169-branch-2.0001.patch Okay so build on YARN-3368_branch2 works fine. Now attaching a patch for branch2. YARN-7169-branch-2.0001.patch > Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2) > -- > > Key: YARN-7169 > URL: https://issues.apache.org/jira/browse/YARN-7169 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C >Priority: Critical > Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png, > YARN-7169-YARN-3368_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0002.patch, > YARN-7169-YARN-5355_branch2.0003.patch, > YARN-7169-YARN-5355_branch2.0004.patch, > YARN-7169-YARN-5355_branch2.0004.patch, YARN-7169-branch-2.0001.patch, > ui_commits(1) > > > Jira to track the backport of the new yarn-ui onto branch2. Right now adding > into Timeline Service v2's branch2 which is YARN-5355_branch2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
[
https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208542#comment-16208542
]
Hadoop QA commented on YARN-7169:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m
49s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} YARN-3368_branch2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
12s{color} | {color:green} YARN-3368_branch2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:17213a0 |
| JIRA Issue | YARN-7169 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892693/YARN-7169-YARN-3368_branch2.0001.patch
|
| Optional Tests | asflicense shadedclient |
| uname | Linux ae68d8b8fc5a 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-3368_branch2 / 3a5f1a6 |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17997/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
> --
>
> Key: YARN-7169
> URL: https://issues.apache.org/jira/browse/YARN-7169
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>Priority: Critical
> Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png,
> YARN-7169-YARN-3368_branch2.0001.patch,
> YARN-7169-YARN-5355_branch2.0001.patch,
> YARN-7169-YARN-5355_branch2.0002.patch,
> YARN-7169-YARN-5355_branch2.0003.patch,
> YARN-7169-YARN-5355_branch2.0004.patch,
> YARN-7169-YARN-5355_branch2.0004.patch, ui_commits(1)
>
>
> Jira to track the backport of the new yarn-ui onto branch2. Right now adding
> into Timeline Service v2's branch2 which is YARN-5355_branch2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
[ https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208534#comment-16208534 ] Vrushali C commented on YARN-7169: -- I have triggered https://builds.apache.org/job/PreCommit-YARN-Build/17997/console > Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2) > -- > > Key: YARN-7169 > URL: https://issues.apache.org/jira/browse/YARN-7169 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C >Priority: Critical > Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png, > YARN-7169-YARN-3368_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0002.patch, > YARN-7169-YARN-5355_branch2.0003.patch, > YARN-7169-YARN-5355_branch2.0004.patch, > YARN-7169-YARN-5355_branch2.0004.patch, ui_commits(1) > > > Jira to track the backport of the new yarn-ui onto branch2. Right now adding > into Timeline Service v2's branch2 which is YARN-5355_branch2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7351) High CPU issue in RegistryDNS
Jian He created YARN-7351: - Summary: High CPU issue in RegistryDNS Key: YARN-7351 URL: https://issues.apache.org/jira/browse/YARN-7351 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Thanks [~aw] for finding this issue. The current RegistryDNS implementation is always running on high CPU and pretty much eats one core. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7351) High CPU usage issue in RegistryDNS
[ https://issues.apache.org/jira/browse/YARN-7351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7351: -- Summary: High CPU usage issue in RegistryDNS (was: High CPU issue in RegistryDNS) > High CPU usage issue in RegistryDNS > --- > > Key: YARN-7351 > URL: https://issues.apache.org/jira/browse/YARN-7351 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > > Thanks [~aw] for finding this issue. > The current RegistryDNS implementation is always running on high CPU and > pretty much eats one core. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
[
https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208522#comment-16208522
]
Hadoop QA commented on YARN-7169:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} YARN-3368_branch2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
10s{color} | {color:green} YARN-3368_branch2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:17213a0 |
| JIRA Issue | YARN-7169 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892693/YARN-7169-YARN-3368_branch2.0001.patch
|
| Optional Tests | asflicense shadedclient |
| uname | Linux fb8f935a39c0 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11
12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-3368_branch2 / 3a5f1a6 |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17995/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
> --
>
> Key: YARN-7169
> URL: https://issues.apache.org/jira/browse/YARN-7169
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>Priority: Critical
> Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png,
> YARN-7169-YARN-3368_branch2.0001.patch,
> YARN-7169-YARN-5355_branch2.0001.patch,
> YARN-7169-YARN-5355_branch2.0002.patch,
> YARN-7169-YARN-5355_branch2.0003.patch,
> YARN-7169-YARN-5355_branch2.0004.patch,
> YARN-7169-YARN-5355_branch2.0004.patch, ui_commits(1)
>
>
> Jira to track the backport of the new yarn-ui onto branch2. Right now adding
> into Timeline Service v2's branch2 which is YARN-5355_branch2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7344) Unit test for white list mount fails on CentOS 7
[
https://issues.apache.org/jira/browse/YARN-7344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208516#comment-16208516
]
Eric Badger commented on YARN-7344:
---
Looks like the problem here is that realpath for {{/bin}} returns {{/usr/bin/}}
on CentOS7/RHEL7. So when the test checks for the permitted mounts, it sees
{{/usr/bin/ls}} or {{/usr/bin}} instead of {{/bin/ls}} or {{/bin}}
{noformat}
[ebadger@foobar]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.1 (Maipo)
[ebadger@foobar]$ realpath /bin
/usr/bin
[ebadger@foobar]$ realpath /bin/ls
/usr/bin/ls
{noformat}
> Unit test for white list mount fails on CentOS 7
>
>
> Key: YARN-7344
> URL: https://issues.apache.org/jira/browse/YARN-7344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-beta1
> Environment: CentOS Linux release 7.4.1708 (Core)
> cmake3-3.6.3-1.el7.x86_64
> openjdk version "1.8.0_144"
> OpenJDK Runtime Environment (build 1.8.0_144-b01)
> OpenJDK 64-Bit Server VM (build 25.144-b01, mixed mode)
> Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426;
> 2017-04-03T15:39:06-04:00)
> libprotoc 2.5.0
>Reporter: Eric Yang
>
> YARN-6623 introduced ability to turn off docker support for container
> executor. When running C++ unit tests, the newly introduced tests failed.
> {code}
> [ FAILED ] 3 tests, listed below:
> [ FAILED ] TestDockerUtil.test_check_mount_permitted
> [ FAILED ] TestDockerUtil.test_normalize_mounts
> [ FAILED ] TestDockerUtil.test_add_rw_mounts
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7217) PUT method for update service for Service API doesn't function correctly
[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208504#comment-16208504
]
Hadoop QA commented on YARN-7217:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-7217 does not apply to yarn-native-services. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7217 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892694/YARN-7217.yarn-native-services.003.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17996/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> PUT method for update service for Service API doesn't function correctly
>
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch,
> YARN-7217.yarn-native-services.002.patch,
> YARN-7217.yarn-native-services.003.patch
>
>
> The PUT method for updateService API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for
> configuration vs status. This increase the number of action that can be
> performed. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/config
> Request Data:
> {
> "name":"[service_name]",
> "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "[service_name]",
> "state": "STOPPED|STARTED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7217) PUT method for update service for Service API doesn't function correctly
[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7217:
Attachment: YARN-7217.yarn-native-services.003.patch
The pre-commit test seems to use a previous build of hadoop-yarn-common to test
with hadoop-yarn-services-core and hadoop-yarn-services-api. The undefined
symbol are changes in hadoop-yarn-common. Findbugs reported error is also
existing issue in hadoop-yarn-api project, and not related to the patch.
> PUT method for update service for Service API doesn't function correctly
>
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch,
> YARN-7217.yarn-native-services.002.patch,
> YARN-7217.yarn-native-services.003.patch
>
>
> The PUT method for updateService API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for
> configuration vs status. This increase the number of action that can be
> performed. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/config
> Request Data:
> {
> "name":"[service_name]",
> "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "[service_name]",
> "state": "STOPPED|STARTED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6546) SLS is slow while loading 10k queues
[
https://issues.apache.org/jira/browse/YARN-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208499#comment-16208499
]
Haibo Chen commented on YARN-6546:
--
The patch LGTM. Will commit it shortly.
> SLS is slow while loading 10k queues
>
>
> Key: YARN-6546
> URL: https://issues.apache.org/jira/browse/YARN-6546
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler-load-simulator
>Affects Versions: 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: Desktop.png, YARN-6546.001.patch, YARN-6546.002.patch
>
>
> It takes a long time (more than 10 minutes) to load 10k queues in SLS. The
> problem should be in {{com.codahale.metrics.CsvReporter}} based on the result
> from profiler. SLS creates 14 .csv files for each leaf queue, and update them
> constantly during execution. It is not necessary to log information for
> inactive queues.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
[ https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-7169: - Attachment: (was: YARN-7169-YARN-3368_branch2.0001.patch) > Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2) > -- > > Key: YARN-7169 > URL: https://issues.apache.org/jira/browse/YARN-7169 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C >Priority: Critical > Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png, > YARN-7169-YARN-3368_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0002.patch, > YARN-7169-YARN-5355_branch2.0003.patch, > YARN-7169-YARN-5355_branch2.0004.patch, > YARN-7169-YARN-5355_branch2.0004.patch, ui_commits(1) > > > Jira to track the backport of the new yarn-ui onto branch2. Right now adding > into Timeline Service v2's branch2 which is YARN-5355_branch2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
[ https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-7169: - Attachment: YARN-7169-YARN-3368_branch2.0001.patch Uploading a patch with just some new line changes since the earlier patch did not apply, as the branch already had the commits. This patch is just to trigger the build. > Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2) > -- > > Key: YARN-7169 > URL: https://issues.apache.org/jira/browse/YARN-7169 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C >Priority: Critical > Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png, > YARN-7169-YARN-3368_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0002.patch, > YARN-7169-YARN-5355_branch2.0003.patch, > YARN-7169-YARN-5355_branch2.0004.patch, > YARN-7169-YARN-5355_branch2.0004.patch, ui_commits(1) > > > Jira to track the backport of the new yarn-ui onto branch2. Right now adding > into Timeline Service v2's branch2 which is YARN-5355_branch2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6927) Add support for individual resource types requests in MapReduce
[ https://issues.apache.org/jira/browse/YARN-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208491#comment-16208491 ] Daniel Templeton commented on YARN-6927: Yeah, forgot about that. Is there a way to tell what's being given on the command line versus what's being set from a config file or default? > Add support for individual resource types requests in MapReduce > --- > > Key: YARN-6927 > URL: https://issues.apache.org/jira/browse/YARN-6927 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Daniel Templeton >Assignee: Gergo Repas > Attachments: YARN-6927.000.patch, YARN-6927.001.patch > > > YARN-6504 adds support for resource profiles in MapReduce jobs, but resource > profiles don't give users much flexibility in their resource requests. To > satisfy users' needs, MapReduce should also allow users to specify arbitrary > resource requests. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7286) Add support for docker to have no capabilities
[
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208489#comment-16208489
]
Hadoop QA commented on YARN-7286:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m
15s{color} | {color:red} Docker failed to build yetus/hadoop:0de40f0. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7286 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892691/YARN-7286.007.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17991/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add support for docker to have no capabilities
> --
>
> Key: YARN-7286
> URL: https://issues.apache.org/jira/browse/YARN-7286
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7286.001.patch, YARN-7286.002.patch,
> YARN-7286.003.patch, YARN-7286.004.patch, YARN-7286.005.patch,
> YARN-7286.006.patch, YARN-7286.007.patch
>
>
> Support for controlling capabilities was introduced in YARN-4258. However, it
> does not allow for the capabilities list to be NULL, since {{getStrings()}}
> will treat an empty value the same as it treats an unset property. So, a NULL
> list will actually give the default capabilities list.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
[
https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208492#comment-16208492
]
Hadoop QA commented on YARN-7169:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} YARN-7169 does not apply to YARN-3368_branch2. Rebase required?
Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7169 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892686/YARN-7169-YARN-3368_branch2.0001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17994/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
> --
>
> Key: YARN-7169
> URL: https://issues.apache.org/jira/browse/YARN-7169
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>Priority: Critical
> Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png,
> YARN-7169-YARN-3368_branch2.0001.patch,
> YARN-7169-YARN-5355_branch2.0001.patch,
> YARN-7169-YARN-5355_branch2.0002.patch,
> YARN-7169-YARN-5355_branch2.0003.patch,
> YARN-7169-YARN-5355_branch2.0004.patch,
> YARN-7169-YARN-5355_branch2.0004.patch, ui_commits(1)
>
>
> Jira to track the backport of the new yarn-ui onto branch2. Right now adding
> into Timeline Service v2's branch2 which is YARN-5355_branch2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
[
https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208486#comment-16208486
]
Hadoop QA commented on YARN-7169:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 9s{color}
| {color:red} YARN-7169 does not apply to YARN-3368_branch2. Rebase required?
Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7169 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892686/YARN-7169-YARN-3368_branch2.0001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17990/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
> --
>
> Key: YARN-7169
> URL: https://issues.apache.org/jira/browse/YARN-7169
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineclient, timelinereader, timelineserver
>Reporter: Vrushali C
>Assignee: Vrushali C
>Priority: Critical
> Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png,
> YARN-7169-YARN-3368_branch2.0001.patch,
> YARN-7169-YARN-5355_branch2.0001.patch,
> YARN-7169-YARN-5355_branch2.0002.patch,
> YARN-7169-YARN-5355_branch2.0003.patch,
> YARN-7169-YARN-5355_branch2.0004.patch,
> YARN-7169-YARN-5355_branch2.0004.patch, ui_commits(1)
>
>
> Jira to track the backport of the new yarn-ui onto branch2. Right now adding
> into Timeline Service v2's branch2 which is YARN-5355_branch2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2497) Changes for fair scheduler to support allocate resource respect labels
[ https://issues.apache.org/jira/browse/YARN-2497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208484#comment-16208484 ] Daniel Templeton commented on YARN-2497: The primary motivation for queue labels is to allow for access control of labeled resources. If we don't allow labels at the queue level, we have to live without ACLs or build support for ACLs into node labels, which is a pretty big change. There are certainly new issues that will crop up because of node labels. For example, we can block an app from going into a queue that doesn't offer the labels it says it needs, but we can't do anything about the app making resource requests with labels that the queue doesn't support. In that case the requests will simply remain pending forever because the queue will not accept any node that doesn't have a compatible label. I looked at how to mitigate that issue specifically, and I don't see anything that's not a significant overhaul of the way things work today. Capacity scheduler has the same issue, and thus far it doesn't sound like it has been a big problem. > Changes for fair scheduler to support allocate resource respect labels > -- > > Key: YARN-2497 > URL: https://issues.apache.org/jira/browse/YARN-2497 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Wangda Tan >Assignee: Daniel Templeton > Attachments: YARN-2497.001.patch, YARN-2497.002.patch, > YARN-2497.003.patch, YARN-2497.004.patch, YARN-2497.005.patch, > YARN-2497.006.patch, YARN-2497.007.patch, YARN-2497.008.patch, > YARN-2497.009.patch, YARN-2497.010.patch, YARN-2499.WIP01.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7197) Add support for a volume blacklist for docker containers
[ https://issues.apache.org/jira/browse/YARN-7197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208479#comment-16208479 ] Eric Badger commented on YARN-7197: --- [~eyang], just like YARN-5534, I'll hold off looking at this patch until YARN-7344 is resolved > Add support for a volume blacklist for docker containers > > > Key: YARN-7197 > URL: https://issues.apache.org/jira/browse/YARN-7197 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Eric Yang > Attachments: YARN-7197.001.patch > > > Docker supports bind mounting host directories into containers. Work is > underway to allow admins to configure a whilelist of volume mounts. While > this is a much needed and useful feature, it opens the door for > misconfiguration that may lead to users being able to compromise or crash the > system. > One example would be allowing users to mount /run from a host running > systemd, and then running systemd in that container, rendering the host > mostly unusable. > This issue is to add support for a default blacklist. The default blacklist > would be where we put files and directories that if mounted into a container, > are likely to have negative consequences. Users are encouraged not to remove > items from the default blacklist, but may do so if necessary. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5534) Allow whitelisted volume mounts
[ https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208475#comment-16208475 ] Eric Badger commented on YARN-5534: --- [~eyang], ah yes good point. I'll try and take a look at those unit tests. But yea, we can keep this open until that's resolved > Allow whitelisted volume mounts > > > Key: YARN-5534 > URL: https://issues.apache.org/jira/browse/YARN-5534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: luhuichun >Assignee: Shane Kumpf > Attachments: YARN-5534.001.patch, YARN-5534.002.patch, > YARN-5534.003.patch > > > Introduction > Mounting files or directories from the host is one way of passing > configuration and other information into a docker container. > We could allow the user to set a list of mounts in the environment of > ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2). > These would be mounted read-only to the specified target locations. This has > been resolved in YARN-4595 > 2.Problem Definition > Bug mounting arbitrary volumes into a Docker container can be a security risk. > 3.Possible solutions > one approach to provide safe mounts is to allow the cluster administrator to > configure a set of parent directories as white list mounting directories. > Add a property named yarn.nodemanager.volume-mounts.white-list, when > container executor do mount checking, only the allowed directories or > sub-directories can be mounted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7286) Add support for docker to have no capabilities
[
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated YARN-7286:
--
Attachment: YARN-7286.007.patch
Thanks for the review, [~jlowe]! Uploading a new patch that fixes the copy and
paste error and gets rid of the unnecessary try/catch blocks
> Add support for docker to have no capabilities
> --
>
> Key: YARN-7286
> URL: https://issues.apache.org/jira/browse/YARN-7286
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7286.001.patch, YARN-7286.002.patch,
> YARN-7286.003.patch, YARN-7286.004.patch, YARN-7286.005.patch,
> YARN-7286.006.patch, YARN-7286.007.patch
>
>
> Support for controlling capabilities was introduced in YARN-4258. However, it
> does not allow for the capabilities list to be NULL, since {{getStrings()}}
> will treat an empty value the same as it treats an unset property. So, a NULL
> list will actually give the default capabilities list.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7244) ShuffleHandler is not aware of disks that are added
[
https://issues.apache.org/jira/browse/YARN-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208456#comment-16208456
]
Hadoop QA commented on YARN-7244:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 5m
33s{color} | {color:red} Docker failed to build yetus/hadoop:0de40f0. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7244 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892684/YARN-7244.012.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17988/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ShuffleHandler is not aware of disks that are added
> ---
>
> Key: YARN-7244
> URL: https://issues.apache.org/jira/browse/YARN-7244
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: YARN-7244.001.patch, YARN-7244.002.patch,
> YARN-7244.003.patch, YARN-7244.004.patch, YARN-7244.005.patch,
> YARN-7244.006.patch, YARN-7244.007.patch, YARN-7244.008.patch,
> YARN-7244.009.patch, YARN-7244.010.patch, YARN-7244.011.patch,
> YARN-7244.012.patch
>
>
> The ShuffleHandler permanently remembers the list of "good" disks on NM
> startup. If disks later are added to the node then map tasks will start using
> them but the ShuffleHandler will not be aware of them. The end result is that
> the data cannot be shuffled from the node leading to fetch failures and
> re-runs of the map tasks.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7350) TestSignalContainer should check both FS and CS
Miklos Szegedi created YARN-7350: Summary: TestSignalContainer should check both FS and CS Key: YARN-7350 URL: https://issues.apache.org/jira/browse/YARN-7350 Project: Hadoop YARN Issue Type: Bug Reporter: Miklos Szegedi -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7294) TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with Fair scheduler
[ https://issues.apache.org/jira/browse/YARN-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208438#comment-16208438 ] Miklos Szegedi edited comment on YARN-7294 at 10/17/17 9:41 PM: Thank you, [~yufeigu] for the review. Yes, that should be a different issue, since my code change was added after that part of the test: TestSignalContainer.java:86. I opened YARN-7349 for that issue. I opened YARN-7350 to make the test work with both FS and CS. was (Author: miklos.szeg...@cloudera.com): Thank you, [~yufeigu] for the review. Yes, that should be a different issue, since my code change was added after that part of the test: TestSignalContainer.java:86. I opened YARN-7349 for that issue. > TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with > Fair scheduler > -- > > Key: YARN-7294 > URL: https://issues.apache.org/jira/browse/YARN-7294 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-7294.000.patch > > > This issue exists due to the fact that FS needs an update after allocation > and more node updates for all the requests to be fulfilled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7349) TestSignalContainer.testSignalRequestDeliveryToNM fails with attempt state is not correct
Miklos Szegedi created YARN-7349: Summary: TestSignalContainer.testSignalRequestDeliveryToNM fails with attempt state is not correct Key: YARN-7349 URL: https://issues.apache.org/jira/browse/YARN-7349 Project: Hadoop YARN Issue Type: Bug Reporter: Miklos Szegedi java.lang.AssertionError: Attempt state is not correct (timeout). Expected :ALLOCATED Actual :SCHEDULED at org.junit.Assert.fail(Assert.java:88) at org.junit.Assert.failNotEquals(Assert.java:743) at org.junit.Assert.assertEquals(Assert.java:118) at org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:358) at org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:317) at org.apache.hadoop.yarn.server.resourcemanager.MockRM.waitForState(MockRM.java:298) at org.apache.hadoop.yarn.server.resourcemanager.MockRM.sendAMLaunched(MockRM.java:955) at org.apache.hadoop.yarn.server.resourcemanager.TestSignalContainer.testSignalRequestDeliveryToNM(TestSignalContainer.java:68) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7294) TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with Fair scheduler
[ https://issues.apache.org/jira/browse/YARN-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208438#comment-16208438 ] Miklos Szegedi edited comment on YARN-7294 at 10/17/17 9:39 PM: Thank you, [~yufeigu] for the review. Yes, that should be a different issue, since my code change was added after that part of the test: TestSignalContainer.java:86. I opened YARN-7349 for that issue. was (Author: miklos.szeg...@cloudera.com): Thank you, [~yufeigu] for the review. Yes, that should be a different issue, since my code change was added after that part of the test: TestSignalContainer.java:86 > TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with > Fair scheduler > -- > > Key: YARN-7294 > URL: https://issues.apache.org/jira/browse/YARN-7294 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-7294.000.patch > > > This issue exists due to the fact that FS needs an update after allocation > and more node updates for all the requests to be fulfilled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7294) TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with Fair scheduler
[ https://issues.apache.org/jira/browse/YARN-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208438#comment-16208438 ] Miklos Szegedi commented on YARN-7294: -- Thank you, [~yufeigu] for the review. Yes, that should be a different issue, since my code change was added after that part of the test: TestSignalContainer.java:86 > TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with > Fair scheduler > -- > > Key: YARN-7294 > URL: https://issues.apache.org/jira/browse/YARN-7294 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi > Attachments: YARN-7294.000.patch > > > This issue exists due to the fact that FS needs an update after allocation > and more node updates for all the requests to be fulfilled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4163) Audit getQueueInfo and getApplications calls
[
https://issues.apache.org/jira/browse/YARN-4163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-4163:
-
Attachment: YARN-4163.007.patch
Thanks [~jlowe] for the review.
I have made some "generic" APIs for successful and failure logs that take the
common set of arguments plus an {{ArgsBuilder}} that contains the
operation-specific arguments.
These generic APIs could be used to replace the existing success and failure
log methods. I suggest that a separate JIRA be created for that.
I have a separate branch-2.8 patch that I will upload once the pre-commit build
completes for this patch.
> Audit getQueueInfo and getApplications calls
>
>
> Key: YARN-4163
> URL: https://issues.apache.org/jira/browse/YARN-4163
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Chang Li
>Assignee: Chang Li
> Attachments: YARN-4163.004.patch, YARN-4163.005.patch,
> YARN-4163.006.branch-2.8.patch, YARN-4163.006.patch, YARN-4163.007.patch,
> YARN-4163.2.patch, YARN-4163.2.patch, YARN-4163.3.patch, YARN-4163.patch
>
>
> getQueueInfo and getApplications seem to sometimes cause spike of load but
> not able to confirm due to they are not audit logged. This patch propose to
> add them to audit log
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-6546) SLS is slow while loading 10k queues
[
https://issues.apache.org/jira/browse/YARN-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated YARN-6546:
-
Comment: was deleted
(was: +1. Will commit it later if there is no objection.)
> SLS is slow while loading 10k queues
>
>
> Key: YARN-6546
> URL: https://issues.apache.org/jira/browse/YARN-6546
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler-load-simulator
>Affects Versions: 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: Desktop.png, YARN-6546.001.patch, YARN-6546.002.patch
>
>
> It takes a long time (more than 10 minutes) to load 10k queues in SLS. The
> problem should be in {{com.codahale.metrics.CsvReporter}} based on the result
> from profiler. SLS creates 14 .csv files for each leaf queue, and update them
> constantly during execution. It is not necessary to log information for
> inactive queues.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7169) Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2)
[ https://issues.apache.org/jira/browse/YARN-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated YARN-7169: - Attachment: YARN-7169-YARN-3368_branch2.0001.patch I have committed the backport of the UI code to branch: YARN-3368_branch2 https://github.com/apache/hadoop/tree/YARN-3368_branch2 Uploading YARN-7169-YARN-3368_branch2.0001.patch which was generated after the ATSv2 merge to branch2. So this patch is based on the diff at commit 5e7491d5320d0c9bbceacebb7d884978677f804b using the command: git diff --binary --no-prefix ^5e7491d5320d0c9bbceacebb7d884978677f804b > Backport new yarn-ui to branch2 code (starting with YARN-5355_branch2) > -- > > Key: YARN-7169 > URL: https://issues.apache.org/jira/browse/YARN-7169 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineclient, timelinereader, timelineserver >Reporter: Vrushali C >Assignee: Vrushali C >Priority: Critical > Attachments: FlowRunDetails_Sleepjob.png, Metrics_Yarn_UI.png, > YARN-7169-YARN-3368_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0001.patch, > YARN-7169-YARN-5355_branch2.0002.patch, > YARN-7169-YARN-5355_branch2.0003.patch, > YARN-7169-YARN-5355_branch2.0004.patch, > YARN-7169-YARN-5355_branch2.0004.patch, ui_commits(1) > > > Jira to track the backport of the new yarn-ui onto branch2. Right now adding > into Timeline Service v2's branch2 which is YARN-5355_branch2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7244) ShuffleHandler is not aware of disks that are added
[ https://issues.apache.org/jira/browse/YARN-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated YARN-7244: -- Attachment: YARN-7244.012.patch Fixing minor checkstyle issues. :( > ShuffleHandler is not aware of disks that are added > --- > > Key: YARN-7244 > URL: https://issues.apache.org/jira/browse/YARN-7244 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: YARN-7244.001.patch, YARN-7244.002.patch, > YARN-7244.003.patch, YARN-7244.004.patch, YARN-7244.005.patch, > YARN-7244.006.patch, YARN-7244.007.patch, YARN-7244.008.patch, > YARN-7244.009.patch, YARN-7244.010.patch, YARN-7244.011.patch, > YARN-7244.012.patch > > > The ShuffleHandler permanently remembers the list of "good" disks on NM > startup. If disks later are added to the node then map tasks will start using > them but the ShuffleHandler will not be aware of them. The end result is that > the data cannot be shuffled from the node leading to fetch failures and > re-runs of the map tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6546) SLS is slow while loading 10k queues
[
https://issues.apache.org/jira/browse/YARN-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208344#comment-16208344
]
Haibo Chen commented on YARN-6546:
--
+1. Will commit it later if there is no objection.
> SLS is slow while loading 10k queues
>
>
> Key: YARN-6546
> URL: https://issues.apache.org/jira/browse/YARN-6546
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler-load-simulator
>Affects Versions: 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: Desktop.png, YARN-6546.001.patch, YARN-6546.002.patch
>
>
> It takes a long time (more than 10 minutes) to load 10k queues in SLS. The
> problem should be in {{com.codahale.metrics.CsvReporter}} based on the result
> from profiler. SLS creates 14 .csv files for each leaf queue, and update them
> constantly during execution. It is not necessary to log information for
> inactive queues.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7244) ShuffleHandler is not aware of disks that are added
[
https://issues.apache.org/jira/browse/YARN-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208336#comment-16208336
]
Hadoop QA commented on YARN-7244:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 57s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 12s{color} | {color:orange} root: The patch generated 1 new + 317 unchanged
- 2 fixed = 318 total (was 319) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
41s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 7s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} hadoop-mapreduce-client-shuffle in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 24s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7244 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892665/YARN-7244.011.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 8cb6586f23bf 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
|
[jira] [Commented] (YARN-4827) Document configuration of ReservationSystem for FairScheduler
[ https://issues.apache.org/jira/browse/YARN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208319#comment-16208319 ] Yufei Gu commented on YARN-4827: Will do soon. > Document configuration of ReservationSystem for FairScheduler > - > > Key: YARN-4827 > URL: https://issues.apache.org/jira/browse/YARN-4827 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Subru Krishnan >Assignee: Yufei Gu >Priority: Blocker > > This JIRA tracks the effort to add documentation on how to configure > ReservationSystem for FairScheduler -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7346) Fix compilation errors against hbase2 alpha release
[ https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208287#comment-16208287 ] Haibo Chen commented on YARN-7346: -- YARN-7213 is the effort to move to hbase 2.0 (alpha3 though because we want to stick with a release). There were some hbase issues blocked the progress there. I'll retry now that some of them have been fixed. > Fix compilation errors against hbase2 alpha release > --- > > Key: YARN-7346 > URL: https://issues.apache.org/jira/browse/YARN-7346 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Ted Yu >Assignee: Vrushali C > > When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, > I got the following errors: > https://pastebin.com/Ms4jYEVB > This issue is to fix the compilation errors. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7217) PUT method for update service for Service API doesn't function correctly
[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208278#comment-16208278
]
Eric Yang commented on YARN-7217:
-
[~billie.rinaldi] A noble idea doesn't always translate to reality. Is it
possible to add a standby namenode without restart datanode? Is it possible to
add another resource manager without restart node manager? Is it possible to
add another Hive server2 without restart Knox for load balance? All above
questions have the same answer, no. Hadoop have spent significant 12 years of
resources to ensure rpc call retries and elastic datanode can happen. In the
recent years, Ambari has been invented to ensure Hadoop configuration are
recorded before service operation are performed for serviceability. However,
most applications will not receive the same amount of investment like Hadoop in
developing reliability and serviceability. Therefore, I do not know if flex
without restart would be deem as important as it was once promised.
Please note that, the current PUT method retains flex operation as it was
written. It only provide additional end point to record number of container
needs to be increased or decreased in the event of service restart. Service
can resume with the same number of containers prior to stop. Perhaps, we can
add a flag to Service object in component section. A flag to indicate if the
component can increase/decrease nodes without restart. This will be an hint to
backend to allow increase node count without restart. This helps to keep
existing slider flex functionality for HBase Region server and new emerging
application. Does this sound like a reasonable enhancement?
> PUT method for update service for Service API doesn't function correctly
>
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch,
> YARN-7217.yarn-native-services.002.patch
>
>
> The PUT method for updateService API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for
> configuration vs status. This increase the number of action that can be
> performed. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/config
> Request Data:
> {
> "name":"[service_name]",
> "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "[service_name]",
> "state": "STOPPED|STARTED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7102) NM heartbeat stuck when responseId overflows MAX_INT
[ https://issues.apache.org/jira/browse/YARN-7102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208274#comment-16208274 ] Jason Lowe commented on YARN-7102: -- Thanks for updating the patch! +1 for the trunk patch. It does not apply to branch-2 or branch-2.8. Could you provide patches against those branches? > NM heartbeat stuck when responseId overflows MAX_INT > > > Key: YARN-7102 > URL: https://issues.apache.org/jira/browse/YARN-7102 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Critical > Attachments: YARN-7102.v1.patch, YARN-7102.v2.patch, > YARN-7102.v3.patch, YARN-7102.v4.patch, YARN-7102.v5.patch, > YARN-7102.v6.patch, YARN-7102.v7.patch, YARN-7102.v8.patch, YARN-7102.v9.patch > > > ResponseId overflow problem in NM-RM heartbeat. This is same as AM-RM > heartbeat in YARN-6640, please refer to YARN-6640 for details. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7217) PUT method for update service for Service API doesn't function correctly
[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208273#comment-16208273
]
Gour Saha commented on YARN-7217:
-
{quote}
Do we need a flex operation without restart service? The answer is likely no.
There are 3 services (Datanode, NodeManager, HBase region server) that can add
slave nodes without restarting masters. The majority of use case for changing
node count will result in configuration changes and force restart. Majority of
software follow the second model to make config changes, then restart services.
This ensure the change request is repeatable. Hence, the need to support flex
operation without configuration change would be greatly reduced.
{quote}
Scale up and scale down of a specific version of a running service is the most
important feature of an enterprise-grade service (pay only for what you use,
financial web-app services scaled up during the day and scaled down during the
night to make way for batch processes, etc.). Configuration changes or newer
version of the service falls into the bucket of an upgrade. In fact, Slider
supports the notion of rolling-upgrade of a service or specific components of a
service in a service-owner defined orchestrated fashion, such that to the
end-user the service never stops running. There was never a need to stop the
service as a whole, unless you ran out of IT budget.
> PUT method for update service for Service API doesn't function correctly
>
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch,
> YARN-7217.yarn-native-services.002.patch
>
>
> The PUT method for updateService API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for
> configuration vs status. This increase the number of action that can be
> performed. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/config
> Request Data:
> {
> "name":"[service_name]",
> "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "[service_name]",
> "state": "STOPPED|STARTED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7341) TestRouterWebServiceUtil#testMergeMetrics is flakey
[
https://issues.apache.org/jira/browse/YARN-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208270#comment-16208270
]
Robert Kanter commented on YARN-7341:
-
Thanks [~subru].
> TestRouterWebServiceUtil#testMergeMetrics is flakey
> ---
>
> Key: YARN-7341
> URL: https://issues.apache.org/jira/browse/YARN-7341
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.0.0-beta1
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: YARN-7341.001.patch
>
>
> {{TestRouterWebServiceUtil#testMergeMetrics}} is flakey. It sometimes fails
> with something like:
> {noformat}
> Running org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil
> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.252 sec <<<
> FAILURE! - in
> org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil
> testMergeMetrics(org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil)
> Time elapsed: 0.005 sec <<< FAILURE!
> java.lang.AssertionError: expected:<1092> but was:<584>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil.testMergeMetrics(TestRouterWebServiceUtil.java:473)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4827) Document configuration of ReservationSystem for FairScheduler
[ https://issues.apache.org/jira/browse/YARN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208262#comment-16208262 ] Subru Krishnan commented on YARN-4827: -- [~yufeigu] - now that you have resolved YARN-4859/YARN-7311, can you post a patch for this and I'll take a look at it. Thanks. > Document configuration of ReservationSystem for FairScheduler > - > > Key: YARN-4827 > URL: https://issues.apache.org/jira/browse/YARN-4827 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Subru Krishnan >Assignee: Yufei Gu >Priority: Blocker > > This JIRA tracks the effort to add documentation on how to configure > ReservationSystem for FairScheduler -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7342) Application page doesn't show correct metrics for reservation runs
[ https://issues.apache.org/jira/browse/YARN-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-7342: --- Target Version/s: 2.9.0, 3.1.0 > Application page doesn't show correct metrics for reservation runs > --- > > Key: YARN-7342 > URL: https://issues.apache.org/jira/browse/YARN-7342 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler, reservation system >Affects Versions: 3.1.0 >Reporter: Yufei Gu > Attachments: Screen Shot 2017-10-16 at 17.27.48.png > > > As the screen shot shows, there are some bugs on the webUI while running job > with reservation. For examples, queue name should just be "root.queueA" > instead of internal queue name. All metrics(Allocated CPU, % of queue, etc) > are missing for reservation runs. These should be a blocker though. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4859) [Bug] Unable to submit a job to a reservation when using FairScheduler
[ https://issues.apache.org/jira/browse/YARN-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu resolved YARN-4859. Resolution: Works for Me > [Bug] Unable to submit a job to a reservation when using FairScheduler > -- > > Key: YARN-4859 > URL: https://issues.apache.org/jira/browse/YARN-4859 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Subru Krishnan >Assignee: Yufei Gu >Priority: Blocker > Attachments: Screen Shot 2017-10-16 at 17.27.48.png > > > Jobs submitted to a reservation get stuck at scheduled stage when using > FairScheduler. I came across this when working on YARN-4827 (documentation > for configuring ReservationSystem for FairScheduler) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7230) Document DockerContainerRuntime for branch-2.8 with proper scope and claim as an experimental feature
[ https://issues.apache.org/jira/browse/YARN-7230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208254#comment-16208254 ] Junping Du commented on YARN-7230: -- Thanks [~dan...@cloudera.com] and [~ebadger] for review. The patch LGTM as well. Will go ahead to commit to 2.8.x branches if no further comments. > Document DockerContainerRuntime for branch-2.8 with proper scope and claim as > an experimental feature > - > > Key: YARN-7230 > URL: https://issues.apache.org/jira/browse/YARN-7230 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 2.8.1 >Reporter: Junping Du >Assignee: Shane Kumpf >Priority: Blocker > Labels: ready-to-commit > Attachments: YARN-7230.branch-2.8.001.patch, > YARN-7230.branch-2.8.002.patch, YARN-7230.branch-2.8.003.patch > > > YARN-5258 is to document new feature for docker container runtime which > already get checked in trunk/branch-2. We need a similar one for branch-2.8. > However, given we missed several patches, we need to define narrowed scope of > these feature/improvements which match with existing patches landed in 2.8. > Also, like YARN-6622, to document it as experimental. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4859) [Bug] Unable to submit a job to a reservation when using FairScheduler
[ https://issues.apache.org/jira/browse/YARN-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208253#comment-16208253 ] Yufei Gu commented on YARN-4859: Thanks [~subru]. Filed YARN-7347 for the "root.root" issue. Filed YARN-7348 for the issue in fair policy queues. > [Bug] Unable to submit a job to a reservation when using FairScheduler > -- > > Key: YARN-4859 > URL: https://issues.apache.org/jira/browse/YARN-4859 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Subru Krishnan >Assignee: Yufei Gu >Priority: Blocker > Attachments: Screen Shot 2017-10-16 at 17.27.48.png > > > Jobs submitted to a reservation get stuck at scheduled stage when using > FairScheduler. I came across this when working on YARN-4827 (documentation > for configuring ReservationSystem for FairScheduler) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7348) Ignore the vcore in reservation request for fair policy queue
Yufei Gu created YARN-7348: -- Summary: Ignore the vcore in reservation request for fair policy queue Key: YARN-7348 URL: https://issues.apache.org/jira/browse/YARN-7348 Project: Hadoop YARN Issue Type: Improvement Components: fairscheduler, reservation system Reporter: Yufei Gu The reservation calculation is based on steady fair share, whose vcores always be zero for fair policy queues. We can set vcores to zero in reservation request. Beside, we could do some improvement, like ignoring the vcores if the request is for fair policy queues. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7347) Fixe the bug in Fair scheduler to handle a queue named "root.root"
Yufei Gu created YARN-7347:
--
Summary: Fixe the bug in Fair scheduler to handle a queue named
"root.root"
Key: YARN-7347
URL: https://issues.apache.org/jira/browse/YARN-7347
Project: Hadoop YARN
Issue Type: Bug
Components: fairscheduler, reservation system
Reporter: Yufei Gu
Assignee: Yufei Gu
A queue named "root.root" may cause issue in Fair scheduler. For example, if we
set the queue(root.root) to be reservable, then submit a job into the queue. We
got following error.
{code}
java.io.IOException: org.apache.hadoop.yarn.exceptions.YarnException: Failed to
submit application_1508176133973_0002 to YARN : root.root is not a leaf queue
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:339)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:253)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1962)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1588)
at
org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:307)
at
org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:360)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:368)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit
application_1508176133973_0002 to YARN : root.root is not a leaf queue
at
org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:293)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:298)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:324)
... 25 more
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7345) Incorrect minor device numbers written to devices.deny file
[
https://issues.apache.org/jira/browse/YARN-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208237#comment-16208237
]
Hadoop QA commented on YARN-7345:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
28m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 17s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 49s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0de40f0 |
| JIRA Issue | YARN-7345 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12892663/YARN-7345.001.patch |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 7073a9a5bb1f 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / acabc65 |
| Default Java | 1.8.0_144 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/17987/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/17987/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17987/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Incorrect minor device numbers written to devices.deny file
> ---
>
> Key: YARN-7345
> URL: https://issues.apache.org/jira/browse/YARN-7345
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
> Attachments: YARN-7345.001.patch
>
>
> Currently the minor numbers written to devices.deny file is 0->(num devices
> to block - 1). But blocked devices are not necessarily sequential starting
> from 0.
--
This message was sent by Atlassian JIRA
[jira] [Commented] (YARN-7311) Fix TestRMWebServicesReservation parametrization for fair scheduler
[
https://issues.apache.org/jira/browse/YARN-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208236#comment-16208236
]
Yufei Gu commented on YARN-7311:
Thanks for the review, [~subru] and [~seanpo03].
> Fix TestRMWebServicesReservation parametrization for fair scheduler
> ---
>
> Key: YARN-7311
> URL: https://issues.apache.org/jira/browse/YARN-7311
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, reservation system
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: YARN-7311.001.patch, YARN-7311.002.patch,
> YARN-7311.WIP.patch
>
>
> YARN-4248 introduced the REST API for submit/update/delete Reservations.
> Class {{TestRMWebServicesReservation}} intends to test both FS and CS. The
> test cases designed for fair scheduler actually test capacity scheduler. The
> following code in method {{configureServlets}} shows it sets the scheduler to
> CS even test cases are for fair scheduler.
> {code}
> conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
> ResourceScheduler.class);
> CapacitySchedulerConfiguration csconf =
> new CapacitySchedulerConfiguration(conf);
> String[] queues = { "default", "dedicated" };
> csconf.setQueues("root", queues);
> csconf.setCapacity("root.default", 50.0f);
> csconf.setCapacity("root.dedicated", 50.0f);
> csconf.setReservable("root.dedicated", true);
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7311) Fix TestRMWebServicesReservation parametrization for fair scheduler
[
https://issues.apache.org/jira/browse/YARN-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-7311:
---
Priority: Blocker (was: Major)
> Fix TestRMWebServicesReservation parametrization for fair scheduler
> ---
>
> Key: YARN-7311
> URL: https://issues.apache.org/jira/browse/YARN-7311
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, reservation system
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Blocker
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: YARN-7311.001.patch, YARN-7311.002.patch,
> YARN-7311.WIP.patch
>
>
> YARN-4248 introduced the REST API for submit/update/delete Reservations.
> Class {{TestRMWebServicesReservation}} intends to test both FS and CS. The
> test cases designed for fair scheduler actually test capacity scheduler. The
> following code in method {{configureServlets}} shows it sets the scheduler to
> CS even test cases are for fair scheduler.
> {code}
> conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
> ResourceScheduler.class);
> CapacitySchedulerConfiguration csconf =
> new CapacitySchedulerConfiguration(conf);
> String[] queues = { "default", "dedicated" };
> csconf.setQueues("root", queues);
> csconf.setCapacity("root.default", 50.0f);
> csconf.setCapacity("root.dedicated", 50.0f);
> csconf.setReservable("root.dedicated", true);
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7311) Fix TestRMWebServicesReservation parametrization for fair scheduler
[
https://issues.apache.org/jira/browse/YARN-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208233#comment-16208233
]
Hudson commented on YARN-7311:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13099 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13099/])
YARN-7311. Fix TestRMWebServicesReservation parametrization for fair (subu: rev
75323394fbc4211596a2c8fbb5e584f3183f742f)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesReservation.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/FairReservationSystem.java
> Fix TestRMWebServicesReservation parametrization for fair scheduler
> ---
>
> Key: YARN-7311
> URL: https://issues.apache.org/jira/browse/YARN-7311
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, reservation system
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: YARN-7311.001.patch, YARN-7311.002.patch,
> YARN-7311.WIP.patch
>
>
> YARN-4248 introduced the REST API for submit/update/delete Reservations.
> Class {{TestRMWebServicesReservation}} intends to test both FS and CS. The
> test cases designed for fair scheduler actually test capacity scheduler. The
> following code in method {{configureServlets}} shows it sets the scheduler to
> CS even test cases are for fair scheduler.
> {code}
> conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
> ResourceScheduler.class);
> CapacitySchedulerConfiguration csconf =
> new CapacitySchedulerConfiguration(conf);
> String[] queues = { "default", "dedicated" };
> csconf.setQueues("root", queues);
> csconf.setCapacity("root.default", 50.0f);
> csconf.setCapacity("root.dedicated", 50.0f);
> csconf.setReservable("root.dedicated", true);
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7217) PUT method for update service for Service API doesn't function correctly
[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208228#comment-16208228
]
Billie Rinaldi commented on YARN-7217:
--
bq. Flex operation was implemented sub-optimally in Slider. i.e. Increased node
count for HBase Region server, stop the service, and resume the service.
This is not the case. The service is not stopped when flexing an application.
The flex operation can be performed while the app is running and it will take
effect immediately, or it can be performed while the app is stopped, in which
case it will take effect on the next start. It is not acceptable to require
stopping the application before flexing takes effect.
> PUT method for update service for Service API doesn't function correctly
>
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch,
> YARN-7217.yarn-native-services.002.patch
>
>
> The PUT method for updateService API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for
> configuration vs status. This increase the number of action that can be
> performed. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/config
> Request Data:
> {
> "name":"[service_name]",
> "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "[service_name]",
> "state": "STOPPED|STARTED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7342) Application page doesn't show correct metrics for reservation runs
[ https://issues.apache.org/jira/browse/YARN-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208229#comment-16208229 ] Subru Krishnan commented on YARN-7342: -- [~yufeigu], thanks for raising this. Can you get this in for 2.9.0 as this looks critical? > Application page doesn't show correct metrics for reservation runs > --- > > Key: YARN-7342 > URL: https://issues.apache.org/jira/browse/YARN-7342 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler, reservation system >Affects Versions: 3.1.0 >Reporter: Yufei Gu > Attachments: Screen Shot 2017-10-16 at 17.27.48.png > > > As the screen shot shows, there are some bugs on the webUI while running job > with reservation. For examples, queue name should just be "root.queueA" > instead of internal queue name. All metrics(Allocated CPU, % of queue, etc) > are missing for reservation runs. These should be a blocker though. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4859) [Bug] Unable to submit a job to a reservation when using FairScheduler
[ https://issues.apache.org/jira/browse/YARN-4859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208227#comment-16208227 ] Subru Krishnan commented on YARN-4859: -- Thanks [~yufeigu] for the diligent work and [~asuresh] for the feedback. Can you close this with "WorksForMe" and create follow up jiras for the improvements you suggest. FYI I have committed YARN-7311. > [Bug] Unable to submit a job to a reservation when using FairScheduler > -- > > Key: YARN-4859 > URL: https://issues.apache.org/jira/browse/YARN-4859 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Subru Krishnan >Assignee: Yufei Gu >Priority: Blocker > Attachments: Screen Shot 2017-10-16 at 17.27.48.png > > > Jobs submitted to a reservation get stuck at scheduled stage when using > FairScheduler. I came across this when working on YARN-4827 (documentation > for configuring ReservationSystem for FairScheduler) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7341) TestRouterWebServiceUtil#testMergeMetrics is flakey
[
https://issues.apache.org/jira/browse/YARN-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208221#comment-16208221
]
Subru Krishnan commented on YARN-7341:
--
Thanks [~rkanter]/[~jlowe] for bringing this to my attention. I cherry-picked
this along with YARN-7095 to branch-2.
> TestRouterWebServiceUtil#testMergeMetrics is flakey
> ---
>
> Key: YARN-7341
> URL: https://issues.apache.org/jira/browse/YARN-7341
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.0.0-beta1
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: YARN-7341.001.patch
>
>
> {{TestRouterWebServiceUtil#testMergeMetrics}} is flakey. It sometimes fails
> with something like:
> {noformat}
> Running org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil
> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.252 sec <<<
> FAILURE! - in
> org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil
> testMergeMetrics(org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil)
> Time elapsed: 0.005 sec <<< FAILURE!
> java.lang.AssertionError: expected:<1092> but was:<584>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil.testMergeMetrics(TestRouterWebServiceUtil.java:473)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7341) TestRouterWebServiceUtil#testMergeMetrics is flakey
[
https://issues.apache.org/jira/browse/YARN-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-7341:
-
Target Version/s: (was: 3.0.0)
> TestRouterWebServiceUtil#testMergeMetrics is flakey
> ---
>
> Key: YARN-7341
> URL: https://issues.apache.org/jira/browse/YARN-7341
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.0.0-beta1
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: YARN-7341.001.patch
>
>
> {{TestRouterWebServiceUtil#testMergeMetrics}} is flakey. It sometimes fails
> with something like:
> {noformat}
> Running org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil
> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.252 sec <<<
> FAILURE! - in
> org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil
> testMergeMetrics(org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil)
> Time elapsed: 0.005 sec <<< FAILURE!
> java.lang.AssertionError: expected:<1092> but was:<584>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil.testMergeMetrics(TestRouterWebServiceUtil.java:473)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7341) TestRouterWebServiceUtil#testMergeMetrics is flakey
[
https://issues.apache.org/jira/browse/YARN-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-7341:
-
Fix Version/s: 3.1.0
2.9.0
> TestRouterWebServiceUtil#testMergeMetrics is flakey
> ---
>
> Key: YARN-7341
> URL: https://issues.apache.org/jira/browse/YARN-7341
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.0.0-beta1
>Reporter: Robert Kanter
>Assignee: Robert Kanter
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: YARN-7341.001.patch
>
>
> {{TestRouterWebServiceUtil#testMergeMetrics}} is flakey. It sometimes fails
> with something like:
> {noformat}
> Running org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil
> Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.252 sec <<<
> FAILURE! - in
> org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil
> testMergeMetrics(org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil)
> Time elapsed: 0.005 sec <<< FAILURE!
> java.lang.AssertionError: expected:<1092> but was:<584>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.router.webapp.TestRouterWebServiceUtil.testMergeMetrics(TestRouterWebServiceUtil.java:473)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7311) Fix TestRMWebServicesReservation parametrization for fair scheduler
[
https://issues.apache.org/jira/browse/YARN-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-7311:
-
Summary: Fix TestRMWebServicesReservation parametrization for fair
scheduler (was: TestRMWebServicesReservation doesn't really test fair
scheduler)
> Fix TestRMWebServicesReservation parametrization for fair scheduler
> ---
>
> Key: YARN-7311
> URL: https://issues.apache.org/jira/browse/YARN-7311
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, reservation system
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-7311.001.patch, YARN-7311.002.patch,
> YARN-7311.WIP.patch
>
>
> YARN-4248 introduced the REST API for submit/update/delete Reservations.
> Class {{TestRMWebServicesReservation}} intends to test both FS and CS. The
> test cases designed for fair scheduler actually test capacity scheduler. The
> following code in method {{configureServlets}} shows it sets the scheduler to
> CS even test cases are for fair scheduler.
> {code}
> conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
> ResourceScheduler.class);
> CapacitySchedulerConfiguration csconf =
> new CapacitySchedulerConfiguration(conf);
> String[] queues = { "default", "dedicated" };
> csconf.setQueues("root", queues);
> csconf.setCapacity("root.default", 50.0f);
> csconf.setCapacity("root.dedicated", 50.0f);
> csconf.setReservable("root.dedicated", true);
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7295) Backport ATSv2 to branch-2
[
https://issues.apache.org/jira/browse/YARN-7295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208194#comment-16208194
]
Hadoop QA commented on YARN-7295:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 102 new or modified
test files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
48s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
21s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
48s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
25s{color} | {color:green} branch-2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 12m
42s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
49s{color} | {color:green} branch-2 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 34s{color}
| {color:red} root generated 1 new + 1445 unchanged - 0 fixed = 1446 total (was
1445) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 44s{color} | {color:orange} root: The patch generated 77 new + 3545
unchanged - 304 fixed = 3622 total (was 3849) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 10m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
8s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
2s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
8s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-project hadoop-assemblies hadoop-yarn-project/hadoop-yarn
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site hadoop-yarn-project . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 18m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 26m 31s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green}
[jira] [Commented] (YARN-7217) PUT method for update service for Service API doesn't function correctly
[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208192#comment-16208192
]
Eric Yang commented on YARN-7217:
-
[~gsaha] said:
{quote}
Are you saying that the API will not support flexing of no of containers when
the service is running?
{quote}
Flex operation was implemented sub-optimally in Slider. i.e. Increased node
count for HBase Region server, stop the service, and resume the service.
Should the service resume from 4 nodes that was initially started or 10 nodes
that was later increased? While Slider had a noble idea to have flex
operation, but an operation without storing configuration can cause operation
hazard because the operation become non-repeatable.
YARN-7216 is proposed to decouple configuration changes and performing an
operation. This JIRA propose to separate out PUT method for updateService into
two calls for config changes, and service operation. This will provide more
insights into over all transaction, and provide ability to reproduce.
Do we need a flex operation without restart service? The answer is likely no.
There are 3 services (Datanode, NodeManager, HBase region server) that can add
slave nodes without restarting masters. The majority of use case for changing
node count will result in configuration changes and force restart. Majority of
software follow the second model to make config changes, then restart services.
This ensure the change request is repeatable. Hence, the need to support flex
operation without configuration change would be greatly reduced.
> PUT method for update service for Service API doesn't function correctly
>
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
>Reporter: Eric Yang
>Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch,
> YARN-7217.yarn-native-services.002.patch
>
>
> The PUT method for updateService API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for
> configuration vs status. This increase the number of action that can be
> performed. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/config
> Request Data:
> {
> "name":"[service_name]",
> "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "[service_name]",
> "state": "STOPPED|STARTED"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5534) Allow whitelisted volume mounts
[ https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208161#comment-16208161 ] Eric Yang commented on YARN-5534: - [~ebadger] YARN-6623 is committed, but there seems to have some issues in the implementation that it worked on Ubuntu but not CentOS. We need to monitor the development of YARN-7344 to determine if we are on the right path. Can we wait a few days before deciding closure of this JIRA? Thanks > Allow whitelisted volume mounts > > > Key: YARN-5534 > URL: https://issues.apache.org/jira/browse/YARN-5534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: luhuichun >Assignee: Shane Kumpf > Attachments: YARN-5534.001.patch, YARN-5534.002.patch, > YARN-5534.003.patch > > > Introduction > Mounting files or directories from the host is one way of passing > configuration and other information into a docker container. > We could allow the user to set a list of mounts in the environment of > ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2). > These would be mounted read-only to the specified target locations. This has > been resolved in YARN-4595 > 2.Problem Definition > Bug mounting arbitrary volumes into a Docker container can be a security risk. > 3.Possible solutions > one approach to provide safe mounts is to allow the cluster administrator to > configure a set of parent directories as white list mounting directories. > Add a property named yarn.nodemanager.volume-mounts.white-list, when > container executor do mount checking, only the allowed directories or > sub-directories can be mounted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7286) Add support for docker to have no capabilities
[
https://issues.apache.org/jira/browse/YARN-7286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208159#comment-16208159
]
Jason Lowe commented on YARN-7286:
--
Thanks for updating the patch!
Copy-n-paste error on the assert message here. It doesn't match the test case.
{code}
conf.setStrings(YarnConfiguration.NM_DOCKER_CONTAINER_CAPABILITIES,
"CHOWN", "DAC_OVERRIDE", "NONE");
runtime.initialize(conf);
Assert.fail("Initialize didn't fail with invalid capabilities " +
"'none', 'CHOWN', 'DAC_OVERRIDE'");
{code}
There's no need to surround unit test code with a try/catch block and
explicitly fail when we do not expect it to throw. The test will just fail
naturally if it does throw.
> Add support for docker to have no capabilities
> --
>
> Key: YARN-7286
> URL: https://issues.apache.org/jira/browse/YARN-7286
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Eric Badger
>Assignee: Eric Badger
> Attachments: YARN-7286.001.patch, YARN-7286.002.patch,
> YARN-7286.003.patch, YARN-7286.004.patch, YARN-7286.005.patch,
> YARN-7286.006.patch
>
>
> Support for controlling capabilities was introduced in YARN-4258. However, it
> does not allow for the capabilities list to be NULL, since {{getStrings()}}
> will treat an empty value the same as it treats an unset property. So, a NULL
> list will actually give the default capabilities list.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7311) TestRMWebServicesReservation doesn't really test fair scheduler
[
https://issues.apache.org/jira/browse/YARN-7311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208154#comment-16208154
]
Sean Po commented on YARN-7311:
---
Thanks [~yufeigu] for working on this patch. The changes in the tests look good
to me, +1.
> TestRMWebServicesReservation doesn't really test fair scheduler
> ---
>
> Key: YARN-7311
> URL: https://issues.apache.org/jira/browse/YARN-7311
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, reservation system
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-7311.001.patch, YARN-7311.002.patch,
> YARN-7311.WIP.patch
>
>
> YARN-4248 introduced the REST API for submit/update/delete Reservations.
> Class {{TestRMWebServicesReservation}} intends to test both FS and CS. The
> test cases designed for fair scheduler actually test capacity scheduler. The
> following code in method {{configureServlets}} shows it sets the scheduler to
> CS even test cases are for fair scheduler.
> {code}
> conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
> ResourceScheduler.class);
> CapacitySchedulerConfiguration csconf =
> new CapacitySchedulerConfiguration(conf);
> String[] queues = { "default", "dedicated" };
> csconf.setQueues("root", queues);
> csconf.setCapacity("root.default", 50.0f);
> csconf.setCapacity("root.dedicated", 50.0f);
> csconf.setReservable("root.dedicated", true);
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7244) ShuffleHandler is not aware of disks that are added
[ https://issues.apache.org/jira/browse/YARN-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kuhu Shukla updated YARN-7244: -- Attachment: YARN-7244.011.patch Updated patch addressing comments from [~sunilg]. > ShuffleHandler is not aware of disks that are added > --- > > Key: YARN-7244 > URL: https://issues.apache.org/jira/browse/YARN-7244 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: YARN-7244.001.patch, YARN-7244.002.patch, > YARN-7244.003.patch, YARN-7244.004.patch, YARN-7244.005.patch, > YARN-7244.006.patch, YARN-7244.007.patch, YARN-7244.008.patch, > YARN-7244.009.patch, YARN-7244.010.patch, YARN-7244.011.patch > > > The ShuffleHandler permanently remembers the list of "good" disks on NM > startup. If disks later are added to the node then map tasks will start using > them but the ShuffleHandler will not be aware of them. The end result is that > the data cannot be shuffled from the node leading to fetch failures and > re-runs of the map tasks. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7338) Support same origin policy for cross site scripting prevention.
[
https://issues.apache.org/jira/browse/YARN-7338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208131#comment-16208131
]
Eric Yang commented on YARN-7338:
-
[~sunilg] It is very important to have CORS header included for Javascript, if
javascript is utilizing third party libraries. In ui2 case, it downloads a
number of third party javascript libraries during build time. Third party
javascript libraries has potential to enable hacker to trigger unexpected
javascript calls to leak information to other servers. CORS header will help
to ground the communication between browser and servers to the same origin.
Here is an example of Bank of America website javascript.
Request
{code}
Request URL:https://aero.bankofamerica.com/30306/I3n.js
Request Method:GET
Status Code:200 OK
Remote Address:123.123.123.123:443
Referrer Policy:no-referrer-when-downgrade
{code}
Response Headers
{code}
Access-Control-Allow-Credentials:true
Access-Control-Allow-Methods:GET, OPTIONS
Access-Control-Allow-Origin:https://www.bankofamerica.com
Cache-Control:no-cache, no-store, must-revalidate
Connection:keep-alive
Content-Encoding:gzip
Content-Type:application/x-javascript
Date:Tue, 17 Oct 2017 18:22:23 GMT
{code}
There is Access-Control-Allow-Origin header being sent from server.
It is best to start the leak prevention before mistake is made.
> Support same origin policy for cross site scripting prevention.
> ---
>
> Key: YARN-7338
> URL: https://issues.apache.org/jira/browse/YARN-7338
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Vrushali C
>
> Opening jira as suggested b [~eyang] on the thread for merging YARN-3368 (new
> web UI) to branch2
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201610.mbox/%3ccad++ecmvvqnzqz9ynkvkcxaczdkg50yiofxktgk3mmms9sh...@mail.gmail.com%3E
> --
> Ui2 does not seem to support same origin policy for cross site scripting
> prevention.
> The following parameters has no effect for /ui2:
> hadoop.http.cross-origin.enabled = true
> yarn.resourcemanager.webapp.cross-origin.enabled = true
> This is because ui2 is designed as a separate web application. WebFilters
> setup for existing resource manager doesn’t apply to the new web application.
> Please open JIRA to track the security issue and resolve the problem prior to
> backporting this to branch-2.
> This would minimize the risk to open up security hole in branch-2.
> --
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7323) Data structure update in service REST API
[ https://issues.apache.org/jira/browse/YARN-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7323: -- Description: Changes include - removed the functionality of creating default component if no component specified, this means we always expect user to specify a component. - removed number_of_containers, launch_command, containers from the Service resource object, assuming that each Component has to specify these. resource and artifact in Service object will still remain as the default value for component. - Added a ComponentState object. - Changed the actionCreate to look for the example service from a secondary location. - Fixed a bug in ComponentInstance to handle STOP event on INIT, in case container completed before it gets launched. was: Changes include - removed the functionality of creating default component if no component specified, this means we always expect user to specify a component. - removed number_of_containers, launch_command, containers from the Service resource object, assuming that each Component has to - specify these. resource and artifact in Service object will still remain as the default value for component. - Added a ComponentState object. - Changed the actionCreate to look for the example service from a secondary location. - Fixed a bug in ComponentInstance to handle STOP event on INIT, in case container completed before it gets launched. > Data structure update in service REST API > - > > Key: YARN-7323 > URL: https://issues.apache.org/jira/browse/YARN-7323 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7323.yarn-native-services.01.patch, > YARN-7323.yarn-native-services.02.patch, > YARN-7323.yarn-native-services.03.patch > > > Changes include > - removed the functionality of creating default component if no component > specified, this means we always expect user to specify a component. > - removed number_of_containers, launch_command, containers from the Service > resource object, assuming that each Component has to > specify these. resource and artifact in Service object will still remain as > the default value for component. > - Added a ComponentState object. > - Changed the actionCreate to look for the example service from a secondary > location. > - Fixed a bug in ComponentInstance to handle STOP event on INIT, in case > container completed before it gets launched. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7323) Data structure update in service REST API
[ https://issues.apache.org/jira/browse/YARN-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-7323: -- Description: Changes include - removed the functionality of creating default component if no component specified, this means we always expect user to specify a component. - removed number_of_containers, launch_command, containers from the Service resource object, assuming that each Component has to - specify these. resource and artifact in Service object will still remain as the default value for component. - Added a ComponentState object. - Changed the actionCreate to look for the example service from a secondary location. - Fixed a bug in ComponentInstance to handle STOP event on INIT, in case container completed before it gets launched. was: Unix terminology implies daemon process, such as httpd as a service. In Hadoop terminology, hdfs is a file system service, and hdfs has various components such as namenode, journal nodes, and datanodes. YARN service data model was more closely describing daemon process as a service. This JIRA propose to remap the YARN service data model to align with Hadoop terminology. Service is composed of a collection of components. Component is either a unix process or docker container. Data structure for Yarnfile organized launched_command and state at Service level, but the grouping definition was following Hadoop terminology. For clarity, we will change data structure to match Hadoop terminology. State and launch commands are moved to component level. > Data structure update in service REST API > - > > Key: YARN-7323 > URL: https://issues.apache.org/jira/browse/YARN-7323 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-7323.yarn-native-services.01.patch, > YARN-7323.yarn-native-services.02.patch, > YARN-7323.yarn-native-services.03.patch > > > Changes include > - removed the functionality of creating default component if no component > specified, this means we always expect user to specify a component. > - removed number_of_containers, launch_command, containers from the Service > resource object, assuming that each Component has to - > specify these. resource and artifact in Service object will still remain as > the default value for component. > - Added a ComponentState object. > - Changed the actionCreate to look for the example service from a secondary > location. > - Fixed a bug in ComponentInstance to handle STOP event on INIT, in case > container completed before it gets launched. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7345) Incorrect minor device numbers written to devices.deny file
[ https://issues.apache.org/jira/browse/YARN-7345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16208114#comment-16208114 ] Jonathan Hung commented on YARN-7345: - 001 patch fixes issue and adds test which blocks 1,3 devices. > Incorrect minor device numbers written to devices.deny file > --- > > Key: YARN-7345 > URL: https://issues.apache.org/jira/browse/YARN-7345 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-7345.001.patch > > > Currently the minor numbers written to devices.deny file is 0->(num devices > to block - 1). But blocked devices are not necessarily sequential starting > from 0. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
