[jira] [Created] (YARN-7507) TestNodeLabelContainerAllocation failing in trunk
Bibin A Chundatt created YARN-7507:
--
Summary: TestNodeLabelContainerAllocation failing in trunk
Key: YARN-7507
URL: https://issues.apache.org/jira/browse/YARN-7507
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bibin A Chundatt
https://builds.apache.org/job/PreCommit-YARN-Build/18498/testReport/
{code}
TestNodeLabelContainerAllocation.testPreferenceOfNeedyPrioritiesUnderSameAppTowardsNodePartitions:786->checkPendingResource:557
expected:<1024> but was:<0>
TestNodeLabelContainerAllocation.testPreferenceOfQueuesTowardsNodePartitions:985->checkPendingResource:557
expected:<5120> but was:<0>
TestNodeLabelContainerAllocation.testQueueMetricsWithLabels:1962 expected:<0>
but was:<1024>
TestNodeLabelContainerAllocation.testQueueMetricsWithLabelsOnDefaultLabelNode:2065
expected:<1024> but was:<2048>
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7500) LogAggregation DeletionService should consider completedTime for long running jobs
[ https://issues.apache.org/jira/browse/YARN-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254905#comment-16254905 ] Prabhu Joseph commented on YARN-7500: - [~jlowe] Oh yes, missed it. The issue is our customer has a long running custom Yarn Application which has started before yarn.log-aggregation.retain-seconds and was running yesterday as per the RM UI, and today we din see any logs under app-logs. Logs where there for running job. Looks only possibility is the custom app has not updated the logs for many days. Will check RM logs and hdfs-audit logs to validate and give more information. > LogAggregation DeletionService should consider completedTime for long running > jobs > -- > > Key: YARN-7500 > URL: https://issues.apache.org/jira/browse/YARN-7500 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Prabhu Joseph > > Currently LogAggregation deletes the application logs based on start time of > the job. For long running jobs (started before > yarn.log-aggregation.retain-seconds), say it is failed yesterday for some > reason and we won't have the job logs today for debugging. > Better to consider the completedTime of the job as part of the deletion > condition. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7508) NPE in FiCaSchedulerApp when debug log enabled and try to commit outdated reserved proposal in async-scheduling mode
Tao Yang created YARN-7508:
--
Summary: NPE in FiCaSchedulerApp when debug log enabled and try to
commit outdated reserved proposal in async-scheduling mode
Key: YARN-7508
URL: https://issues.apache.org/jira/browse/YARN-7508
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler
Affects Versions: 3.0.0-alpha4, 2.9.0
Reporter: Tao Yang
Assignee: Tao Yang
YARN-6678 have fixed the IllegalStateException problem but the debug log it
added may cause NPE when trying to print containerId of non-existed reserved
container on this node. Replace
{{schedulerContainer.getSchedulerNode().getReservedContainer().getContainerId()}}
with {{schedulerContainer.getSchedulerNode().getReservedContainer()}} can fix
this problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7492) Set up SASS for new YARN UI styling
[ https://issues.apache.org/jira/browse/YARN-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7492: -- Summary: Set up SASS for new YARN UI styling (was: Set up SASS for UI styling) > Set up SASS for new YARN UI styling > --- > > Key: YARN-7492 > URL: https://issues.apache.org/jira/browse/YARN-7492 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Attachments: YARN-7492.001.patch, YARN-7492.002.patch, > YARN-7492.003.patch, YARN-7492.004.patch > > > SASS will help in improving the quality and maintainablity of our styles. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7508) NPE in FiCaSchedulerApp when debug log enabled and try to commit outdated reserved proposal in async-scheduling mode
[
https://issues.apache.org/jira/browse/YARN-7508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Yang updated YARN-7508:
---
Attachment: YARN-7508.001.patch
Uploading v1 patch. [~sunilg], Could you help to review, please?
> NPE in FiCaSchedulerApp when debug log enabled and try to commit outdated
> reserved proposal in async-scheduling mode
>
>
> Key: YARN-7508
> URL: https://issues.apache.org/jira/browse/YARN-7508
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 2.9.0, 3.0.0-alpha4
>Reporter: Tao Yang
>Assignee: Tao Yang
> Attachments: YARN-7508.001.patch
>
>
> YARN-6678 have fixed the IllegalStateException problem but the debug log it
> added may cause NPE when trying to print containerId of non-existed reserved
> container on this node. Replace
> {{schedulerContainer.getSchedulerNode().getReservedContainer().getContainerId()}}
> with {{schedulerContainer.getSchedulerNode().getReservedContainer()}} can
> fix this problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7489) ConcurrentModificationException in RMAppImpl#getRMAppMetrics
[
https://issues.apache.org/jira/browse/YARN-7489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254945#comment-16254945
]
Bibin A Chundatt commented on YARN-7489:
+1 LGTM. [~rohithsharma] any comments??
> ConcurrentModificationException in RMAppImpl#getRMAppMetrics
>
>
> Key: YARN-7489
> URL: https://issues.apache.org/jira/browse/YARN-7489
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Reporter: Tao Yang
>Assignee: Tao Yang
> Attachments: YARN-7489.001.patch
>
>
> The REST clients have sometimes failed to query applications through apps
> REST API in RMWebService and it happened when iterating
> attempts(RMWebServices#getApps --> AppInfo# -->
> RMAppImpl#getRMAppMetrics) and meanwhile these attempts
> changed(AttemptFailedTransition#transition -->
> RMAppImpl#createAndStartNewAttempt --> RMAppImpl#createNewAttempt).
> Application state changed within the lockup period of writeLock in RMAppImpl,
> so that we can add readLock before iterating attempts to fix this problem.
> Exception stack:
> {noformat}
> java.util.ConcurrentModificationException
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
> at
> java.util.LinkedHashMap$LinkedValueIterator.next(LinkedHashMap.java:747)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.getRMAppMetrics(RMAppImpl.java:1487)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.AppInfo.(AppInfo.java:199)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebServices.getApps(RMWebServices.java:597)
> at sun.reflect.GeneratedMethodAccessor81.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:178)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
> at
> com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
> at
> com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7492) Set up SASS for new YARN UI styling
[ https://issues.apache.org/jira/browse/YARN-7492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254949#comment-16254949 ] Hudson commented on YARN-7492: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13243 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13243/]) YARN-7492. Set up SASS for new YARN UI styling. Contributed by Vasudevan (sunilg: rev 09a13426086aed9d1bc63a844858dcac560763a6) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/components/per-app-memusage-by-nodes-stacked-barchart.js * (edit) .gitignore * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/components/nodes-heatmap.js * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/styles/app.scss * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/styles/colors.scss * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/components/tree-selector.js * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/package.json * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/styles/variables.scss * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/styles/app.css * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/app/utils/color-utils.js > Set up SASS for new YARN UI styling > --- > > Key: YARN-7492 > URL: https://issues.apache.org/jira/browse/YARN-7492 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Reporter: Vasudevan Skm >Assignee: Vasudevan Skm > Fix For: 3.1.0 > > Attachments: YARN-7492.001.patch, YARN-7492.002.patch, > YARN-7492.003.patch, YARN-7492.004.patch > > > SASS will help in improving the quality and maintainablity of our styles. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7497) Add HDFSSchedulerConfigurationStore for RM HA
[
https://issues.apache.org/jira/browse/YARN-7497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254976#comment-16254976
]
Hadoop QA commented on YARN-7497:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 34s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
10s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
11s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 53s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 16 new + 290 unchanged - 0 fixed = 306 total (was 290) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
15s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 40s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}124m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| | new java.io.IOException(String) not thrown in
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.HDFSSchedulerConfigurationStore.initialize(Configuration,
Configuration, RMContext) At
[jira] [Updated] (YARN-7482) Max applications calculation per queue has to be retrospected with absolute resource support
[ https://issues.apache.org/jira/browse/YARN-7482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7482: -- Attachment: YARN-7482-YARN-5881.001.patch v1 patch. This helps to fix the max application pblm in LeafQueue. cc/ [~rohithsharma] > Max applications calculation per queue has to be retrospected with absolute > resource support > > > Key: YARN-7482 > URL: https://issues.apache.org/jira/browse/YARN-7482 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: YARN-5881 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7482-YARN-5881.001.patch > > > Retrospect max system applications w.r.t absolute resource configuration in a > queue -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7363) ContainerLocalizer don't have a valid log4j config in case of Linux container executor
[ https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-7363: --- Attachment: YARN-7363.001.patch > ContainerLocalizer don't have a valid log4j config in case of Linux container > executor > -- > > Key: YARN-7363 > URL: https://issues.apache.org/jira/browse/YARN-7363 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-7363.001.patch > > > In case of Linux container executor, ContainerLocalizer run as a separated > process. It doesn't access a valid log4j.properties when the application user > is not in the "hadoop" group. The log4j.properties of node manager is in its > classpath, but it isn't readable by users not in hadoop group due to the > security concern. In that case, ContainerLocalizer doesn't have a valid log4j > configuration, and normally no log output. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7363) ContainerLocalizer don't have a valid log4j config in case of Linux container executor
[ https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254995#comment-16254995 ] Yufei Gu commented on YARN-7363: Patch v1 * sets file "container-log4j.properties" as the log4j configuration. * reuses the appender CLA in "container-log4j.properties". * sets the ContainerLocalizer's log file name to "container-localizer-syslog" > ContainerLocalizer don't have a valid log4j config in case of Linux container > executor > -- > > Key: YARN-7363 > URL: https://issues.apache.org/jira/browse/YARN-7363 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-7363.001.patch > > > In case of Linux container executor, ContainerLocalizer run as a separated > process. It doesn't access a valid log4j.properties when the application user > is not in the "hadoop" group. The log4j.properties of node manager is in its > classpath, but it isn't readable by users not in hadoop group due to the > security concern. In that case, ContainerLocalizer doesn't have a valid log4j > configuration, and normally no log output. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7363) ContainerLocalizer don't have a valid log4j config in case of Linux container executor
[
https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16254995#comment-16254995
]
Yufei Gu edited comment on YARN-7363 at 11/16/17 9:24 AM:
--
Patch v1 does these things for process {{ContainerLocalizer}}.
* sets file "container-log4j.properties" as the log4j configuration.
* reuses the appender CLA in "container-log4j.properties".
* sets the ContainerLocalizer's log file name to "container-localizer-syslog"
was (Author: yufeigu):
Patch v1
* sets file "container-log4j.properties" as the log4j configuration.
* reuses the appender CLA in "container-log4j.properties".
* sets the ContainerLocalizer's log file name to "container-localizer-syslog"
> ContainerLocalizer don't have a valid log4j config in case of Linux container
> executor
> --
>
> Key: YARN-7363
> URL: https://issues.apache.org/jira/browse/YARN-7363
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-7363.001.patch
>
>
> In case of Linux container executor, ContainerLocalizer run as a separated
> process. It doesn't access a valid log4j.properties when the application user
> is not in the "hadoop" group. The log4j.properties of node manager is in its
> classpath, but it isn't readable by users not in hadoop group due to the
> security concern. In that case, ContainerLocalizer doesn't have a valid log4j
> configuration, and normally no log output.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7430) User and Group mapping are incorrect in docker container
[ https://issues.apache.org/jira/browse/YARN-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255005#comment-16255005 ] Varun Vasudev commented on YARN-7430: - Looks like we have consensus of making uid:gid the default behavior. +1 from my end. I'll commit this tomorrow unless someone objects. > User and Group mapping are incorrect in docker container > > > Key: YARN-7430 > URL: https://issues.apache.org/jira/browse/YARN-7430 > Project: Hadoop YARN > Issue Type: Sub-task > Components: security, yarn >Affects Versions: 2.9.0, 3.0.0 >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Blocker > Attachments: YARN-7430.001.patch, YARN-7430.png > > > In YARN-4266, the recommendation was to use -u [uid]:[gid] numeric values to > enforce user and group for the running user. In YARN-6623, this translated > to --user=test --group-add=group1. The code no longer enforce group > correctly for launched process. > In addition, the implementation in YARN-6623 requires the user and group > information to exist in container to translate username and group to uid/gid. > For users on LDAP, there is no good way to populate container with user and > group information. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7508) NPE in FiCaSchedulerApp when debug log enabled and try to commit outdated reserved proposal in async-scheduling mode
[
https://issues.apache.org/jira/browse/YARN-7508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255051#comment-16255051
]
Sunil G commented on YARN-7508:
---
I think its fine to change to
{{schedulerContainer.getSchedulerNode().getReservedContainer()}}
Is there any more instance of similar usage? If so we can move that also as per
this patch's proposal.
> NPE in FiCaSchedulerApp when debug log enabled and try to commit outdated
> reserved proposal in async-scheduling mode
>
>
> Key: YARN-7508
> URL: https://issues.apache.org/jira/browse/YARN-7508
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 2.9.0, 3.0.0-alpha4
>Reporter: Tao Yang
>Assignee: Tao Yang
> Attachments: YARN-7508.001.patch
>
>
> YARN-6678 have fixed the IllegalStateException problem but the debug log it
> added may cause NPE when trying to print containerId of non-existed reserved
> container on this node. Replace
> {{schedulerContainer.getSchedulerNode().getReservedContainer().getContainerId()}}
> with {{schedulerContainer.getSchedulerNode().getReservedContainer()}} can
> fix this problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7509) AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby
[ https://issues.apache.org/jira/browse/YARN-7509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Yang reassigned YARN-7509: -- Assignee: Tao Yang > AsyncScheduleThread and ResourceCommitterService are still running after RM > is transitioned to standby > -- > > Key: YARN-7509 > URL: https://issues.apache.org/jira/browse/YARN-7509 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha4, 2.9.1 >Reporter: Tao Yang >Assignee: Tao Yang > > After RM is transitioned to standby, AsyncScheduleThread and > ResourceCommitterService will receive interrupt signal. When thread is > sleeping, it will ignore the interrupt signal since InterruptedException is > catched inside and the interrupt signal is cleared. > For AsyncScheduleThread, InterruptedException was catched and ignored in > CapacityScheduler#schedule. > For ResourceCommitterService, InterruptedException was catched inside and > ignored in ResourceCommitterService#run. > We should let the interrupt signal out and make these threads exit. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7509) AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby
Tao Yang created YARN-7509: -- Summary: AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby Key: YARN-7509 URL: https://issues.apache.org/jira/browse/YARN-7509 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0-alpha4, 2.9.1 Reporter: Tao Yang After RM is transitioned to standby, AsyncScheduleThread and ResourceCommitterService will receive interrupt signal. When thread is sleeping, it will ignore the interrupt signal since InterruptedException is catched inside and the interrupt signal is cleared. For AsyncScheduleThread, InterruptedException was catched and ignored in CapacityScheduler#schedule. For ResourceCommitterService, InterruptedException was catched inside and ignored in ResourceCommitterService#run. We should let the interrupt signal out and make these threads exit. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7508) NPE in FiCaSchedulerApp when debug log enabled and try to commit outdated reserved proposal in async-scheduling mode
[
https://issues.apache.org/jira/browse/YARN-7508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255059#comment-16255059
]
Hadoop QA commented on YARN-7508:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
58s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 47s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands |
| |
org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA |
| |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| | org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7508 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897942/YARN-7508.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b885b8eb0e6a 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
|
[jira] [Updated] (YARN-7509) AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby
[ https://issues.apache.org/jira/browse/YARN-7509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Yang updated YARN-7509: --- Attachment: YARN-7509.001.patch Attaching v1 patch for review. > AsyncScheduleThread and ResourceCommitterService are still running after RM > is transitioned to standby > -- > > Key: YARN-7509 > URL: https://issues.apache.org/jira/browse/YARN-7509 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha4, 2.9.1 >Reporter: Tao Yang >Assignee: Tao Yang > Attachments: YARN-7509.001.patch > > > After RM is transitioned to standby, AsyncScheduleThread and > ResourceCommitterService will receive interrupt signal. When thread is > sleeping, it will ignore the interrupt signal since InterruptedException is > catched inside and the interrupt signal is cleared. > For AsyncScheduleThread, InterruptedException was catched and ignored in > CapacityScheduler#schedule. > For ResourceCommitterService, InterruptedException was catched inside and > ignored in ResourceCommitterService#run. > We should let the interrupt signal out and make these threads exit. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7363) ContainerLocalizer don't have a valid log4j config in case of Linux container executor
[
https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255098#comment-16255098
]
Hadoop QA commented on YARN-7363:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 37s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 17m 2s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7363 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897948/YARN-7363.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 22dae6c14a7c 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 462e25a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/18519/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18519/testReport/ |
| Max. process+thread count | 435 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/
[jira] [Commented] (YARN-7363) ContainerLocalizer don't have a valid log4j config in case of Linux container executor
[
https://issues.apache.org/jira/browse/YARN-7363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255105#comment-16255105
]
Hadoop QA commented on YARN-7363:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 16s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7363 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897948/YARN-7363.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 95fe8c4e8a00 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 462e25a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/18521/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18521/testReport/ |
| Max. process+thread count | 442 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/
[jira] [Created] (YARN-7510) Merge work for YARN-5881
Sunil G created YARN-7510: - Summary: Merge work for YARN-5881 Key: YARN-7510 URL: https://issues.apache.org/jira/browse/YARN-7510 Project: Hadoop YARN Issue Type: Sub-task Components: capacity scheduler Reporter: Sunil G Assignee: Sunil G Merge YARN-5881 work -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7510) Merge work for YARN-5881
[ https://issues.apache.org/jira/browse/YARN-7510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7510: -- Attachment: YARN-7510.001.patch Attaching an initial branch patch for jenkins > Merge work for YARN-5881 > > > Key: YARN-7510 > URL: https://issues.apache.org/jira/browse/YARN-7510 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7510.001.patch > > > Merge YARN-5881 work -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7482) Max applications calculation per queue has to be retrospected with absolute resource support
[
https://issues.apache.org/jira/browse/YARN-7482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255159#comment-16255159
]
Hadoop QA commented on YARN-7482:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-5881 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
10s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} YARN-5881 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 27s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 55s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerWithMultiResourceTypes
|
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands |
| |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| | org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7482 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897947/YARN-7482-YARN-5881.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9a4f7bda6023 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-5881 / 39f43eb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
h
[jira] [Created] (YARN-7511) NPE in ContainerLocalizer when localization failed for running container
Tao Yang created YARN-7511:
--
Summary: NPE in ContainerLocalizer when localization failed for
running container
Key: YARN-7511
URL: https://issues.apache.org/jira/browse/YARN-7511
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 3.0.0-alpha4, 2.9.1
Reporter: Tao Yang
Assignee: Tao Yang
Error log:
{noformat}
2017-09-30 20:14:32,839 FATAL [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
java.lang.NullPointerException
ย ย ย ย at
java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106)
ย ย ย ย at
java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
ย ย ย ย at
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceSet.resourceLocalizationFailed(ResourceSet.java:151)
ย ย ย ย at
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$ResourceLocalizationFailedWhileRunningTransition.transition(ContainerImpl.java:821)
ย ย ย ย at
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$ResourceLocalizationFailedWhileRunningTransition.transition(ContainerImpl.java:813)
ย ย ย ย at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
ย ย ย ย at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
ย ย ย ย at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
ย ย ย ย at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
ย ย ย ย at
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:1335)
ย ย ย ย at
org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:95)
ย ย ย ย at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1372)
ย ย ย ย at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1365)
ย ย ย ย at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
ย ย ย ย at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
ย ย ย ย at java.lang.Thread.run(Thread.java:834)
2017-09-30 20:14:32,842 INFO [AsyncDispatcher ShutDown handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
{noformat}
Reproduce this problem:
1. Container was running and ContainerManagerImpl#localize was called for this
container
2. Localization failed in ResourceLocalizationService$LocalizerRunner#run and
sent out ContainerResourceFailedEvent with null LocalResourceRequest.
3. NPE when ResourceLocalizationFailedWhileRunningTransition#transition -->
container.resourceSet.resourceLocalizationFailed(null)
I think we can fix this problem through ensuring that request is not null
before remove it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7482) Max applications calculation per queue has to be retrospected with absolute resource support
[
https://issues.apache.org/jira/browse/YARN-7482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255177#comment-16255177
]
Hadoop QA commented on YARN-7482:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-5881 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
56s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} YARN-5881 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 56s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 11s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}117m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerWithMultiResourceTypes
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7482 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12897947/YARN-7482-YARN-5881.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux c9b039519a0c 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-5881 / 39f43eb |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/18520/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/18520/testReport/
[jira] [Updated] (YARN-7511) NPE in ContainerLocalizer when localization failed for running container
[
https://issues.apache.org/jira/browse/YARN-7511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Yang updated YARN-7511:
---
Attachment: YARN-7511.001.patch
Attaching v1 patch for review.
> NPE in ContainerLocalizer when localization failed for running container
>
>
> Key: YARN-7511
> URL: https://issues.apache.org/jira/browse/YARN-7511
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-alpha4, 2.9.1
>Reporter: Tao Yang
>Assignee: Tao Yang
> Attachments: YARN-7511.001.patch
>
>
> Error log:
> {noformat}
> 2017-09-30 20:14:32,839 FATAL [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
> java.lang.NullPointerException
> ย ย ย ย at
> java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106)
> ย ย ย ย at
> java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
> ย ย ย ย at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceSet.resourceLocalizationFailed(ResourceSet.java:151)
> ย ย ย ย at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$ResourceLocalizationFailedWhileRunningTransition.transition(ContainerImpl.java:821)
> ย ย ย ย at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$ResourceLocalizationFailedWhileRunningTransition.transition(ContainerImpl.java:813)
> ย ย ย ย at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
> ย ย ย ย at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> ย ย ย ย at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> ย ย ย ย at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> ย ย ย ย at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:1335)
> ย ย ย ย at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:95)
> ย ย ย ย at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1372)
> ย ย ย ย at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1365)
> ย ย ย ย at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
> ย ย ย ย at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
> ย ย ย ย at java.lang.Thread.run(Thread.java:834)
> 2017-09-30 20:14:32,842 INFO [AsyncDispatcher ShutDown handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
> {noformat}
> Reproduce this problem:
> 1. Container was running and ContainerManagerImpl#localize was called for
> this container
> 2. Localization failed in ResourceLocalizationService$LocalizerRunner#run and
> sent out ContainerResourceFailedEvent with null LocalResourceRequest.
> 3. NPE when ResourceLocalizationFailedWhileRunningTransition#transition -->
> container.resourceSet.resourceLocalizationFailed(null)
> I think we can fix this problem through ensuring that request is not null
> before remove it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7482) Max applications calculation per queue has to be retrospected with absolute resource support
[ https://issues.apache.org/jira/browse/YARN-7482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255198#comment-16255198 ] Rohith Sharma K S commented on YARN-7482: - +1, will commit it soon. [~sunilg] can you confirm one test case failed in both the QA run. Is it related to patch? > Max applications calculation per queue has to be retrospected with absolute > resource support > > > Key: YARN-7482 > URL: https://issues.apache.org/jira/browse/YARN-7482 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: YARN-5881 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7482-YARN-5881.001.patch > > > Retrospect max system applications w.r.t absolute resource configuration in a > queue -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7497) Add HDFSSchedulerConfigurationStore for RM HA
[ https://issues.apache.org/jira/browse/YARN-7497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiandan Yang updated YARN-7497: Attachment: YARN-7497.004.patch fix findbug error > Add HDFSSchedulerConfigurationStore for RM HA > - > > Key: YARN-7497 > URL: https://issues.apache.org/jira/browse/YARN-7497 > Project: Hadoop YARN > Issue Type: New Feature > Components: yarn >Reporter: Jiandan Yang > Attachments: YARN-7497.001.patch, YARN-7497.002.patch, > YARN-7497.003.patch, YARN-7497.004.patch > > > YARN-5947 add LeveldbConfigurationStore using Leveldb as backing store, but > it does not support Yarn RM HA. > YARN-6840 supports RM HA, but too many scheduler configurations may exceed > znode limit, for example 10 thousand queues. > HDFSSchedulerConfigurationStore store conf file in HDFS, when RM failover, > new active RM can load scheduler configuration from HDFS. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7508) NPE in FiCaSchedulerApp when debug log enabled and try to commit outdated reserved proposal in async-scheduling mode
[
https://issues.apache.org/jira/browse/YARN-7508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255213#comment-16255213
]
Bibin A Chundatt commented on YARN-7508:
{{schedulerContainer.getSchedulerNode().getReservedContainer()}} will log
{{null}} for above case and other case {{containerId.toString()}}.i think
should be fine
> NPE in FiCaSchedulerApp when debug log enabled and try to commit outdated
> reserved proposal in async-scheduling mode
>
>
> Key: YARN-7508
> URL: https://issues.apache.org/jira/browse/YARN-7508
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 2.9.0, 3.0.0-alpha4
>Reporter: Tao Yang
>Assignee: Tao Yang
> Attachments: YARN-7508.001.patch
>
>
> YARN-6678 have fixed the IllegalStateException problem but the debug log it
> added may cause NPE when trying to print containerId of non-existed reserved
> container on this node. Replace
> {{schedulerContainer.getSchedulerNode().getReservedContainer().getContainerId()}}
> with {{schedulerContainer.getSchedulerNode().getReservedContainer()}} can
> fix this problem.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7482) Max applications calculation per queue has to be retrospected with absolute resource support
[ https://issues.apache.org/jira/browse/YARN-7482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255229#comment-16255229 ] Sunil G commented on YARN-7482: --- Test case failure is not related to this patch. Its been tracked in YARN-7483 > Max applications calculation per queue has to be retrospected with absolute > resource support > > > Key: YARN-7482 > URL: https://issues.apache.org/jira/browse/YARN-7482 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: YARN-5881 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7482-YARN-5881.001.patch > > > Retrospect max system applications w.r.t absolute resource configuration in a > queue -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7483) Test cases cleanup post YARN-5881
[ https://issues.apache.org/jira/browse/YARN-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7483: -- Description: Few UT cases has to be cleaned or rewritten cleanly to consider both absolute/percentage based configuration. Related test classes # TestLeafQueue # TestCapacityScheduler # TestCapacitySchedulerWithMultiResourceTypes was: Few UT cases has to be cleaned or rewritten cleanly to consider both absolute/percentage based configuration. Related test classes # TestLeafQueue # TestCapacityScheduler > Test cases cleanup post YARN-5881 > - > > Key: YARN-7483 > URL: https://issues.apache.org/jira/browse/YARN-7483 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test >Affects Versions: YARN-5881 >Reporter: Sunil G >Assignee: Sunil G > > Few UT cases has to be cleaned or rewritten cleanly to consider both > absolute/percentage based configuration. > Related test classes > # TestLeafQueue > # TestCapacityScheduler > # TestCapacitySchedulerWithMultiResourceTypes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7510) Merge work for YARN-5881
[
https://issues.apache.org/jira/browse/YARN-7510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255328#comment-16255328
]
Hadoop QA commented on YARN-7510:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 17 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 36s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
12s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
7s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 11m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 1s{color} | {color:orange} root: The patch generated 82 new + 2031
unchanged - 37 fixed = 2113 total (was 2068) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
8m 32s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
47s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
37s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 32s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 11s{color} |
{color:bla
[jira] [Commented] (YARN-7469) Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit
[ https://issues.apache.org/jira/browse/YARN-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255386#comment-16255386 ] Sunil G commented on YARN-7469: --- +1 Committing shortly. > Capacity Scheduler Intra-queue preemption: User can starve if newest app is > exactly at user limit > - > > Key: YARN-7469 > URL: https://issues.apache.org/jira/browse/YARN-7469 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, yarn >Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2 >Reporter: Eric Payne >Assignee: Eric Payne > Attachments: UnitTestToShowStarvedUser.patch, YARN-7469.001.patch > > > Queue Configuration: > - Total Memory: 20GB > - 2 Queues > -- Queue1 > --- Memory: 10GB > --- MULP: 10% > --- ULF: 2.0 > - Minimum Container Size: 0.5GB > Use Case: > - User1 submits app1 to Queue1 and consumes 20GB > - User2 submits app2 to Queue1 and requests 7.5GB > - Preemption monitor preempts 7.5GB from app1. Capacity Scheduler gives those > resources to User2 > - User 3 submits app3 to Queue1. To begin with, app3 is requesting 1 > container for the AM. > - Preemption monitor never preempts a container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7497) Add HDFSSchedulerConfigurationStore for RM HA
[
https://issues.apache.org/jira/browse/YARN-7497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255404#comment-16255404
]
Hadoop QA commented on YARN-7497:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 9s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
13s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
3s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 55s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 16 new + 290 unchanged - 0 fixed = 306 total (was 290) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
39s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 35s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}126m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands |
| |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| | org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce
[jira] [Commented] (YARN-7213) [Umbrella] Test and validate HBase-2.0.x with Atsv2
[ https://issues.apache.org/jira/browse/YARN-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255427#comment-16255427 ] Haibo Chen commented on YARN-7213: -- [~rohithsharma] One of the test failures, testGetEntitiesConfigFilter does not have any Tag involved. This has been some changes made in this patch, e.g. Tag -> ArrayBasedTag, I think as long as Tag semantic does not change, we should be OK. Hopefully, we can quickly resolve the test failures and confirm that. bq. I believe this patch is attempting to change directly hbase version to 2.* rather than doing conditional compilations. Right. I wanted to surface the HBase regression first, and then add the conditional compilation. Hence, the name wip patch. I don't have any experience with conditional compilation. Do you have any pointer in hadoop, hbase or the like for me to learn from? The many dependencies added are indeed required by hbase-2 version. I plan to add comments in the official patch. > [Umbrella] Test and validate HBase-2.0.x with Atsv2 > --- > > Key: YARN-7213 > URL: https://issues.apache.org/jira/browse/YARN-7213 > Project: Hadoop YARN > Issue Type: Task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-7213.prelim.patch, YARN-7213.wip.patch > > > Hbase-2.0.x officially support hadoop-alpha compilations. And also they are > getting ready for Hadoop-beta release so that HBase can release their > versions compatible with Hadoop-beta. So, this JIRA is to keep track of > HBase-2.0 integration issues. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7213) [Umbrella] Test and validate HBase-2.0.x with Atsv2
[ https://issues.apache.org/jira/browse/YARN-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255448#comment-16255448 ] Haibo Chen commented on YARN-7213: -- [~openinx] The problem seems to be around SingleColumnValueFilter.setFilterIfMissing(true). > [Umbrella] Test and validate HBase-2.0.x with Atsv2 > --- > > Key: YARN-7213 > URL: https://issues.apache.org/jira/browse/YARN-7213 > Project: Hadoop YARN > Issue Type: Task >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-7213.prelim.patch, YARN-7213.wip.patch > > > Hbase-2.0.x officially support hadoop-alpha compilations. And also they are > getting ready for Hadoop-beta release so that HBase can release their > versions compatible with Hadoop-beta. So, this JIRA is to keep track of > HBase-2.0 integration issues. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7512) Support service upgrade via YARN Service API and CLI
Gour Saha created YARN-7512: --- Summary: Support service upgrade via YARN Service API and CLI Key: YARN-7512 URL: https://issues.apache.org/jira/browse/YARN-7512 Project: Hadoop YARN Issue Type: Sub-task Reporter: Gour Saha YARN Service API and CLI needs to support service (and containers) upgrade in line with what Slider supported in SLIDER-787 and http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7512) Support service upgrade via YARN Service API and CLI
[ https://issues.apache.org/jira/browse/YARN-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gour Saha updated YARN-7512: Description: YARN Service API and CLI needs to support service (and containers) upgrade in line with what Slider supported in SLIDER-787 (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html) (was: YARN Service API and CLI needs to support service (and containers) upgrade in line with what Slider supported in SLIDER-787 and http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html) > Support service upgrade via YARN Service API and CLI > > > Key: YARN-7512 > URL: https://issues.apache.org/jira/browse/YARN-7512 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Gour Saha > Fix For: yarn-native-services > > > YARN Service API and CLI needs to support service (and containers) upgrade in > line with what Slider supported in SLIDER-787 > (http://slider.incubator.apache.org/docs/slider_specs/application_pkg_upgrade.html) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7486) Race condition in service AM that can cause NPE
[
https://issues.apache.org/jira/browse/YARN-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255545#comment-16255545
]
Hudson commented on YARN-7486:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13245 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13245/])
YARN-7486. Race condition in service AM that can cause NPE. Contributed
(billie: rev f4d5d20286eb05449f6fd7cd6ff0554228205fe2)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/provider/ProviderUtils.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/timelineservice/ServiceTimelinePublisher.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/containerlaunch/ContainerLaunchService.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/ServiceTestUtils.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestServiceAM.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/Component.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/monitor/TestServiceMonitor.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/ComponentEvent.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/instance/ComponentInstance.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/MockServiceAM.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/provider/ProviderService.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ServiceScheduler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/provider/AbstractProviderService.java
> Race condition in service AM that can cause NPE
> ---
>
> Key: YARN-7486
> URL: https://issues.apache.org/jira/browse/YARN-7486
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jian He
>Assignee: Jian He
> Fix For: 3.1.0
>
> Attachments: YARN-7486.01.patch, YARN-7486.02.patch
>
>
> 1. container1 completed for instance1
> 2. instance1 is added to pending list, and send an event asynchronously to
> instance1 to run ContainerStoppedTransition
> 3. container2 allocated, and assigned to instance1, it records the container2
> inside instance1
> 4. in the meantime, instance1 ContainerStoppedTransition is called and that
> set the container back to null.
> This cause the recorded container lost.
> {code}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.service.provider.ProviderUtils.initCompTokensForSubstitute(ProviderUtils.java:402)
> at
> org.apache.hadoop.yarn.service.provider.AbstractProviderService.buildContainerLaunchContext(AbstractProviderService.java:70)
> at
> org.apache.hadoop.yarn.service.containerlaunch.ContainerLaunchService$ContainerLauncher.run(ContainerLaunchService.java:89)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5534) Allow whitelisted volume mounts
[ https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255583#comment-16255583 ] Eric Yang commented on YARN-5534: - [~ebadger] [~shaneku...@gmail.com] In YARN-7430, there was mentioned how do we handle arbitrary docker image from docker hub without consistent uid:gid with the cluster. That discussion is related to allow defining white listed volume. We can check the origin of the docker image, if it comes from private registry, image name that starts with hostname of private registry, then we allow white list volumes. If image is from public repository, then we disallow user defined mount. When image has been approved to move from dockerhub to private repository, then user can store data into HDFS. The approval process is the safe guard to make sure the uid:gid used by image is compatible with the cluster. Does this sound reasonable approach to protect against unknown images? > Allow whitelisted volume mounts > > > Key: YARN-5534 > URL: https://issues.apache.org/jira/browse/YARN-5534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: luhuichun >Assignee: Shane Kumpf > Attachments: YARN-5534.001.patch, YARN-5534.002.patch, > YARN-5534.003.patch > > > Introduction > Mounting files or directories from the host is one way of passing > configuration and other information into a docker container. > We could allow the user to set a list of mounts in the environment of > ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2). > These would be mounted read-only to the specified target locations. This has > been resolved in YARN-4595 > 2.Problem Definition > Bug mounting arbitrary volumes into a Docker container can be a security risk. > 3.Possible solutions > one approach to provide safe mounts is to allow the cluster administrator to > configure a set of parent directories as white list mounting directories. > Add a property named yarn.nodemanager.volume-mounts.white-list, when > container executor do mount checking, only the allowed directories or > sub-directories can be mounted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7483) Test cases cleanup post YARN-5881
[ https://issues.apache.org/jira/browse/YARN-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7483: -- Attachment: YARN-7483.001.patch Attaching patch to clean up test case issues. cc/ [~leftnoteasy] please help to review. > Test cases cleanup post YARN-5881 > - > > Key: YARN-7483 > URL: https://issues.apache.org/jira/browse/YARN-7483 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test >Affects Versions: YARN-5881 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7483.001.patch > > > Few UT cases has to be cleaned or rewritten cleanly to consider both > absolute/percentage based configuration. > Related test classes > # TestLeafQueue > # TestCapacityScheduler > # TestCapacitySchedulerWithMultiResourceTypes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7218) ApiServer REST API naming convention /ws/v1 is already used in Hadoop v2
[ https://issues.apache.org/jira/browse/YARN-7218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255598#comment-16255598 ] Eric Yang commented on YARN-7218: - [~jianhe] Yes and no, it is related, but it is not something that we introduced. We are currently using serializer that comes with resource manager. Hence, the problem was there. > ApiServer REST API naming convention /ws/v1 is already used in Hadoop v2 > > > Key: YARN-7218 > URL: https://issues.apache.org/jira/browse/YARN-7218 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, applications >Reporter: Eric Yang >Assignee: Eric Yang > Attachments: YARN-7218.001.patch, YARN-7218.002.patch, > YARN-7218.003.patch, YARN-7218.004.patch > > > In YARN-6626, there is a desire to have ability to run ApiServer REST API in > Resource Manager, this can eliminate the requirement to deploy another daemon > service for submitting docker applications. In YARN-5698, a new UI has been > implemented as a separate web application. There are some problems in the > arrangement that can cause conflicts of how Java session are being managed. > The root context of Resource Manager web application is /ws. This is hard > coded in startWebapp method in ResourceManager.java. This means all the > session management is applied to Web URL of /ws prefix. /ui2 is independent > of /ws context, therefore session management code doesn't apply to /ui2. > This could be a session management problem, if servlet based code is going to > be introduced into /ui2 web application. > ApiServer code base is designed as a separate web application. There is no > easy way to inject a separate web application into the same /ws context > because ResourceManager is already setup to bind to RMWebServices. Unless > ApiServer code is moved into RMWebServices, otherwise, they will not share > the same session management. > The alternate solution is to keep ApiServer prefix URL independent of /ws > context. However, this will be a departure from YARN web services naming > convention. This can be loaded as a separate web application in Resource > Manager jetty server. One possible proposal is /app/v1/services. This can > keep ApiServer code modular and independent from Resource Manager. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7483) Test cases cleanup post YARN-5881
[
https://issues.apache.org/jira/browse/YARN-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255599#comment-16255599
]
Hadoop QA commented on YARN-7483:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-7483 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7483 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12898012/YARN-7483.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/18524/console |
| Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Test cases cleanup post YARN-5881
> -
>
> Key: YARN-7483
> URL: https://issues.apache.org/jira/browse/YARN-7483
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Affects Versions: YARN-5881
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-7483.001.patch
>
>
> Few UT cases has to be cleaned or rewritten cleanly to consider both
> absolute/percentage based configuration.
> Related test classes
> # TestLeafQueue
> # TestCapacityScheduler
> # TestCapacitySchedulerWithMultiResourceTypes
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7483) Test cases cleanup post YARN-5881
[ https://issues.apache.org/jira/browse/YARN-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-7483: -- Attachment: YARN-7483-YARN-5881.001.patch renamed patch to apply to YARN-5881 branch > Test cases cleanup post YARN-5881 > - > > Key: YARN-7483 > URL: https://issues.apache.org/jira/browse/YARN-7483 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test >Affects Versions: YARN-5881 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7483-YARN-5881.001.patch, YARN-7483.001.patch > > > Few UT cases has to be cleaned or rewritten cleanly to consider both > absolute/percentage based configuration. > Related test classes > # TestLeafQueue > # TestCapacityScheduler > # TestCapacitySchedulerWithMultiResourceTypes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7483) Test cases cleanup post YARN-5881
[ https://issues.apache.org/jira/browse/YARN-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255610#comment-16255610 ] Wangda Tan commented on YARN-7483: -- Patch looks good, pending Jenkins. > Test cases cleanup post YARN-5881 > - > > Key: YARN-7483 > URL: https://issues.apache.org/jira/browse/YARN-7483 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test >Affects Versions: YARN-5881 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7483-YARN-5881.001.patch, YARN-7483.001.patch > > > Few UT cases has to be cleaned or rewritten cleanly to consider both > absolute/percentage based configuration. > Related test classes > # TestLeafQueue > # TestCapacityScheduler > # TestCapacitySchedulerWithMultiResourceTypes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7509) AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby
[ https://issues.apache.org/jira/browse/YARN-7509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt reassigned YARN-7509: -- Assignee: Bibin A Chundatt (was: Tao Yang) > AsyncScheduleThread and ResourceCommitterService are still running after RM > is transitioned to standby > -- > > Key: YARN-7509 > URL: https://issues.apache.org/jira/browse/YARN-7509 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha4, 2.9.1 >Reporter: Tao Yang >Assignee: Bibin A Chundatt > Attachments: YARN-7509.001.patch > > > After RM is transitioned to standby, AsyncScheduleThread and > ResourceCommitterService will receive interrupt signal. When thread is > sleeping, it will ignore the interrupt signal since InterruptedException is > catched inside and the interrupt signal is cleared. > For AsyncScheduleThread, InterruptedException was catched and ignored in > CapacityScheduler#schedule. > For ResourceCommitterService, InterruptedException was catched inside and > ignored in ResourceCommitterService#run. > We should let the interrupt signal out and make these threads exit. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7509) AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby
[ https://issues.apache.org/jira/browse/YARN-7509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt reassigned YARN-7509: -- Assignee: Tao Yang (was: Bibin A Chundatt) > AsyncScheduleThread and ResourceCommitterService are still running after RM > is transitioned to standby > -- > > Key: YARN-7509 > URL: https://issues.apache.org/jira/browse/YARN-7509 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha4, 2.9.1 >Reporter: Tao Yang >Assignee: Tao Yang > Attachments: YARN-7509.001.patch > > > After RM is transitioned to standby, AsyncScheduleThread and > ResourceCommitterService will receive interrupt signal. When thread is > sleeping, it will ignore the interrupt signal since InterruptedException is > catched inside and the interrupt signal is cleared. > For AsyncScheduleThread, InterruptedException was catched and ignored in > CapacityScheduler#schedule. > For ResourceCommitterService, InterruptedException was catched inside and > ignored in ResourceCommitterService#run. > We should let the interrupt signal out and make these threads exit. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5534) Allow whitelisted volume mounts
[ https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255612#comment-16255612 ] Eric Badger commented on YARN-5534: --- Arbitrary docker images will need to be handled separately than what we consider to be "trusted" images more than just in the whitelisted volumes regard. These containers shouldn't be bind-mounting anything IMO and should be running without any capabilities. Even at that point, I'm not sure I'm comfortable allowing untrusted images run containers on the node, since the container will be running as root. This, of course, is unless we can figure out how to leverage user namespace remapping from Docker. https://docs.docker.com/engine/security/userns-remap/ Bottom line, if we are going to allow support for arbitrary images, I think we should open up a separate JIRA and create a complete plan over there with how we can utilize the current state of docker support while also creating a secure environment for these images to run. > Allow whitelisted volume mounts > > > Key: YARN-5534 > URL: https://issues.apache.org/jira/browse/YARN-5534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: luhuichun >Assignee: Shane Kumpf > Attachments: YARN-5534.001.patch, YARN-5534.002.patch, > YARN-5534.003.patch > > > Introduction > Mounting files or directories from the host is one way of passing > configuration and other information into a docker container. > We could allow the user to set a list of mounts in the environment of > ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2). > These would be mounted read-only to the specified target locations. This has > been resolved in YARN-4595 > 2.Problem Definition > Bug mounting arbitrary volumes into a Docker container can be a security risk. > 3.Possible solutions > one approach to provide safe mounts is to allow the cluster administrator to > configure a set of parent directories as white list mounting directories. > Add a property named yarn.nodemanager.volume-mounts.white-list, when > container executor do mount checking, only the allowed directories or > sub-directories can be mounted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7469) Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit
[ https://issues.apache.org/jira/browse/YARN-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255631#comment-16255631 ] Hudson commented on YARN-7469: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13246 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13246/]) YARN-7469. Capacity Scheduler Intra-queue preemption: User can starve if (sunilg: rev 61ace174cdcbca9d22abce7aa0aa71148f37ad55) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/FifoIntraQueuePreemptionPlugin.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicyIntraQueueUserLimit.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicyMockFramework.java > Capacity Scheduler Intra-queue preemption: User can starve if newest app is > exactly at user limit > - > > Key: YARN-7469 > URL: https://issues.apache.org/jira/browse/YARN-7469 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, yarn >Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2 >Reporter: Eric Payne >Assignee: Eric Payne > Attachments: UnitTestToShowStarvedUser.patch, YARN-7469.001.patch > > > Queue Configuration: > - Total Memory: 20GB > - 2 Queues > -- Queue1 > --- Memory: 10GB > --- MULP: 10% > --- ULF: 2.0 > - Minimum Container Size: 0.5GB > Use Case: > - User1 submits app1 to Queue1 and consumes 20GB > - User2 submits app2 to Queue1 and requests 7.5GB > - Preemption monitor preempts 7.5GB from app1. Capacity Scheduler gives those > resources to User2 > - User 3 submits app3 to Queue1. To begin with, app3 is requesting 1 > container for the AM. > - Preemption monitor never preempts a container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7448) [API] Add SchedulingRequest to the AllocateRequest
[
https://issues.apache.org/jira/browse/YARN-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255653#comment-16255653
]
Panagiotis Garefalakis commented on YARN-7448:
--
[~asuresh] patch looks good to me.
Some minor comments:
* We need to expose an AllocateRequest new instance with schedulingRequest List
as parameter
* ResourceSizing is never set in the SchedulingRequest constructor
* We might need to implement equals and hashcode for the complex ResourceSizing
object
> [API] Add SchedulingRequest to the AllocateRequest
> --
>
> Key: YARN-7448
> URL: https://issues.apache.org/jira/browse/YARN-7448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-7448-YARN-6592.001.patch,
> YARN-7448-YARN-6592.002.patch
>
>
> YARN-6594 introduces the {{SchedulingRequest}}. This JIRA tracks the
> inclusion of the SchedulingRequest into the AllocateRequest.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7448) [API] Add SchedulingRequest to the AllocateRequest
[
https://issues.apache.org/jira/browse/YARN-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Panagiotis Garefalakis updated YARN-7448:
-
Attachment: YARN-7448-YARN-6592.003.patch
Addressing comments based on [~asuresh] patch
> [API] Add SchedulingRequest to the AllocateRequest
> --
>
> Key: YARN-7448
> URL: https://issues.apache.org/jira/browse/YARN-7448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-7448-YARN-6592.001.patch,
> YARN-7448-YARN-6592.002.patch, YARN-7448-YARN-6592.003.patch
>
>
> YARN-6594 introduces the {{SchedulingRequest}}. This JIRA tracks the
> inclusion of the SchedulingRequest into the AllocateRequest.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7513) FindBugs in FSAppAttempt.getWeight()
Wilfred Spiegelenburg created YARN-7513: --- Summary: FindBugs in FSAppAttempt.getWeight() Key: YARN-7513 URL: https://issues.apache.org/jira/browse/YARN-7513 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler Affects Versions: 3.1.0 Reporter: Wilfred Spiegelenburg Assignee: Wilfred Spiegelenburg Priority: Minor With the change from YARN-7414 a new FindBugs warning was introduced. The code that was moved from the FairScheduler to the FSAppAttempt can also be simplified by removing the unneeded locking. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6486) FairScheduler: Deprecate continuous scheduling
[ https://issues.apache.org/jira/browse/YARN-6486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wilfred Spiegelenburg updated YARN-6486: Attachment: YARN-6486.002.patch Updated patch: fix for javac warning about deprecated API use findbugs warning logged as YARN-7513 test failures are not related to the code logged as YARN-7507 > FairScheduler: Deprecate continuous scheduling > -- > > Key: YARN-6486 > URL: https://issues.apache.org/jira/browse/YARN-6486 > Project: Hadoop YARN > Issue Type: Task > Components: fairscheduler >Affects Versions: 2.9.0 >Reporter: Wilfred Spiegelenburg >Assignee: Wilfred Spiegelenburg > Attachments: YARN-6486.001.patch, YARN-6486.002.patch > > > Mark continuous scheduling as deprecated in 2.9 and remove the code in 3.0. > Removing continuous scheduling from the code will be logged as a separate jira -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5534) Allow whitelisted volume mounts
[
https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255681#comment-16255681
]
Shane Kumpf commented on YARN-5534:
---
{code}
We can check the origin of the docker image, if it comes from private registry,
image name that starts with hostname of private registry, then we allow white
list volumes.
{code}
IMO the hosted docker private repositories should be allowed. Checking that the
image isn't from docker.io would be problematic for that case. The docker hub
private repository solution gives users a private space to store images without
needing to manage the private registry infrastructure themselves. Using the
docker hub private repositories also gives the user vulnerability scanning "for
free", so it's appealing to new users where pull bandwidth isn't of major
concern. IMO, this is a pretty safe alternative to running your own private
registry.
As [~ebadger] mentioned, there are other items that need to change to support
these types of images beyond the whitelist; don't override the CMD/ENTRYPOINT,
don't bind mount the container log dir, usercache, appcache, don't override the
--user option, etc. I would prefer if we worked through those details
holistically on a separate JIRA and see if it's even necessary to support that
use case.
> Allow whitelisted volume mounts
>
>
> Key: YARN-5534
> URL: https://issues.apache.org/jira/browse/YARN-5534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: luhuichun
>Assignee: Shane Kumpf
> Attachments: YARN-5534.001.patch, YARN-5534.002.patch,
> YARN-5534.003.patch
>
>
> Introduction
> Mounting files or directories from the host is one way of passing
> configuration and other information into a docker container.
> We could allow the user to set a list of mounts in the environment of
> ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2).
> These would be mounted read-only to the specified target locations. This has
> been resolved in YARN-4595
> 2.Problem Definition
> Bug mounting arbitrary volumes into a Docker container can be a security risk.
> 3.Possible solutions
> one approach to provide safe mounts is to allow the cluster administrator to
> configure a set of parent directories as white list mounting directories.
> Add a property named yarn.nodemanager.volume-mounts.white-list, when
> container executor do mount checking, only the allowed directories or
> sub-directories can be mounted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5534) Allow whitelisted volume mounts
[
https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255681#comment-16255681
]
Shane Kumpf edited comment on YARN-5534 at 11/16/17 5:45 PM:
-
{quote}
We can check the origin of the docker image, if it comes from private registry,
image name that starts with hostname of private registry, then we allow white
list volumes.
{quote}
IMO the hosted docker private repositories should be allowed. Checking that the
image isn't from docker.io would be problematic for that case. The docker hub
private repository solution gives users a private space to store images without
needing to manage the private registry infrastructure themselves. Using the
docker hub private repositories also gives the user vulnerability scanning "for
free", so it's appealing to new users where pull bandwidth isn't of major
concern. IMO, this is a pretty safe alternative to running your own private
registry.
As [~ebadger] mentioned, there are other items that need to change to support
these types of images beyond the whitelist; don't override the CMD/ENTRYPOINT,
don't bind mount the container log dir, usercache, appcache, don't override the
--user option, etc. I would prefer if we worked through those details
holistically on a separate JIRA and see if it's even necessary to support that
use case.
was (Author: shaneku...@gmail.com):
{code}
We can check the origin of the docker image, if it comes from private registry,
image name that starts with hostname of private registry, then we allow white
list volumes.
{code}
IMO the hosted docker private repositories should be allowed. Checking that the
image isn't from docker.io would be problematic for that case. The docker hub
private repository solution gives users a private space to store images without
needing to manage the private registry infrastructure themselves. Using the
docker hub private repositories also gives the user vulnerability scanning "for
free", so it's appealing to new users where pull bandwidth isn't of major
concern. IMO, this is a pretty safe alternative to running your own private
registry.
As [~ebadger] mentioned, there are other items that need to change to support
these types of images beyond the whitelist; don't override the CMD/ENTRYPOINT,
don't bind mount the container log dir, usercache, appcache, don't override the
--user option, etc. I would prefer if we worked through those details
holistically on a separate JIRA and see if it's even necessary to support that
use case.
> Allow whitelisted volume mounts
>
>
> Key: YARN-5534
> URL: https://issues.apache.org/jira/browse/YARN-5534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: luhuichun
>Assignee: Shane Kumpf
> Attachments: YARN-5534.001.patch, YARN-5534.002.patch,
> YARN-5534.003.patch
>
>
> Introduction
> Mounting files or directories from the host is one way of passing
> configuration and other information into a docker container.
> We could allow the user to set a list of mounts in the environment of
> ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2).
> These would be mounted read-only to the specified target locations. This has
> been resolved in YARN-4595
> 2.Problem Definition
> Bug mounting arbitrary volumes into a Docker container can be a security risk.
> 3.Possible solutions
> one approach to provide safe mounts is to allow the cluster administrator to
> configure a set of parent directories as white list mounting directories.
> Add a property named yarn.nodemanager.volume-mounts.white-list, when
> container executor do mount checking, only the allowed directories or
> sub-directories can be mounted.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7505) RM REST endpoints generate malformed JSON
[
https://issues.apache.org/jira/browse/YARN-7505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255682#comment-16255682
]
Eric Yang commented on YARN-7505:
-
Hi [~templedf],
Part of Hadoop switched to com.fastxml version of jackson in Hadoop 3, but some
parts were left in the old version. I am not sure com.fastxml.* and
org.codehaus.jackson.jaxrs can mix well. Can the code apply jaxb from the new
library?
{code}
com.fasterxml.jackson.jaxrs
jackson-jaxrs-json-provider
{code}
> RM REST endpoints generate malformed JSON
> -
>
> Key: YARN-7505
> URL: https://issues.apache.org/jira/browse/YARN-7505
> Project: Hadoop YARN
> Issue Type: Bug
> Components: restapi
>Affects Versions: 3.0.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-7505.001.patch, YARN-7505.002.patch
>
>
> For all endpoints that return DAOs that contain maps, the generated JSON is
> malformed. For example:
> % curl 'http://localhost:8088/ws/v1/cluster/apps'
> {"apps":{"app":[{"id":"application_1510777276702_0001","user":"daniel","name":"QuasiMonteCarlo","queue":"root.daniel","state":"RUNNING","finalStatus":"UNDEFINED","progress":5.0,"trackingUI":"ApplicationMaster","trackingUrl":"http://dhcp-10-16-0-181.pa.cloudera.com:8088/proxy/application_1510777276702_0001/","diagnostics":"","clusterId":1510777276702,"applicationType":"MAPREDUCE","applicationTags":"","priority":0,"startedTime":1510777317853,"finishedTime":0,"elapsedTime":21623,"amContainerLogs":"http://dhcp-10-16-0-181.pa.cloudera.com:8042/node/containerlogs/container_1510777276702_0001_01_01/daniel","amHostHttpAddress":"dhcp-10-16-0-181.pa.cloudera.com:8042","amRPCAddress":"dhcp-10-16-0-181.pa.cloudera.com:63371","allocatedMB":5120,"allocatedVCores":4,"reservedMB":0,"reservedVCores":0,"runningContainers":4,"memorySeconds":49820,"vcoreSeconds":26,"queueUsagePercentage":62.5,"clusterUsagePercentage":62.5,"resourceSecondsMap":{"entry":{"key":"test2","value":"0"},"entry":{"key":"test","value":"0"},"entry":{"key":"memory-mb","value":"49820"},"entry":{"key":"vcores","value":"26"}},"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":0,"preemptedVcoreSeconds":0,"preemptedResourceSecondsMap":{},"resourceRequests":[{"priority":20,"resourceName":"dhcp-10-16-0-181.pa.cloudera.com","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"/default-rack","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"*","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false}],"logAggregationStatus":"DISABLED","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":{"timeout":[{"type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1}]}}]}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7513) FindBugs in FSAppAttempt.getWeight()
[ https://issues.apache.org/jira/browse/YARN-7513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wilfred Spiegelenburg updated YARN-7513: Attachment: YARN-7513.001.patch patch to remove the locking and clean up the ported code > FindBugs in FSAppAttempt.getWeight() > > > Key: YARN-7513 > URL: https://issues.apache.org/jira/browse/YARN-7513 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Wilfred Spiegelenburg >Assignee: Wilfred Spiegelenburg >Priority: Minor > Attachments: YARN-7513.001.patch > > > With the change from YARN-7414 a new FindBugs warning was introduced. > The code that was moved from the FairScheduler to the FSAppAttempt can also > be simplified by removing the unneeded locking. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7448) [API] Add SchedulingRequest to the AllocateRequest
[
https://issues.apache.org/jira/browse/YARN-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255698#comment-16255698
]
Arun Suresh commented on YARN-7448:
---
[~pg1...@imperial.ac.uk], next time let me know before I submit a patch that
you'd like to work on it - Ill assign it to you then :)
bq. We need to expose an AllocateRequest new instance with schedulingRequest
List as parameter
That was intentional - In general, we would like to start using builders more,
rather than add more newInstance methods.
bq. ResourceSizing is never set in the SchedulingRequest constructor
You mean the SchedulingRequestBuilder constructor - True, but neither are a
bunch of other things like PlacementConstraints, Allocation tags etc. But,
given that it is probably a required field - might not be a bad idea to add it.
bq. We might need to implement equals and hashcode for the complex
ResourceSizing object
No harm in adding - but when testing the Equality of SchedulingRequest objects,
the internal PBImpl object already implements an equals and hashcode which
takes care of its inner fields.
Will assign this to you - can you put in another patch with the extra
newInstance method ?
> [API] Add SchedulingRequest to the AllocateRequest
> --
>
> Key: YARN-7448
> URL: https://issues.apache.org/jira/browse/YARN-7448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-7448-YARN-6592.001.patch,
> YARN-7448-YARN-6592.002.patch, YARN-7448-YARN-6592.003.patch
>
>
> YARN-6594 introduces the {{SchedulingRequest}}. This JIRA tracks the
> inclusion of the SchedulingRequest into the AllocateRequest.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7448) [API] Add SchedulingRequest to the AllocateRequest
[
https://issues.apache.org/jira/browse/YARN-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh reassigned YARN-7448:
-
Assignee: Panagiotis Garefalakis (was: Arun Suresh)
> [API] Add SchedulingRequest to the AllocateRequest
> --
>
> Key: YARN-7448
> URL: https://issues.apache.org/jira/browse/YARN-7448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Panagiotis Garefalakis
> Attachments: YARN-7448-YARN-6592.001.patch,
> YARN-7448-YARN-6592.002.patch, YARN-7448-YARN-6592.003.patch
>
>
> YARN-6594 introduces the {{SchedulingRequest}}. This JIRA tracks the
> inclusion of the SchedulingRequest into the AllocateRequest.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7514) TestAggregatedLogDeletionService.testRefreshLogRetentionSettings is flaky
Haibo Chen created YARN-7514: Summary: TestAggregatedLogDeletionService.testRefreshLogRetentionSettings is flaky Key: YARN-7514 URL: https://issues.apache.org/jira/browse/YARN-7514 Project: Hadoop YARN Issue Type: Bug Components: log-aggregation Affects Versions: 3.0.0-beta1 Reporter: Haibo Chen TestAggregatedLogDeletionService.testRefreshLogRetentionSettings fails occasionally with Error Message Argument(s) are different! Wanted: fileSystem.delete( mockfs://foo/tmp/logs/me/logs/application_1510201418065_0002, true ); -> at org.apache.hadoop.yarn.logaggregation.TestAggregatedLogDeletionService.testRefreshLogRetentionSettings(TestAggregatedLogDeletionService.java:300) Actual invocation has different arguments: fileSystem.delete( mockfs://foo/tmp/logs/me/logs/application_1510201418024_0001, true ); -> at org.apache.hadoop.fs.FilterFileSystem.delete(FilterFileSystem.java:252) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7514) TestAggregatedLogDeletionService.testRefreshLogRetentionSettings is flaky
[ https://issues.apache.org/jira/browse/YARN-7514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-7514: - Description: TestAggregatedLogDeletionService.testRefreshLogRetentionSettings fails occasionally with *Error Message* Argument(s) are different! Wanted: fileSystem.delete( mockfs://foo/tmp/logs/me/logs/application_1510201418065_0002, true ); -> at org.apache.hadoop.yarn.logaggregation.TestAggregatedLogDeletionService.testRefreshLogRetentionSettings(TestAggregatedLogDeletionService.java:300) Actual invocation has different arguments: fileSystem.delete( mockfs://foo/tmp/logs/me/logs/application_1510201418024_0001, true ); -> at org.apache.hadoop.fs.FilterFileSystem.delete(FilterFileSystem.java:252) *Stacktrace* org.mockito.exceptions.verification.junit.ArgumentsAreDifferent: Argument(s) are different! Wanted: fileSystem.delete( mockfs://foo/tmp/logs/me/logs/application_1510201418065_0002, true ); -> at org.apache.hadoop.yarn.logaggregation.TestAggregatedLogDeletionService.testRefreshLogRetentionSettings(TestAggregatedLogDeletionService.java:300) Actual invocation has different arguments: fileSystem.delete( mockfs://foo/tmp/logs/me/logs/application_1510201418024_0001, true ); -> at org.apache.hadoop.fs.FilterFileSystem.delete(FilterFileSystem.java:252) at org.apache.hadoop.yarn.logaggregation.TestAggregatedLogDeletionService.testRefreshLogRetentionSettings(TestAggregatedLogDeletionService.java:300) *Standard Output* 2017-11-08 20:23:38,138 INFO [Timer-0] logaggregation.AggregatedLogDeletionService (AggregatedLogDeletionService.java:run(79)) - aggregated log deletion started. 2017-11-08 20:23:38,146 INFO [Timer-0] logaggregation.AggregatedLogDeletionService (AggregatedLogDeletionService.java:deleteOldLogDirsFrom(106)) - Deleting aggregated logs in mockfs://foo/tmp/logs/me/logs/application_1510201418024_0001 2017-11-08 20:23:38,146 INFO [Timer-0] logaggregation.AggregatedLogDeletionService (AggregatedLogDeletionService.java:run(92)) - aggregated log deletion finished. 2017-11-08 20:23:38,167 INFO [Timer-1] logaggregation.AggregatedLogDeletionService (AggregatedLogDeletionService.java:run(79)) - aggregated log deletion started. 2017-11-08 20:23:38,172 INFO [Timer-1] logaggregation.AggregatedLogDeletionService (AggregatedLogDeletionService.java:deleteOldLogDirsFrom(106)) - Deleting aggregated logs in mockfs://foo/tmp/logs/me/logs/application_1510201418024_0001 2017-11-08 20:23:38,173 INFO [Timer-1] logaggregation.AggregatedLogDeletionService (AggregatedLogDeletionService.java:deleteOldLogDirsFrom(106)) - Deleting aggregated logs in mockfs://foo/tmp/logs/me/logs/application_1510201418065_0002 2017-11-08 20:23:38,181 INFO [Timer-1] logaggregation.AggregatedLogDeletionService (AggregatedLogDeletionService.java:run(92)) - aggregated log deletion finished. was: TestAggregatedLogDeletionService.testRefreshLogRetentionSettings fails occasionally with Error Message Argument(s) are different! Wanted: fileSystem.delete( mockfs://foo/tmp/logs/me/logs/application_1510201418065_0002, true ); -> at org.apache.hadoop.yarn.logaggregation.TestAggregatedLogDeletionService.testRefreshLogRetentionSettings(TestAggregatedLogDeletionService.java:300) Actual invocation has different arguments: fileSystem.delete( mockfs://foo/tmp/logs/me/logs/application_1510201418024_0001, true ); -> at org.apache.hadoop.fs.FilterFileSystem.delete(FilterFileSystem.java:252) > TestAggregatedLogDeletionService.testRefreshLogRetentionSettings is flaky > - > > Key: YARN-7514 > URL: https://issues.apache.org/jira/browse/YARN-7514 > Project: Hadoop YARN > Issue Type: Bug > Components: log-aggregation >Affects Versions: 3.0.0-beta1 >Reporter: Haibo Chen > > TestAggregatedLogDeletionService.testRefreshLogRetentionSettings fails > occasionally with > *Error Message* > Argument(s) are different! Wanted: > fileSystem.delete( > mockfs://foo/tmp/logs/me/logs/application_1510201418065_0002, > true > ); > -> at > org.apache.hadoop.yarn.logaggregation.TestAggregatedLogDeletionService.testRefreshLogRetentionSettings(TestAggregatedLogDeletionService.java:300) > Actual invocation has different arguments: > fileSystem.delete( > mockfs://foo/tmp/logs/me/logs/application_1510201418024_0001, > true > ); > -> at org.apache.hadoop.fs.FilterFileSystem.delete(FilterFileSystem.java:252) > *Stacktrace* > org.mockito.exceptions.verification.junit.ArgumentsAreDifferent: > Argument(s) are different! Wanted: > fileSystem.delete( > mockfs://foo/tmp/logs/me/logs/application_1510201418065_0002, > true > ); > -> at > org.apache.hadoop.yarn.logaggregation.TestAggregatedLogDeletionService.testRefreshLogRetentionSettings(Tes
[jira] [Commented] (YARN-7513) FindBugs in FSAppAttempt.getWeight()
[ https://issues.apache.org/jira/browse/YARN-7513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255727#comment-16255727 ] Daniel Templeton commented on YARN-7513: Are you sure it's safe to remove the lock? > FindBugs in FSAppAttempt.getWeight() > > > Key: YARN-7513 > URL: https://issues.apache.org/jira/browse/YARN-7513 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Wilfred Spiegelenburg >Assignee: Wilfred Spiegelenburg >Priority: Minor > Attachments: YARN-7513.001.patch > > > With the change from YARN-7414 a new FindBugs warning was introduced. > The code that was moved from the FairScheduler to the FSAppAttempt can also > be simplified by removing the unneeded locking. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7505) RM REST endpoints generate malformed JSON
[
https://issues.apache.org/jira/browse/YARN-7505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255742#comment-16255742
]
Daniel Templeton commented on YARN-7505:
Good point. The two don't play together well, but it should be easy to swap
out codehaus for fasterxml.
I've been digging into what it would take to put all of this under a v2, and
it's not as bad as I thought, but it's still pretty ugly. At this point I
still haven't figured out how to apply the change to only v2 and not v1, but
I'm not done poking at it yet.
> RM REST endpoints generate malformed JSON
> -
>
> Key: YARN-7505
> URL: https://issues.apache.org/jira/browse/YARN-7505
> Project: Hadoop YARN
> Issue Type: Bug
> Components: restapi
>Affects Versions: 3.0.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-7505.001.patch, YARN-7505.002.patch
>
>
> For all endpoints that return DAOs that contain maps, the generated JSON is
> malformed. For example:
> % curl 'http://localhost:8088/ws/v1/cluster/apps'
> {"apps":{"app":[{"id":"application_1510777276702_0001","user":"daniel","name":"QuasiMonteCarlo","queue":"root.daniel","state":"RUNNING","finalStatus":"UNDEFINED","progress":5.0,"trackingUI":"ApplicationMaster","trackingUrl":"http://dhcp-10-16-0-181.pa.cloudera.com:8088/proxy/application_1510777276702_0001/","diagnostics":"","clusterId":1510777276702,"applicationType":"MAPREDUCE","applicationTags":"","priority":0,"startedTime":1510777317853,"finishedTime":0,"elapsedTime":21623,"amContainerLogs":"http://dhcp-10-16-0-181.pa.cloudera.com:8042/node/containerlogs/container_1510777276702_0001_01_01/daniel","amHostHttpAddress":"dhcp-10-16-0-181.pa.cloudera.com:8042","amRPCAddress":"dhcp-10-16-0-181.pa.cloudera.com:63371","allocatedMB":5120,"allocatedVCores":4,"reservedMB":0,"reservedVCores":0,"runningContainers":4,"memorySeconds":49820,"vcoreSeconds":26,"queueUsagePercentage":62.5,"clusterUsagePercentage":62.5,"resourceSecondsMap":{"entry":{"key":"test2","value":"0"},"entry":{"key":"test","value":"0"},"entry":{"key":"memory-mb","value":"49820"},"entry":{"key":"vcores","value":"26"}},"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":0,"preemptedVcoreSeconds":0,"preemptedResourceSecondsMap":{},"resourceRequests":[{"priority":20,"resourceName":"dhcp-10-16-0-181.pa.cloudera.com","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"/default-rack","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"*","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false}],"logAggregationStatus":"DISABLED","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":{"timeout":[{"type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1}]}}]}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7448) [API] Add SchedulingRequest to the AllocateRequest
[
https://issues.apache.org/jira/browse/YARN-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255770#comment-16255770
]
Hadoop QA commented on YARN-7448:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-6592 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
31s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
52s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 38s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
8s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
YARN-6592 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} YARN-6592 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 19s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
37s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
58s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 66m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7448 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12898022/YARN-7448-YARN-6592.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 6b5301c0aded 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-6592 / 24494ae |
| maven | version:
[jira] [Updated] (YARN-7330) Add support to show GPU on UI/metrics
[
https://issues.apache.org/jira/browse/YARN-7330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-7330:
-
Attachment: YARN-7330.011.patch
Attached ver.11 patch. Reverted changes of simplyUnit. As discussed with
[~sunilg] offline, the usage/logic is expected.
I found some of the previous changes to using Constants have issues which broke
all pages.
[~sunilg]/[~skmvasu], could you help to check what's the proper way in JS to
use constants? I tried but couldn't figure it out. My node js version is:
{code}
$ node -v
v8.9.1
{code}
> Add support to show GPU on UI/metrics
> -
>
> Key: YARN-7330
> URL: https://issues.apache.org/jira/browse/YARN-7330
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: YARN-7330.0-wip.patch, YARN-7330.003.patch,
> YARN-7330.004.patch, YARN-7330.006.patch, YARN-7330.007.patch,
> YARN-7330.008.patch, YARN-7330.009.patch, YARN-7330.011.patch,
> YARN-7330.1-wip.patch, YARN-7330.2-wip.patch, screencapture-0-wip.png
>
>
> We should be able to view GPU metrics from UI/REST API.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7390) All reservation related test cases failed when TestYarnClient runs against Fair Scheduler.
[
https://issues.apache.org/jira/browse/YARN-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255783#comment-16255783
]
Yufei Gu commented on YARN-7390:
Thanks [~haibo.chen] for review and commit.
> All reservation related test cases failed when TestYarnClient runs against
> Fair Scheduler.
> --
>
> Key: YARN-7390
> URL: https://issues.apache.org/jira/browse/YARN-7390
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, reservation system
>Affects Versions: 2.9.0, 3.0.0, 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 3.0.1
>
> Attachments: YARN-7390.001.patch, YARN-7390.002.patch,
> YARN-7390.003.patch, YARN-7390.004.patch, YARN-7390.005.patch
>
>
> All reservation related test cases failed when {{TestYarnClient}} runs
> against Fair Scheduler. To reproduce it, you need to set scheduler class to
> Fair Scheduler in yarn-default.xml.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7505) RM REST endpoints generate malformed JSON
[
https://issues.apache.org/jira/browse/YARN-7505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255792#comment-16255792
]
Wangda Tan commented on YARN-7505:
--
[~templedf]/[~sunilg]
I'm fine with changing any fields which are not documented by
https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html#REST_APIs
Otherwise we cannot break compatibility even if it is malformed (according to
this SO:
https://stackoverflow.com/questions/21832701/does-json-syntax-allow-duplicate-keys-in-an-object),
duplicated key of JSON fields are discouraged but allowed, implementation of
JSON parsers may not support this though.
> RM REST endpoints generate malformed JSON
> -
>
> Key: YARN-7505
> URL: https://issues.apache.org/jira/browse/YARN-7505
> Project: Hadoop YARN
> Issue Type: Bug
> Components: restapi
>Affects Versions: 3.0.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-7505.001.patch, YARN-7505.002.patch
>
>
> For all endpoints that return DAOs that contain maps, the generated JSON is
> malformed. For example:
> % curl 'http://localhost:8088/ws/v1/cluster/apps'
> {"apps":{"app":[{"id":"application_1510777276702_0001","user":"daniel","name":"QuasiMonteCarlo","queue":"root.daniel","state":"RUNNING","finalStatus":"UNDEFINED","progress":5.0,"trackingUI":"ApplicationMaster","trackingUrl":"http://dhcp-10-16-0-181.pa.cloudera.com:8088/proxy/application_1510777276702_0001/","diagnostics":"","clusterId":1510777276702,"applicationType":"MAPREDUCE","applicationTags":"","priority":0,"startedTime":1510777317853,"finishedTime":0,"elapsedTime":21623,"amContainerLogs":"http://dhcp-10-16-0-181.pa.cloudera.com:8042/node/containerlogs/container_1510777276702_0001_01_01/daniel","amHostHttpAddress":"dhcp-10-16-0-181.pa.cloudera.com:8042","amRPCAddress":"dhcp-10-16-0-181.pa.cloudera.com:63371","allocatedMB":5120,"allocatedVCores":4,"reservedMB":0,"reservedVCores":0,"runningContainers":4,"memorySeconds":49820,"vcoreSeconds":26,"queueUsagePercentage":62.5,"clusterUsagePercentage":62.5,"resourceSecondsMap":{"entry":{"key":"test2","value":"0"},"entry":{"key":"test","value":"0"},"entry":{"key":"memory-mb","value":"49820"},"entry":{"key":"vcores","value":"26"}},"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":0,"preemptedVcoreSeconds":0,"preemptedResourceSecondsMap":{},"resourceRequests":[{"priority":20,"resourceName":"dhcp-10-16-0-181.pa.cloudera.com","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"/default-rack","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"*","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false}],"logAggregationStatus":"DISABLED","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":{"timeout":[{"type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1}]}}]}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7515) HA related tests fails in trunk
Bibin A Chundatt created YARN-7515:
--
Summary: HA related tests fails in trunk
Key: YARN-7515
URL: https://issues.apache.org/jira/browse/YARN-7515
Project: Hadoop YARN
Issue Type: Bug
Reporter: Bibin A Chundatt
{noformat}
org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands
org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7505) RM REST endpoints generate malformed JSON
[
https://issues.apache.org/jira/browse/YARN-7505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255797#comment-16255797
]
Wangda Tan commented on YARN-7505:
--
[~templedf], and also, it looks like the new compatbility guide changed REST
API compatibility def:
https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/Compatibility.html#REST_APIs.
I'm not sure if we should consider all undocumented REST APIs as stable/public.
> RM REST endpoints generate malformed JSON
> -
>
> Key: YARN-7505
> URL: https://issues.apache.org/jira/browse/YARN-7505
> Project: Hadoop YARN
> Issue Type: Bug
> Components: restapi
>Affects Versions: 3.0.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-7505.001.patch, YARN-7505.002.patch
>
>
> For all endpoints that return DAOs that contain maps, the generated JSON is
> malformed. For example:
> % curl 'http://localhost:8088/ws/v1/cluster/apps'
> {"apps":{"app":[{"id":"application_1510777276702_0001","user":"daniel","name":"QuasiMonteCarlo","queue":"root.daniel","state":"RUNNING","finalStatus":"UNDEFINED","progress":5.0,"trackingUI":"ApplicationMaster","trackingUrl":"http://dhcp-10-16-0-181.pa.cloudera.com:8088/proxy/application_1510777276702_0001/","diagnostics":"","clusterId":1510777276702,"applicationType":"MAPREDUCE","applicationTags":"","priority":0,"startedTime":1510777317853,"finishedTime":0,"elapsedTime":21623,"amContainerLogs":"http://dhcp-10-16-0-181.pa.cloudera.com:8042/node/containerlogs/container_1510777276702_0001_01_01/daniel","amHostHttpAddress":"dhcp-10-16-0-181.pa.cloudera.com:8042","amRPCAddress":"dhcp-10-16-0-181.pa.cloudera.com:63371","allocatedMB":5120,"allocatedVCores":4,"reservedMB":0,"reservedVCores":0,"runningContainers":4,"memorySeconds":49820,"vcoreSeconds":26,"queueUsagePercentage":62.5,"clusterUsagePercentage":62.5,"resourceSecondsMap":{"entry":{"key":"test2","value":"0"},"entry":{"key":"test","value":"0"},"entry":{"key":"memory-mb","value":"49820"},"entry":{"key":"vcores","value":"26"}},"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":0,"preemptedVcoreSeconds":0,"preemptedResourceSecondsMap":{},"resourceRequests":[{"priority":20,"resourceName":"dhcp-10-16-0-181.pa.cloudera.com","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"/default-rack","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"*","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false}],"logAggregationStatus":"DISABLED","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":{"timeout":[{"type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1}]}}]}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7515) HA related tests fails in trunk
[
https://issues.apache.org/jira/browse/YARN-7515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-7515:
---
Description:
https://builds.apache.org/job/PreCommit-YARN-Build/18498/testReport/
{noformat}
org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands
org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
{noformat}
was:
{noformat}
org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands
org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
{noformat}
> HA related tests fails in trunk
> ---
>
> Key: YARN-7515
> URL: https://issues.apache.org/jira/browse/YARN-7515
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>
> https://builds.apache.org/job/PreCommit-YARN-Build/18498/testReport/
> {noformat}
> org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands
>
> org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA
>
> org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
>
> org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7516) Security check for untrusted docker image
Eric Yang created YARN-7516: --- Summary: Security check for untrusted docker image Key: YARN-7516 URL: https://issues.apache.org/jira/browse/YARN-7516 Project: Hadoop YARN Issue Type: Improvement Reporter: Eric Yang Hadoop YARN Services can support using private docker registry image or docker image from docker hub. In current implementation, Hadoop security is enforced through username and group membership, and enforce uid:gid consistency in docker container and distributed file system. There is cloud use case for having ability to run untrusted docker image on the same cluster for testing. The basic requirement for untrusted container is to ensure all kernel and root privileges are dropped, and there is no interaction with distributed file system to avoid contamination. We can probably enforce detection of untrusted docker image by checking the following: # If docker image is from public docker hub repository, the container is automatically flagged as insecure, and disk volume mount are disabled automatically, and drop all kernel capabilities. # If docker image is from private repository in docker hub, and there is a white list to allow the private repository, disk volume mount is allowed, kernel capabilities follows the allowed list. # If docker image is from private trusted repository with image name like "private.registry.local:5000/centos", and white list allows this private trusted repository. Disk volume mount is allowed, kernel capabilities follows the allowed list. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7516) Security check for untrusted docker image
[ https://issues.apache.org/jira/browse/YARN-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-7516: Description: Hadoop YARN Services can support using private docker registry image or docker image from docker hub. In current implementation, Hadoop security is enforced through username and group membership, and enforce uid:gid consistency in docker container and distributed file system. There is cloud use case for having ability to run untrusted docker image on the same cluster for testing. The basic requirement for untrusted container is to ensure all kernel and root privileges are dropped, and there is no interaction with distributed file system to avoid contamination. We can probably enforce detection of untrusted docker image by checking the following: # If docker image is from public docker hub repository, the container is automatically flagged as insecure, and disk volume mount are disabled automatically, and drop all kernel capabilities. # If docker image is from private repository in docker hub, and there is a white list to allow the private repository, disk volume mount is allowed, kernel capabilities follows the allowed list. # If docker image is from private trusted registry with image name like "private.registry.local:5000/centos", and white list allows this private trusted repository. Disk volume mount is allowed, kernel capabilities follows the allowed list. was: Hadoop YARN Services can support using private docker registry image or docker image from docker hub. In current implementation, Hadoop security is enforced through username and group membership, and enforce uid:gid consistency in docker container and distributed file system. There is cloud use case for having ability to run untrusted docker image on the same cluster for testing. The basic requirement for untrusted container is to ensure all kernel and root privileges are dropped, and there is no interaction with distributed file system to avoid contamination. We can probably enforce detection of untrusted docker image by checking the following: # If docker image is from public docker hub repository, the container is automatically flagged as insecure, and disk volume mount are disabled automatically, and drop all kernel capabilities. # If docker image is from private repository in docker hub, and there is a white list to allow the private repository, disk volume mount is allowed, kernel capabilities follows the allowed list. # If docker image is from private trusted repository with image name like "private.registry.local:5000/centos", and white list allows this private trusted repository. Disk volume mount is allowed, kernel capabilities follows the allowed list. > Security check for untrusted docker image > - > > Key: YARN-7516 > URL: https://issues.apache.org/jira/browse/YARN-7516 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Eric Yang > > Hadoop YARN Services can support using private docker registry image or > docker image from docker hub. In current implementation, Hadoop security is > enforced through username and group membership, and enforce uid:gid > consistency in docker container and distributed file system. There is cloud > use case for having ability to run untrusted docker image on the same cluster > for testing. > The basic requirement for untrusted container is to ensure all kernel and > root privileges are dropped, and there is no interaction with distributed > file system to avoid contamination. We can probably enforce detection of > untrusted docker image by checking the following: > # If docker image is from public docker hub repository, the container is > automatically flagged as insecure, and disk volume mount are disabled > automatically, and drop all kernel capabilities. > # If docker image is from private repository in docker hub, and there is a > white list to allow the private repository, disk volume mount is allowed, > kernel capabilities follows the allowed list. > # If docker image is from private trusted registry with image name like > "private.registry.local:5000/centos", and white list allows this private > trusted repository. Disk volume mount is allowed, kernel capabilities > follows the allowed list. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7390) All reservation related test cases failed when TestYarnClient runs against Fair Scheduler.
[
https://issues.apache.org/jira/browse/YARN-7390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255806#comment-16255806
]
Hudson commented on YARN-7390:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13247 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13247/])
YARN-7390. All reservation related test cases failed when TestYarnClient
(haibochen: rev 28d0fcbef40930ca5652c0e9a5d777910f3ad3c4)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java
* (add)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClientWithReservation.java
> All reservation related test cases failed when TestYarnClient runs against
> Fair Scheduler.
> --
>
> Key: YARN-7390
> URL: https://issues.apache.org/jira/browse/YARN-7390
> Project: Hadoop YARN
> Issue Type: Bug
> Components: fairscheduler, reservation system
>Affects Versions: 2.9.0, 3.0.0, 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 3.0.1
>
> Attachments: YARN-7390.001.patch, YARN-7390.002.patch,
> YARN-7390.003.patch, YARN-7390.004.patch, YARN-7390.005.patch
>
>
> All reservation related test cases failed when {{TestYarnClient}} runs
> against Fair Scheduler. To reproduce it, you need to set scheduler class to
> Fair Scheduler in yarn-default.xml.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7516) Security check for untrusted docker image
[ https://issues.apache.org/jira/browse/YARN-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255811#comment-16255811 ] Eric Yang commented on YARN-7516: - Other possible solution is to use docker user remapping feature to enforce user namespace consistency with Hadoop cluster. However, the current implementation of docker user remapping is a map of json of users to append to docker daemon. It does not look like a scalable solution for arbitrary username that might exist in docker hub. > Security check for untrusted docker image > - > > Key: YARN-7516 > URL: https://issues.apache.org/jira/browse/YARN-7516 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Eric Yang > > Hadoop YARN Services can support using private docker registry image or > docker image from docker hub. In current implementation, Hadoop security is > enforced through username and group membership, and enforce uid:gid > consistency in docker container and distributed file system. There is cloud > use case for having ability to run untrusted docker image on the same cluster > for testing. > The basic requirement for untrusted container is to ensure all kernel and > root privileges are dropped, and there is no interaction with distributed > file system to avoid contamination. We can probably enforce detection of > untrusted docker image by checking the following: > # If docker image is from public docker hub repository, the container is > automatically flagged as insecure, and disk volume mount are disabled > automatically, and drop all kernel capabilities. > # If docker image is from private repository in docker hub, and there is a > white list to allow the private repository, disk volume mount is allowed, > kernel capabilities follows the allowed list. > # If docker image is from private trusted registry with image name like > "private.registry.local:5000/centos", and white list allows this private > trusted repository. Disk volume mount is allowed, kernel capabilities > follows the allowed list. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7483) Test cases cleanup post YARN-5881
[
https://issues.apache.org/jira/browse/YARN-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255813#comment-16255813
]
Hadoop QA commented on YARN-7483:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-5881 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
57s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
13s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
31s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} YARN-5881 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} YARN-5881 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
57s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 50s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 16 new + 373 unchanged - 0 fixed = 389 total (was 373) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 51s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
36s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 60m 45s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}125m 58s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands |
| |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
| | org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7483 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12898015/YARN-7483-YARN-5881.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 61ca52d4a37f 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30
[jira] [Commented] (YARN-7506) Overhaul the design of the Linux container-executor regarding Docker and future runtimes
[ https://issues.apache.org/jira/browse/YARN-7506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255814#comment-16255814 ] Eric Badger commented on YARN-7506: --- I suppose we could move the docker portion of container-executor into a standalone root java process, but I'm not convinced that gives us a whole lot in terms of security. Moving the docker portion out of container-executor doesn't get rid of the container-executor. So if someone compromises yarn, they will still be able to wield the container-executor. I guess what moving docker out of the container-executor would give us is that we would have a smaller surface area to introduce bugs that would misuse executing system calls. But we weren't making those system calls in the docker portion of the container-executor anyway. We would also have a smaller C code surface area, which may be more secure since most Hadoop programmers are more comfortable in Java as opposed to C. The idea of moving the docker portion of container-executor into a java process is attractive because of the development aspect. However, that would be a pretty giant effort and would require a complete rewrite of most of the docker implementation. So I'm not sure that the effort would be worth it. > Overhaul the design of the Linux container-executor regarding Docker and > future runtimes > > > Key: YARN-7506 > URL: https://issues.apache.org/jira/browse/YARN-7506 > Project: Hadoop YARN > Issue Type: Wish > Components: nodemanager >Reporter: Miklos Szegedi > Labels: Docker, container-executor > > I raise this topic to discuss a potential improvement of the container > executor tool in node manager. > container-executor has two main purposes. It executes Linux *system calls not > available from Java*, and it executes tasks *available to root that are not > available to the yarn user*. Historically container-executor did both by > doing impersonation. The yarn user is separated from root because it runs > network services, so *the yarn user should be restricted* by design. Because > of this it has it's own config file container-executor.cfg writable by root > only that specifies what actions are allowed for the yarn user. However, the > requirements have changed with Docker and that raises the following questions: > 1. The Docker feature of YARN requires root permissions to *access the Docker > socket* but it does not run any system calls, so could the Docker related > code in container-executor be *refactored into a separate Java process ran as > root*? Java would make the development much faster and more secure. > 2. The Docker feature only needs the Docker unix socket. It is not a good > idea to let the yarn user directly access the socket, since that would > elevate its privileges to root. However, the Java tool running as root > mentioned in the previous question could act as a *proxy on the Docker > socket* operating directly on the Docker REST API *eliminating the need to > use the Docker CLI*. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7516) Security check for untrusted docker image
[ https://issues.apache.org/jira/browse/YARN-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated YARN-7516: -- Issue Type: Sub-task (was: Improvement) Parent: YARN-3611 > Security check for untrusted docker image > - > > Key: YARN-7516 > URL: https://issues.apache.org/jira/browse/YARN-7516 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang > > Hadoop YARN Services can support using private docker registry image or > docker image from docker hub. In current implementation, Hadoop security is > enforced through username and group membership, and enforce uid:gid > consistency in docker container and distributed file system. There is cloud > use case for having ability to run untrusted docker image on the same cluster > for testing. > The basic requirement for untrusted container is to ensure all kernel and > root privileges are dropped, and there is no interaction with distributed > file system to avoid contamination. We can probably enforce detection of > untrusted docker image by checking the following: > # If docker image is from public docker hub repository, the container is > automatically flagged as insecure, and disk volume mount are disabled > automatically, and drop all kernel capabilities. > # If docker image is from private repository in docker hub, and there is a > white list to allow the private repository, disk volume mount is allowed, > kernel capabilities follows the allowed list. > # If docker image is from private trusted registry with image name like > "private.registry.local:5000/centos", and white list allows this private > trusted repository. Disk volume mount is allowed, kernel capabilities > follows the allowed list. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7516) Security check for untrusted docker image
[ https://issues.apache.org/jira/browse/YARN-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255816#comment-16255816 ] Eric Badger commented on YARN-7516: --- Hey [~eyang], could you open up new docker jiras as subtasks of YARN-3611 instead of linking them? It makes it much easier to view the complete development of the docker code on YARN-3611. Thanks! > Security check for untrusted docker image > - > > Key: YARN-7516 > URL: https://issues.apache.org/jira/browse/YARN-7516 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang > > Hadoop YARN Services can support using private docker registry image or > docker image from docker hub. In current implementation, Hadoop security is > enforced through username and group membership, and enforce uid:gid > consistency in docker container and distributed file system. There is cloud > use case for having ability to run untrusted docker image on the same cluster > for testing. > The basic requirement for untrusted container is to ensure all kernel and > root privileges are dropped, and there is no interaction with distributed > file system to avoid contamination. We can probably enforce detection of > untrusted docker image by checking the following: > # If docker image is from public docker hub repository, the container is > automatically flagged as insecure, and disk volume mount are disabled > automatically, and drop all kernel capabilities. > # If docker image is from private repository in docker hub, and there is a > white list to allow the private repository, disk volume mount is allowed, > kernel capabilities follows the allowed list. > # If docker image is from private trusted registry with image name like > "private.registry.local:5000/centos", and white list allows this private > trusted repository. Disk volume mount is allowed, kernel capabilities > follows the allowed list. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7419) CapacityScheduler: Allow auto leaf queue creation after queue mapping
[ https://issues.apache.org/jira/browse/YARN-7419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7419: - Summary: CapacityScheduler: Allow auto leaf queue creation after queue mapping (was: Implement Auto Queue Creation with modifications to queue mapping flow) > CapacityScheduler: Allow auto leaf queue creation after queue mapping > - > > Key: YARN-7419 > URL: https://issues.apache.org/jira/browse/YARN-7419 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Attachments: YARN-7419.1.patch, YARN-7419.2.patch, YARN-7419.3.patch, > YARN-7419.4.patch, YARN-7419.5.patch, YARN-7419.6.patch, YARN-7419.7.patch, > YARN-7419.8.patch, YARN-7419.9.patch, YARN-7419.patch > > > This involves changes to queue mapping flow to pass along context information > for auto queue creation. Auto creation of queues will be part of Capacity > Scheduler flow while attempting to resolve queues during application > submission. The leaf queues which do not exist are auto created under parent > queues which have been explicitly enabled for auto queue creation . In order > to determine which parent queue to create the leaf queues under - parent > queues need to be specified in queue mapping configuration -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7503) Configurable heap size / JVM opts in service AM
[ https://issues.apache.org/jira/browse/YARN-7503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255832#comment-16255832 ] Hudson commented on YARN-7503: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13248 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13248/]) YARN-7503. Configurable heap size / JVM opts in service AM. Contributed (jianhe: rev 6bf2c301924a3acae5a7510b8473f6292a5a471b) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/client/ServiceClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/conf/YarnServiceConf.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/containerlaunch/JavaCommandLineBuilder.java > Configurable heap size / JVM opts in service AM > --- > > Key: YARN-7503 > URL: https://issues.apache.org/jira/browse/YARN-7503 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 3.1.0 > > Attachments: YARN-7503.001.patch, YARN-7503.002.patch > > > Currently AM heap size / JVM opts is not configurable, should make it > configurable to prevent heap from growing well beyond the (configurable) AM > container memory size. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7483) CapacityScheduler test cases cleanup post YARN-5881
[ https://issues.apache.org/jira/browse/YARN-7483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7483: - Summary: CapacityScheduler test cases cleanup post YARN-5881 (was: Test cases cleanup post YARN-5881) > CapacityScheduler test cases cleanup post YARN-5881 > --- > > Key: YARN-7483 > URL: https://issues.apache.org/jira/browse/YARN-7483 > Project: Hadoop YARN > Issue Type: Sub-task > Components: test >Affects Versions: YARN-5881 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-7483-YARN-5881.001.patch, YARN-7483.001.patch > > > Few UT cases has to be cleaned or rewritten cleanly to consider both > absolute/percentage based configuration. > Related test classes > # TestLeafQueue > # TestCapacityScheduler > # TestCapacitySchedulerWithMultiResourceTypes -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7505) RM REST endpoints generate malformed JSON
[
https://issues.apache.org/jira/browse/YARN-7505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255837#comment-16255837
]
Eric Yang commented on YARN-7505:
-
[~templedf] YARN-7218 describes the same challenges with YARN services API.
Everything binded under Guice is pretty much stuck with the same Jaxb
configuration for jersey. Everything in /ws/* would be impacted if a change is
made. Hence, we took another approach to initialize YARN services API in a
separate servlet with prefix /app/v1/* which uses Jackson natural JSON for
serialization. Maybe YARN-7218 suggested approach can contain the problem by
separate out /ws/v2 into a separate servlet, and keep /ws/v1 to custom
serialization for backward compatibility only.
> RM REST endpoints generate malformed JSON
> -
>
> Key: YARN-7505
> URL: https://issues.apache.org/jira/browse/YARN-7505
> Project: Hadoop YARN
> Issue Type: Bug
> Components: restapi
>Affects Versions: 3.0.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-7505.001.patch, YARN-7505.002.patch
>
>
> For all endpoints that return DAOs that contain maps, the generated JSON is
> malformed. For example:
> % curl 'http://localhost:8088/ws/v1/cluster/apps'
> {"apps":{"app":[{"id":"application_1510777276702_0001","user":"daniel","name":"QuasiMonteCarlo","queue":"root.daniel","state":"RUNNING","finalStatus":"UNDEFINED","progress":5.0,"trackingUI":"ApplicationMaster","trackingUrl":"http://dhcp-10-16-0-181.pa.cloudera.com:8088/proxy/application_1510777276702_0001/","diagnostics":"","clusterId":1510777276702,"applicationType":"MAPREDUCE","applicationTags":"","priority":0,"startedTime":1510777317853,"finishedTime":0,"elapsedTime":21623,"amContainerLogs":"http://dhcp-10-16-0-181.pa.cloudera.com:8042/node/containerlogs/container_1510777276702_0001_01_01/daniel","amHostHttpAddress":"dhcp-10-16-0-181.pa.cloudera.com:8042","amRPCAddress":"dhcp-10-16-0-181.pa.cloudera.com:63371","allocatedMB":5120,"allocatedVCores":4,"reservedMB":0,"reservedVCores":0,"runningContainers":4,"memorySeconds":49820,"vcoreSeconds":26,"queueUsagePercentage":62.5,"clusterUsagePercentage":62.5,"resourceSecondsMap":{"entry":{"key":"test2","value":"0"},"entry":{"key":"test","value":"0"},"entry":{"key":"memory-mb","value":"49820"},"entry":{"key":"vcores","value":"26"}},"preemptedResourceMB":0,"preemptedResourceVCores":0,"numNonAMContainerPreempted":0,"numAMContainerPreempted":0,"preemptedMemorySeconds":0,"preemptedVcoreSeconds":0,"preemptedResourceSecondsMap":{},"resourceRequests":[{"priority":20,"resourceName":"dhcp-10-16-0-181.pa.cloudera.com","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"/default-rack","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false},{"priority":20,"resourceName":"*","capability":{"memory":1024,"vCores":1},"numContainers":8,"relaxLocality":true,"nodeLabelExpression":"","executionTypeRequest":{"executionType":"GUARANTEED","enforceExecutionType":true},"enforceExecutionType":false}],"logAggregationStatus":"DISABLED","unmanagedApplication":false,"amNodeLabelExpression":"","timeouts":{"timeout":[{"type":"LIFETIME","expiryTime":"UNLIMITED","remainingTimeInSeconds":-1}]}}]}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7503) Configurable heap size / JVM opts in service AM
[ https://issues.apache.org/jira/browse/YARN-7503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255846#comment-16255846 ] Jonathan Hung commented on YARN-7503: - Thanks [~jianhe]! > Configurable heap size / JVM opts in service AM > --- > > Key: YARN-7503 > URL: https://issues.apache.org/jira/browse/YARN-7503 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Fix For: 3.1.0 > > Attachments: YARN-7503.001.patch, YARN-7503.002.patch > > > Currently AM heap size / JVM opts is not configurable, should make it > configurable to prevent heap from growing well beyond the (configurable) AM > container memory size. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5534) Allow whitelisted volume mounts
[ https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255850#comment-16255850 ] Eric Yang commented on YARN-5534: - [~shaneku...@gmail.com][~ebadger] . Thanks for the input. I open a separate JIRA YARN-7516 to track the security check for running untrusted image. We can continue the discussion there. > Allow whitelisted volume mounts > > > Key: YARN-5534 > URL: https://issues.apache.org/jira/browse/YARN-5534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: luhuichun >Assignee: Shane Kumpf > Attachments: YARN-5534.001.patch, YARN-5534.002.patch, > YARN-5534.003.patch > > > Introduction > Mounting files or directories from the host is one way of passing > configuration and other information into a docker container. > We could allow the user to set a list of mounts in the environment of > ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2). > These would be mounted read-only to the specified target locations. This has > been resolved in YARN-4595 > 2.Problem Definition > Bug mounting arbitrary volumes into a Docker container can be a security risk. > 3.Possible solutions > one approach to provide safe mounts is to allow the cluster administrator to > configure a set of parent directories as white list mounting directories. > Add a property named yarn.nodemanager.volume-mounts.white-list, when > container executor do mount checking, only the allowed directories or > sub-directories can be mounted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7515) HA related tests fails in trunk
[
https://issues.apache.org/jira/browse/YARN-7515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated YARN-7515:
---
Affects Version/s: 3.0.0-beta1
> HA related tests fails in trunk
> ---
>
> Key: YARN-7515
> URL: https://issues.apache.org/jira/browse/YARN-7515
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: Bibin A Chundatt
>
> https://builds.apache.org/job/PreCommit-YARN-Build/18498/testReport/
> {noformat}
> org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands
>
> org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA
>
> org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
>
> org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7473) Implement Framework and policy for capacity management of auto created queues
[ https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suma Shivaprasad updated YARN-7473: --- Attachment: YARN-7473.1.patch > Implement Framework and policy for capacity management of auto created queues > -- > > Key: YARN-7473 > URL: https://issues.apache.org/jira/browse/YARN-7473 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Attachments: YARN-7473.1.patch > > > This jira mainly addresses the following > > 1.Support adding pluggable policies on parent queue for dynamically managing > capacity/state for leaf queues. > 2. Implement a default policy that manages capacity based on pending > applications and either grants guaranteed or zero capacity to queues based on > parent's available guaranteed capacity. > 3. Integrate with SchedulingEditPolicy framework to trigger this periodically > and signal scheduler to take necessary actions for capacity/queue management. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7473) Implement Framework and policy for capacity management of auto created queues
[ https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255854#comment-16255854 ] Suma Shivaprasad commented on YARN-7473: Uploading initial patch for reviews > Implement Framework and policy for capacity management of auto created queues > -- > > Key: YARN-7473 > URL: https://issues.apache.org/jira/browse/YARN-7473 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad > Attachments: YARN-7473.1.patch > > > This jira mainly addresses the following > > 1.Support adding pluggable policies on parent queue for dynamically managing > capacity/state for leaf queues. > 2. Implement a default policy that manages capacity based on pending > applications and either grants guaranteed or zero capacity to queues based on > parent's available guaranteed capacity. > 3. Integrate with SchedulingEditPolicy framework to trigger this periodically > and signal scheduler to take necessary actions for capacity/queue management. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7516) Security check for untrusted docker image
[ https://issues.apache.org/jira/browse/YARN-7516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255856#comment-16255856 ] Eric Yang commented on YARN-7516: - [~ebadger] Will do. Thanks for the advice. > Security check for untrusted docker image > - > > Key: YARN-7516 > URL: https://issues.apache.org/jira/browse/YARN-7516 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang > > Hadoop YARN Services can support using private docker registry image or > docker image from docker hub. In current implementation, Hadoop security is > enforced through username and group membership, and enforce uid:gid > consistency in docker container and distributed file system. There is cloud > use case for having ability to run untrusted docker image on the same cluster > for testing. > The basic requirement for untrusted container is to ensure all kernel and > root privileges are dropped, and there is no interaction with distributed > file system to avoid contamination. We can probably enforce detection of > untrusted docker image by checking the following: > # If docker image is from public docker hub repository, the container is > automatically flagged as insecure, and disk volume mount are disabled > automatically, and drop all kernel capabilities. > # If docker image is from private repository in docker hub, and there is a > white list to allow the private repository, disk volume mount is allowed, > kernel capabilities follows the allowed list. > # If docker image is from private trusted registry with image name like > "private.registry.local:5000/centos", and white list allows this private > trusted repository. Disk volume mount is allowed, kernel capabilities > follows the allowed list. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7448) [API] Add SchedulingRequest to the AllocateRequest
[
https://issues.apache.org/jira/browse/YARN-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255864#comment-16255864
]
Panagiotis Garefalakis commented on YARN-7448:
--
Thanks for assigning [~asuresh]
> You mean the SchedulingRequestBuilder constructor - True, but neither are a
> bunch of other things like PlacementConstraints, Allocation tags etc. But,
> given that it is probably a required field - might not be a bad idea to add
> it.
I meant the SchedulingRequest newInstance method - it seems resourceSizing
parameter was never used.
> Will assign this to you - can you put in another patch with the extra
> newInstance method ?
Sure, I am submitting now a new patch with some extra tests and the latest
AllocateRequest newInstance method.
> [API] Add SchedulingRequest to the AllocateRequest
> --
>
> Key: YARN-7448
> URL: https://issues.apache.org/jira/browse/YARN-7448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Panagiotis Garefalakis
> Attachments: YARN-7448-YARN-6592.001.patch,
> YARN-7448-YARN-6592.002.patch, YARN-7448-YARN-6592.003.patch
>
>
> YARN-6594 introduces the {{SchedulingRequest}}. This JIRA tracks the
> inclusion of the SchedulingRequest into the AllocateRequest.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7448) [API] Add SchedulingRequest to the AllocateRequest
[
https://issues.apache.org/jira/browse/YARN-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255864#comment-16255864
]
Panagiotis Garefalakis edited comment on YARN-7448 at 11/16/17 7:50 PM:
Thanks for assigning [~asuresh]
bq. You mean the SchedulingRequestBuilder constructor - True, but neither are a
bunch of other things like PlacementConstraints, Allocation tags etc. But,
given that it is probably a required field - might not be a bad idea to add it.
I meant the SchedulingRequest newInstance method - it seems resourceSizing
parameter was never used.
bq. Will assign this to you - can you put in another patch with the extra
newInstance method ?
Sure, I am submitting now a new patch with some extra tests and the latest
AllocateRequest newInstance method.
was (Author: pgaref):
Thanks for assigning [~asuresh]
> You mean the SchedulingRequestBuilder constructor - True, but neither are a
> bunch of other things like PlacementConstraints, Allocation tags etc. But,
> given that it is probably a required field - might not be a bad idea to add
> it.
I meant the SchedulingRequest newInstance method - it seems resourceSizing
parameter was never used.
> Will assign this to you - can you put in another patch with the extra
> newInstance method ?
Sure, I am submitting now a new patch with some extra tests and the latest
AllocateRequest newInstance method.
> [API] Add SchedulingRequest to the AllocateRequest
> --
>
> Key: YARN-7448
> URL: https://issues.apache.org/jira/browse/YARN-7448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Panagiotis Garefalakis
> Attachments: YARN-7448-YARN-6592.001.patch,
> YARN-7448-YARN-6592.002.patch, YARN-7448-YARN-6592.003.patch,
> YARN-7448-YARN-6592.004.patch
>
>
> YARN-6594 introduces the {{SchedulingRequest}}. This JIRA tracks the
> inclusion of the SchedulingRequest into the AllocateRequest.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7448) [API] Add SchedulingRequest to the AllocateRequest
[
https://issues.apache.org/jira/browse/YARN-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Panagiotis Garefalakis updated YARN-7448:
-
Attachment: YARN-7448-YARN-6592.004.patch
> [API] Add SchedulingRequest to the AllocateRequest
> --
>
> Key: YARN-7448
> URL: https://issues.apache.org/jira/browse/YARN-7448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Panagiotis Garefalakis
> Attachments: YARN-7448-YARN-6592.001.patch,
> YARN-7448-YARN-6592.002.patch, YARN-7448-YARN-6592.003.patch,
> YARN-7448-YARN-6592.004.patch
>
>
> YARN-6594 introduces the {{SchedulingRequest}}. This JIRA tracks the
> inclusion of the SchedulingRequest into the AllocateRequest.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7419) CapacityScheduler: Allow auto leaf queue creation after queue mapping
[ https://issues.apache.org/jira/browse/YARN-7419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255867#comment-16255867 ] Hudson commented on YARN-7419: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13249 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13249/]) YARN-7419. CapacityScheduler: Allow auto leaf queue creation after queue (wangda: rev 0987a7b8c2c1e4c2095821d98a7db19644df) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestAppManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacitySchedulerAutoQueueCreation.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AutoCreatedLeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/PlanQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ManagedParentQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/PlacementManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/ApplicationPlacementContext.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/AppAddedSchedulerEvent.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractManagedParentQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java > CapacityScheduler: Allow auto leaf queue creation after queue mapping > - > > Key: YARN-7419 > URL: https://issues.apache.org/jira/browse/YARN-7419 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Su
[jira] [Commented] (YARN-7438) Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest / placement algorithm
[ https://issues.apache.org/jira/browse/YARN-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255878#comment-16255878 ] Wangda Tan commented on YARN-7438: -- [~sunilg], bq. lastPendingAsk and newPendingAsk seems a little tricky to understand and its per Scheduler Key. Could we rename this better. I just more explanations to PendingAskUpdateResult, please let me know if it is sufficient. bq. Once we save set of resource requests associated with a given container, it might have various resource names (node local, rack local etc). But ANY will be common. I agree with this as well as [~asuresh]'s suggestion. Under the context of YARN-6592, we may need SchedulingRequest (or request-agnostic class) to save results. So I prefer to keep it as-is and revisit this in the future. [~asuresh], any suggestions here? bq. lastPendingAsk could be cached? I'm quite sure what does this mean, could you elaborate? [~asuresh], bq. We should change the name of AppPlacementAllocator#updateResourceRequests to updatePendingAsk Done. Attached ver.2 patch. Please kindly review. > Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest > / placement algorithm > --- > > Key: YARN-7438 > URL: https://issues.apache.org/jira/browse/YARN-7438 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-7438.001.patch > > > In additional to YARN-6040, we need to make changes to SchedulingPlacementSet > to make it: > 1) Agnostic to ResourceRequest (so once we have YARN-6592 merged, we can add > new SchedulingPlacementSet implementation in parallel with > LocalitySchedulingPlacementSet to use/manage new requests API) > 2) Agnostic to placement algorithm (now it is bind to delayed scheduling, we > should update APIs to make sure new placement algorithms such as complex > placement algorithms can be implemented by using SchedulingPlacementSet). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7438) Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest / placement algorithm
[ https://issues.apache.org/jira/browse/YARN-7438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7438: - Attachment: YARN-7438.002.patch > Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest > / placement algorithm > --- > > Key: YARN-7438 > URL: https://issues.apache.org/jira/browse/YARN-7438 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-7438.001.patch, YARN-7438.002.patch > > > In additional to YARN-6040, we need to make changes to SchedulingPlacementSet > to make it: > 1) Agnostic to ResourceRequest (so once we have YARN-6592 merged, we can add > new SchedulingPlacementSet implementation in parallel with > LocalitySchedulingPlacementSet to use/manage new requests API) > 2) Agnostic to placement algorithm (now it is bind to delayed scheduling, we > should update APIs to make sure new placement algorithms such as complex > placement algorithms can be implemented by using SchedulingPlacementSet). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7448) [API] Add SchedulingRequest to the AllocateRequest
[
https://issues.apache.org/jira/browse/YARN-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255885#comment-16255885
]
Wangda Tan commented on YARN-7448:
--
Thanks [~pgaref] for working on this. In general patch looks good. [~asuresh],
please proceed committing once you think it is ready.
> [API] Add SchedulingRequest to the AllocateRequest
> --
>
> Key: YARN-7448
> URL: https://issues.apache.org/jira/browse/YARN-7448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Panagiotis Garefalakis
> Attachments: YARN-7448-YARN-6592.001.patch,
> YARN-7448-YARN-6592.002.patch, YARN-7448-YARN-6592.003.patch,
> YARN-7448-YARN-6592.004.patch
>
>
> YARN-6594 introduces the {{SchedulingRequest}}. This JIRA tracks the
> inclusion of the SchedulingRequest into the AllocateRequest.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7487) Make sure volume includes GPU base libraries exists after created by plugin
[ https://issues.apache.org/jira/browse/YARN-7487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-7487: - Attachment: YARN-7487.003.patch Attached ver.3 patch addressed cc warnings. > Make sure volume includes GPU base libraries exists after created by plugin > --- > > Key: YARN-7487 > URL: https://issues.apache.org/jira/browse/YARN-7487 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-7487.002.patch, YARN-7487.003.patch, > YARN-7487.wip.001.patch > > > YARN-7224 will create docker volume includes GPU base libraries when launch a > docker container which needs GPU. > This JIRA will add necessary checks to make sure docker volume exists before > launching the container to reduce debug efforts if container fails. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7469) Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit
[ https://issues.apache.org/jira/browse/YARN-7469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255891#comment-16255891 ] Eric Payne commented on YARN-7469: -- Thank you very much [~sunilg]. Just one concern is that this fix should also go into branch-2.9 since it is also in branch-2.8. > Capacity Scheduler Intra-queue preemption: User can starve if newest app is > exactly at user limit > - > > Key: YARN-7469 > URL: https://issues.apache.org/jira/browse/YARN-7469 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, yarn >Affects Versions: 2.9.0, 3.0.0-beta1, 2.8.2 >Reporter: Eric Payne >Assignee: Eric Payne > Fix For: 2.8.3, 3.0.0, 3.1.0, 2.10.0 > > Attachments: UnitTestToShowStarvedUser.patch, YARN-7469.001.patch > > > Queue Configuration: > - Total Memory: 20GB > - 2 Queues > -- Queue1 > --- Memory: 10GB > --- MULP: 10% > --- ULF: 2.0 > - Minimum Container Size: 0.5GB > Use Case: > - User1 submits app1 to Queue1 and consumes 20GB > - User2 submits app2 to Queue1 and requests 7.5GB > - Preemption monitor preempts 7.5GB from app1. Capacity Scheduler gives those > resources to User2 > - User 3 submits app3 to Queue1. To begin with, app3 is requesting 1 > container for the AM. > - Preemption monitor never preempts a container. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7513) FindBugs in FSAppAttempt.getWeight()
[
https://issues.apache.org/jira/browse/YARN-7513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255896#comment-16255896
]
Hadoop QA commented on YARN-7513:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m
56s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
14s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 26s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}130m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Unreaped Processes | hadoop-yarn-server-resourcemanager:2 |
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAsyncScheduling
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing |
| Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.TestProportionalCapacityPreemptionPolicy
|
| |
org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA |
| |
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7513 |
| JIRA Patch URL |
[jira] [Commented] (YARN-4813) TestRMWebServicesDelegationTokenAuthentication.testDoAs fails intermittently
[
https://issues.apache.org/jira/browse/YARN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255898#comment-16255898
]
Robert Kanter commented on YARN-4813:
-
If there's a system property for disabling the cache, I'd rather us do that
because it will work 100% of the time. Adding some rate limiting sounds like
it will improve things, but can never be 100% perfect unless we really slow it
down a lot. I do agree that we don't want to leak the system property change
to other tests, but we can just make sure to set it back once the test is done.
> TestRMWebServicesDelegationTokenAuthentication.testDoAs fails intermittently
>
>
> Key: YARN-4813
> URL: https://issues.apache.org/jira/browse/YARN-4813
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Gergo Repas
> Attachments: YARN-4813.000.patch
>
>
> {noformat}
> ---
> T E S T S
> ---
> Running
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
> Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 11.627 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication
> testDoAs[0](org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication)
> Time elapsed: 0.208 sec <<< ERROR!
> java.io.IOException: Server returned HTTP response code: 403 for URL:
> http://localhost:8088/ws/v1/cluster/delegation-token
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1626)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication$3.call(TestRMWebServicesDelegationTokenAuthentication.java:407)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication$3.call(TestRMWebServicesDelegationTokenAuthentication.java:398)
> at
> org.apache.hadoop.security.authentication.KerberosTestUtils$1.run(KerberosTestUtils.java:120)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.authentication.KerberosTestUtils.doAs(KerberosTestUtils.java:117)
> at
> org.apache.hadoop.security.authentication.KerberosTestUtils.doAsClient(KerberosTestUtils.java:133)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication.getDelegationToken(TestRMWebServicesDelegationTokenAuthentication.java:398)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokenAuthentication.testDoAs(TestRMWebServicesDelegationTokenAuthentication.java:357)
> Results :
> Tests in error:
>
> TestRMWebServicesDelegationTokenAuthentication.testDoAs:357->getDelegationToken:398
> ยป IO
> Tests run: 8, Failures: 0, Errors: 1, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7448) [API] Add SchedulingRequest to the AllocateRequest
[
https://issues.apache.org/jira/browse/YARN-7448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16255901#comment-16255901
]
Konstantinos Karanasos commented on YARN-7448:
--
Thank you guys for working on this. Please give me until the end of the day to
give the patch a look too. Thanks!
> [API] Add SchedulingRequest to the AllocateRequest
> --
>
> Key: YARN-7448
> URL: https://issues.apache.org/jira/browse/YARN-7448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Panagiotis Garefalakis
> Attachments: YARN-7448-YARN-6592.001.patch,
> YARN-7448-YARN-6592.002.patch, YARN-7448-YARN-6592.003.patch,
> YARN-7448-YARN-6592.004.patch
>
>
> YARN-6594 introduces the {{SchedulingRequest}}. This JIRA tracks the
> inclusion of the SchedulingRequest into the AllocateRequest.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
