[jira] [Commented] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642842#comment-16642842
]
Arun Suresh commented on YARN-8827:
---
Updating patch with checkstyle fixes. The test case failure should be fixed
when we rebase the branch with trunk (it is handled by YARN-8844)
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch,
> YARN-8827-YARN-1011.02.patch, YARN-8827-YARN-1011.03.patch,
> YARN-8827-YARN-1011.04.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-8827:
--
Attachment: YARN-8827-YARN-1011.04.patch
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch,
> YARN-8827-YARN-1011.02.patch, YARN-8827-YARN-1011.03.patch,
> YARN-8827-YARN-1011.04.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8859) Add audit logs for router service
[ https://issues.apache.org/jira/browse/YARN-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-8859: --- Component/s: router > Add audit logs for router service > - > > Key: YARN-8859 > URL: https://issues.apache.org/jira/browse/YARN-8859 > Project: Hadoop YARN > Issue Type: Sub-task > Components: router >Reporter: Bibin A Chundatt >Priority: Major > > Similar to all other yarn services. > RouterClientRMService and RouterWebServices api/rest call should have Audit > logging. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8859) Add audit logs for router service
Bibin A Chundatt created YARN-8859: -- Summary: Add audit logs for router service Key: YARN-8859 URL: https://issues.apache.org/jira/browse/YARN-8859 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bibin A Chundatt Similar to all other yarn services. RouterClientRMService and RouterWebServices api/rest call should have Audit logging. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8858) CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used.
[ https://issues.apache.org/jira/browse/YARN-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642828#comment-16642828 ] Wangda Tan edited comment on YARN-8858 at 10/9/18 5:39 AM: --- [~cheersyang] / [~sunil.gov...@gmail.com], can we get this patch committed to 3.2.0? -YARN-8720- introduced minor behavior change, we should fix it if possible. was (Author: leftnoteasy): [~cheersyang] / [~sunil.gov...@gmail.com], can we get this patch committed to 3.2.0? It is a (minor) behavior change for YARN-8720. > CapacityScheduler should respect maximum node resource when per-queue > maximum-allocation is being used. > --- > > Key: YARN-8858 > URL: https://issues.apache.org/jira/browse/YARN-8858 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sumana Sathish >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-8858.001.patch > > > This issue happens after YARN-8720. > Before that, AMS uses scheduler.getMaximumAllocation to do the normalization. > After that, AMS uses LeafQueue.getMaximumAllocation. The scheduler one uses > nodeTracker.getMaximumAllocation, but the LeafQueue.getMaximum doesn't. > We should use the scheduler.getMaximumAllocation to cap the per-queue's > maximum-allocation every time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8858) CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used.
[ https://issues.apache.org/jira/browse/YARN-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642828#comment-16642828 ] Wangda Tan commented on YARN-8858: -- [~cheersyang] / [~sunil.gov...@gmail.com], can we get this patch committed to 3.2.0? It is a (minor) behavior change for YARN-8720. > CapacityScheduler should respect maximum node resource when per-queue > maximum-allocation is being used. > --- > > Key: YARN-8858 > URL: https://issues.apache.org/jira/browse/YARN-8858 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sumana Sathish >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-8858.001.patch > > > This issue happens after YARN-8720. > Before that, AMS uses scheduler.getMaximumAllocation to do the normalization. > After that, AMS uses LeafQueue.getMaximumAllocation. The scheduler one uses > nodeTracker.getMaximumAllocation, but the LeafQueue.getMaximum doesn't. > We should use the scheduler.getMaximumAllocation to cap the per-queue's > maximum-allocation every time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8858) CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used.
[ https://issues.apache.org/jira/browse/YARN-8858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-8858: - Attachment: YARN-8858.001.patch > CapacityScheduler should respect maximum node resource when per-queue > maximum-allocation is being used. > --- > > Key: YARN-8858 > URL: https://issues.apache.org/jira/browse/YARN-8858 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sumana Sathish >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-8858.001.patch > > > This issue happens after YARN-8720. > Before that, AMS uses scheduler.getMaximumAllocation to do the normalization. > After that, AMS uses LeafQueue.getMaximumAllocation. The scheduler one uses > nodeTracker.getMaximumAllocation, but the LeafQueue.getMaximum doesn't. > We should use the scheduler.getMaximumAllocation to cap the per-queue's > maximum-allocation every time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8858) CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used.
Wangda Tan created YARN-8858: Summary: CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used. Key: YARN-8858 URL: https://issues.apache.org/jira/browse/YARN-8858 Project: Hadoop YARN Issue Type: Bug Reporter: Sumana Sathish Assignee: Wangda Tan This issue happens after YARN-8720. Before that, AMS uses scheduler.getMaximumAllocation to do the normalization. After that, AMS uses LeafQueue.getMaximumAllocation. The scheduler one uses nodeTracker.getMaximumAllocation, but the LeafQueue.getMaximum doesn't. We should use the scheduler.getMaximumAllocation to cap the per-queue's maximum-allocation every time. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5742) Serve aggregated logs of historical apps from timeline service
[ https://issues.apache.org/jira/browse/YARN-5742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642808#comment-16642808 ] Rohith Sharma K S commented on YARN-5742: - [~vrushalic] [~abmodi] Could you please take a look at patch? > Serve aggregated logs of historical apps from timeline service > -- > > Key: YARN-5742 > URL: https://issues.apache.org/jira/browse/YARN-5742 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Rohith Sharma K S >Priority: Critical > Attachments: YARN-5742-POC-v0.patch, YARN-5742.01.patch, > YARN-5742.v0.patch > > > ATSv1.5 daemon has servlet to serve aggregated logs. But enabling only ATSv2, > does not serve logs from CLI and UI for completed application. Log serving > story has completely broken in ATSv2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8856) TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError
[ https://issues.apache.org/jira/browse/YARN-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642797#comment-16642797 ] Rohith Sharma K S commented on YARN-8856: - [~jlowe] Is this failing in trunk? > TestTimelineReaderWebServicesHBaseStorage tests failing with > NoClassDefFoundError > - > > Key: YARN-8856 > URL: https://issues.apache.org/jira/browse/YARN-8856 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jason Lowe >Priority: Major > > TestTimelineReaderWebServicesHBaseStorage has been failing in nightly builds > with NoClassDefFoundError in the tests. Sample error and stacktrace to > follow. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8590) Fair scheduler promotion does not update container execution type and token
[ https://issues.apache.org/jira/browse/YARN-8590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642791#comment-16642791 ] Miklos Szegedi commented on YARN-8590: -- I think it is okay to do the code refactoring in another patch. [~zsiegl], what do you think? > Fair scheduler promotion does not update container execution type and token > --- > > Key: YARN-8590 > URL: https://issues.apache.org/jira/browse/YARN-8590 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: YARN-1011 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8590-YARN-1011.00.patch, > YARN-8590-YARN-1011.01.patch, YARN-8590-YARN-1011.02.patch > > > Fair Scheduler promotion of opportunistic containers does not update > container execution type and token. This leads to incorrect resource > accounting when the promoted containers are released. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-8802) [JDK9] Fail to run yarn application after building hadoop pkg with jdk9 in jdk9 env
[
https://issues.apache.org/jira/browse/YARN-8802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved YARN-8802.
-
Resolution: Duplicate
> [JDK9] Fail to run yarn application after building hadoop pkg with jdk9 in
> jdk9 env
> ---
>
> Key: YARN-8802
> URL: https://issues.apache.org/jira/browse/YARN-8802
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: liyunzhang
>Priority: Critical
> Attachments: HADOOP-14984.patch
>
>
> After building latest code with jdk9. (patch HADOOP-12760.03.patch,
> HDFS-11610.001.patch). And start hdfs, yarn service(HADOOP-14978)
> successfully. I met exception when running TestDFSIO
> {code}
> hadoop jar
> $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.0-SNAPSHOT-tests.jar
> TestDFSIO -write -nrFiles 8 -fileSize 1MB -resFile ./write.1MB.8
> {code}
> the exception
> {code}
> 67 1) Error injecting constructor, java.lang.NoClassDefFoundError:
> javax/activation/DataSource
> 68 at
> org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver.(JAXBContextResolver.java:72)
> 69 at
> org.apache.hadoop.mapreduce.v2.app.webapp.AMWebApp.setup(AMWebApp.java:33)
> 70 while locating
> org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver
> 71
> 72 1 error
> 73 at
> com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1025)
> 74 at
> com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
> 75 at
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory$GuiceInstantiatedComponentProvider.getInstance(GuiceComponentProviderFactory.java:345)
> 76 at
> com.sun.jersey.core.spi.component.ioc.IoCProviderFactory$ManagedSingleton.(IoCProviderFactory.java:202)
> 77 at
> com.sun.jersey.core.spi.component.ioc.IoCProviderFactory.wrap(IoCProviderFactory.java:123)
> 78 at
> com.sun.jersey.core.spi.component.ioc.IoCProviderFactory._getComponentProvider(IoCProviderFactory.java:116)
> 79 at
> com.sun.jersey.core.spi.component.ProviderFactory.getComponentProvider(ProviderFactory.java:153)
> 80 at
> com.sun.jersey.core.spi.component.ProviderServices.getComponent(ProviderServices.java:278)
> 81 at
> com.sun.jersey.core.spi.component.ProviderServices.getProviders(ProviderServices.java:151)
> 82 at
> com.sun.jersey.core.spi.factory.ContextResolverFactory.init(ContextResolverFactory.java:83)
> 83 at
> com.sun.jersey.server.impl.application.WebApplicationImpl._initiate(WebApplicationImpl.java:1332)
> 84 at
> com.sun.jersey.server.impl.application.WebApplicationImpl.access$700(WebApplicationImpl.java:180)
> 85 at
> com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:799)
> 86 at
> com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:795)
> 87 at
> com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
> 88 at
> com.sun.jersey.server.impl.application.WebApplicationImpl.initiate(WebApplicationImpl.java:795)
> 89 at
> com.sun.jersey.guice.spi.container.servlet.GuiceContainer.initiate(GuiceContainer.java:121)
> 90 at
> com.sun.jersey.spi.container.servlet.ServletContainer$InternalWebComponent.initiate(ServletContainer.java:339)
> 91 at
> com.sun.jersey.spi.container.servlet.WebComponent.load(WebComponent.java:605)
> 92 at
> com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:207)
> 93 at
> com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:394)
> 94 at
> com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:744)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8802) [JDK9] Fail to run yarn application after building hadoop pkg with jdk9 in jdk9 env
[
https://issues.apache.org/jira/browse/YARN-8802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642789#comment-16642789
]
Akira Ajisaka commented on YARN-8802:
-
Closing this as duplicate of HADOOP-15775.
> [JDK9] Fail to run yarn application after building hadoop pkg with jdk9 in
> jdk9 env
> ---
>
> Key: YARN-8802
> URL: https://issues.apache.org/jira/browse/YARN-8802
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: liyunzhang
>Priority: Critical
> Attachments: HADOOP-14984.patch
>
>
> After building latest code with jdk9. (patch HADOOP-12760.03.patch,
> HDFS-11610.001.patch). And start hdfs, yarn service(HADOOP-14978)
> successfully. I met exception when running TestDFSIO
> {code}
> hadoop jar
> $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.0-SNAPSHOT-tests.jar
> TestDFSIO -write -nrFiles 8 -fileSize 1MB -resFile ./write.1MB.8
> {code}
> the exception
> {code}
> 67 1) Error injecting constructor, java.lang.NoClassDefFoundError:
> javax/activation/DataSource
> 68 at
> org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver.(JAXBContextResolver.java:72)
> 69 at
> org.apache.hadoop.mapreduce.v2.app.webapp.AMWebApp.setup(AMWebApp.java:33)
> 70 while locating
> org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver
> 71
> 72 1 error
> 73 at
> com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1025)
> 74 at
> com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1051)
> 75 at
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory$GuiceInstantiatedComponentProvider.getInstance(GuiceComponentProviderFactory.java:345)
> 76 at
> com.sun.jersey.core.spi.component.ioc.IoCProviderFactory$ManagedSingleton.(IoCProviderFactory.java:202)
> 77 at
> com.sun.jersey.core.spi.component.ioc.IoCProviderFactory.wrap(IoCProviderFactory.java:123)
> 78 at
> com.sun.jersey.core.spi.component.ioc.IoCProviderFactory._getComponentProvider(IoCProviderFactory.java:116)
> 79 at
> com.sun.jersey.core.spi.component.ProviderFactory.getComponentProvider(ProviderFactory.java:153)
> 80 at
> com.sun.jersey.core.spi.component.ProviderServices.getComponent(ProviderServices.java:278)
> 81 at
> com.sun.jersey.core.spi.component.ProviderServices.getProviders(ProviderServices.java:151)
> 82 at
> com.sun.jersey.core.spi.factory.ContextResolverFactory.init(ContextResolverFactory.java:83)
> 83 at
> com.sun.jersey.server.impl.application.WebApplicationImpl._initiate(WebApplicationImpl.java:1332)
> 84 at
> com.sun.jersey.server.impl.application.WebApplicationImpl.access$700(WebApplicationImpl.java:180)
> 85 at
> com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:799)
> 86 at
> com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:795)
> 87 at
> com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
> 88 at
> com.sun.jersey.server.impl.application.WebApplicationImpl.initiate(WebApplicationImpl.java:795)
> 89 at
> com.sun.jersey.guice.spi.container.servlet.GuiceContainer.initiate(GuiceContainer.java:121)
> 90 at
> com.sun.jersey.spi.container.servlet.ServletContainer$InternalWebComponent.initiate(ServletContainer.java:339)
> 91 at
> com.sun.jersey.spi.container.servlet.WebComponent.load(WebComponent.java:605)
> 92 at
> com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:207)
> 93 at
> com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:394)
> 94 at
> com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:744)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642751#comment-16642751
]
Hadoop QA commented on YARN-8827:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 5m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
33s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
40s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
0s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
49s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
41s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
44s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
7s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 20s{color} | {color:orange} root: The patch generated 45 new + 178 unchanged
- 0 fixed = 223 total (was 178) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
48s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
29s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 23s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 77m
21s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
33s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 2m
6s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}236m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.TestNMProxy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce
[jira] [Commented] (YARN-8807) FairScheduler crashes RM with oversubscription turned on if an application is killed.
[
https://issues.apache.org/jira/browse/YARN-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642705#comment-16642705
]
Hadoop QA commented on YARN-8807:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
41s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 27s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 2s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}137m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.TestApplicationMasterService |
| |
hadoop.yarn.server.resourcemanager.metrics.TestCombinedSystemMetricsPublisher |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8807 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942929/YARN-8807-YARN-1011.01.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 81b730c69666 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-1011 / efd8524 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/22106/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22106/testReport/ |
| Max. process+thread count | 982 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642702#comment-16642702
]
Hadoop QA commented on YARN-8827:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
48s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
28s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
28s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
28s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
16s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
15s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
13s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 14m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 45s{color} | {color:orange} root: The patch generated 41 new + 179 unchanged
- 0 fixed = 220 total (was 179) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 39s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
50s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
28s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 19s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 20s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
28s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}213m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.TestNMProxy |
| | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler |
\\
\\
||
[jira] [Commented] (YARN-8811) Support Container Storage Interface (CSI) in YARN
[
https://issues.apache.org/jira/browse/YARN-8811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642697#comment-16642697
]
Weiwei Yang commented on YARN-8811:
---
Thanks [~eyang] for the feedback. The idea behind this is to create an abstract
layer standing between YARN and various of storage systems, and that layer is
built upon CSI spec. This allows us to support various of storage systems
without changing a lot of YARN internals. Given there is already eco-system
around CSI, I think this is a good direction for us too. And technically, the
CSI driver should be CO independent, this was listed as the first goal to
address per CSI spec
{quote}Enable SP authors to write one CSI compliant Plugin that “just works”
across all COs that implement CSI.
{quote}
Hope that clarifies the motivation.
I agree with your suggestion about making the user interface more easier to
use, e.g use unified source path as URI form to make this more consistent for
user to consume. Lets get the infra stuff built first and see if that can be
done on top of that. Hope that makes sense.
Thanks
> Support Container Storage Interface (CSI) in YARN
> -
>
> Key: YARN-8811
> URL: https://issues.apache.org/jira/browse/YARN-8811
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: Support Container Storage Interface(CSI) in YARN_design
> doc_20180921.pdf, Support Container Storage Interface(CSI) in YARN_design
> doc_20180928.pdf
>
>
> The Container Storage Interface (CSI) is a vendor neutral interface to bridge
> Container Orchestrators and Storage Providers. With the adoption of CSI in
> YARN, it will be easier to integrate 3rd party storage systems, and provide
> the ability to attach persistent volumes for stateful applications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642690#comment-16642690
]
Hadoop QA commented on YARN-8827:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
44s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 24m
37s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
0s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
18s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
23m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
37s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
3s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 15m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 15m
57s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 21s{color} | {color:orange} root: The patch generated 41 new + 178 unchanged
- 0 fixed = 219 total (was 178) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
57s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 49s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 72m
20s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
29s{color} | {color:green} hadoop-sls in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}240m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.TestNMProxy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce
[jira] [Commented] (YARN-8813) Improve debug messages for NM preemption of OPPORTUNISTIC containers
[
https://issues.apache.org/jira/browse/YARN-8813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642675#comment-16642675
]
Hadoop QA commented on YARN-8813:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-1011 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
20s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} YARN-1011 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} YARN-1011 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 21s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 44s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 76m 25s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.TestNMProxy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8813 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942931/YARN-8813-YARN-1011.01.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux fb1264aa6801 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-1011 / efd8524 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (YARN-7225) Add queue and partition info to RM audit log
[
https://issues.apache.org/jira/browse/YARN-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642664#comment-16642664
]
Hadoop QA commented on YARN-7225:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 3 new + 80 unchanged - 0 fixed = 83 total (was 80) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 34s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}135m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-7225 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942912/YARN-7225.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 863b120bc6a3 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 347ea38 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/22104/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| whitespace |
[jira] [Comment Edited] (YARN-7644) NM gets backed up deleting docker containers
[
https://issues.apache.org/jira/browse/YARN-7644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642595#comment-16642595
]
Chandni Singh edited comment on YARN-7644 at 10/9/18 12:43 AM:
---
[~jlowe] please see my response below
{quote}IIUC the launchContainer method for the executor is a synchronous,
blocking call that won't return until the container completes. For example, see
DefaultContainerExecutor#launchContainer where it invokes
Shell.CommandExecutor#execute. That means the executor lock would be held
continuously while the container is running. Therefore I'm not sure how the
thread running ContainerLaunch#reapContainer is going to obtain the executor
lock to be able to proceed to kill the container. Seems like it would just
hang, but maybe I'm missing something. This may be more of an issue with
YARN-8160 than this one, as it looks like this mostly just refactored existing
code to move it into a ContainerCleanup class.
{quote}
Before {{reapContainer()}}, container term/kill signal is always sent. This is
not blocked. With YARN-8160, we wait for {{launchContainer()}} to complete
after the signal is sent and then perform the {{reapContainer(). }}
Note: reapContainer removes the container. The stopping of container by
sending KILL/TERM is not part of reapContainer. It is done before reap.
{quote}To be honest I'm not quite sure what the purpose of the lock is, since
there are many places we invoke the executor without the lock like deactivating
and signalling. The use of the lock seems inconsistent if it's supposed to
guard when we are invoking the executor.
{quote}
This is the comment that describes the issue which the change in YARN-8160
fixed:
https://issues.apache.org/jira/browse/YARN-8160?focusedCommentId=16570774=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16570774
I will summarize it here.
* Container is launched
* Re-init of container is requested
* Re-init triggers container stop and removes the container
* Meanwhile the container launch exits with 255 because the container files
are cleaned up by reap container. This is because after the executor exits the
launch, it performs docker inspect
With the executorLock, we are waiting for the executor.launchContainer to
complete after term/kill signal is sent to it. Once the launch is completed, we
have the correct exit code from the container. Then the reap is performed.
Possibly, the name {{executorLock}} is confusing which I can change?
I will address your other comments in the next patch.
was (Author: csingh):
[~jlowe] please see my response below
{quote}IIUC the launchContainer method for the executor is a synchronous,
blocking call that won't return until the container completes. For example, see
DefaultContainerExecutor#launchContainer where it invokes
Shell.CommandExecutor#execute. That means the executor lock would be held
continuously while the container is running. Therefore I'm not sure how the
thread running ContainerLaunch#reapContainer is going to obtain the executor
lock to be able to proceed to kill the container. Seems like it would just
hang, but maybe I'm missing something. This may be more of an issue with
YARN-8160 than this one, as it looks like this mostly just refactored existing
code to move it into a ContainerCleanup class.
{quote}
Before {{reapContainer()}}, container term/kill signal is always sent. This is
not blocked. With YARN-8160, we wait for {{launchContainer()}} to complete
after the signal is sent and then perform the {{reapContainer()}}
{quote}To be honest I'm not quite sure what the purpose of the lock is, since
there are many places we invoke the executor without the lock like deactivating
and signalling. The use of the lock seems inconsistent if it's supposed to
guard when we are invoking the executor.
{quote}
This is the comment that describes the issue which the change in YARN-8160
fixed:
https://issues.apache.org/jira/browse/YARN-8160?focusedCommentId=16570774=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16570774
I will summarize it here.
* Container is launched
* Re-init of container is requested
* Re-init triggers container stop and removes the container
* Meanwhile the container launch exits with 255 because the container files
are cleaned up by reap container. This is because after the executor exits the
launch, it performs docker inspect
With the executorLock, we are waiting for the executor.launchContainer to
complete after term/kill signal is sent to it. Once the launch is completed, we
have the correct exit code from the container. Then the reap is performed.
Possibly, the name {{executorLock}} is confusing which I can change?
I will address your other comments in the next patch.
> NM gets backed up deleting docker containers
>
[jira] [Commented] (YARN-8448) AM HTTPS Support
[
https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642647#comment-16642647
]
Robert Kanter commented on YARN-8448:
-
Thanks [~haibochen] for the review. That's a good idea about BouncyCastle;
that'll make it easier to iterate on this patch because we won't have to wait
6+ hours each time :). I've filed HADOOP-15832 and put up a patch with just
the pom changes there. Once that's in, I'll update this JIRA's patch.
On your second point, if the policy is REQUIRED, the RM won't proxy you to a
non-HTTPS AM. You'll instead get a warning page, similar to the
{{WebAppProxyServlet#warnUserPage}} code that warns the user in certain
situations when Kerberos is enabled. Take a look at the
{{WebAppProxyServlet#checkHttpsRequiredAndNotProvided}} method to see where
this is done. When set to OPTIONAL, this behavior doesn't trigger. If we fail
the AM, I'm concerned that it's going to make it harder for users with older
apps that can't be updated to use HTTPS. As it is now, with REQUIRED, you can
still run the AM if it's using HTTP, you just can't access it's web page (with
OPTIONAL, you'd still be able to access it's web page). Does that make sense?
> AM HTTPS Support
>
>
> Key: YARN-8448
> URL: https://issues.apache.org/jira/browse/YARN-8448
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Robert Kanter
>Assignee: Robert Kanter
>Priority: Major
> Attachments: YARN-8448.001.patch, YARN-8448.002.patch,
> YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch,
> YARN-8448.006.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8845) hadoop.registry.rm.enabled is not used
[
https://issues.apache.org/jira/browse/YARN-8845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642646#comment-16642646
]
Hadoop QA commented on YARN-8845:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 12s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 42s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
31s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
58s{color} | {color:green} hadoop-yarn-registry in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
17s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}104m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |
[jira] [Created] (YARN-8857) Upgrade BouncyCastle
Robert Kanter created YARN-8857:
---
Summary: Upgrade BouncyCastle
Key: YARN-8857
URL: https://issues.apache.org/jira/browse/YARN-8857
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 3.2.0
Reporter: Robert Kanter
Assignee: Robert Kanter
As part of my work on YARN-6586, I noticed that we're using a very old version
of BouncyCastle:
{code:xml}
org.bouncycastle
bcprov-jdk16
1.46
test
{code}
The *-jdk16 artifacts have been discontinued and are not recommended (see
[http://bouncy-castle.1462172.n4.nabble.com/Bouncycaslte-bcprov-jdk15-vs-bcprov-jdk16-td4656252.html]).
In particular, the newest release, 1.46, is from {color:#FF}2011{color}!
[https://mvnrepository.com/artifact/org.bouncycastle/bcprov-jdk16]
The currently maintained and recommended artifacts are *-jdk15on:
[https://www.bouncycastle.org/latest_releases.html]
They're currently on version 1.60, released only a few months ago.
We should update BouncyCastle to the *-jdk15on artifacts and the 1.60 release.
It's currently a test-only artifact, so there should be no
backwards-compatibility issues with updating this. It's also needed for
YARN-6586, where we'll actually be shipping it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8813) Improve debug messages for NM preemption of OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-8813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-8813: - Attachment: YARN-8813-YARN-1011.01.patch > Improve debug messages for NM preemption of OPPORTUNISTIC containers > > > Key: YARN-8813 > URL: https://issues.apache.org/jira/browse/YARN-8813 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: YARN-1011 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8813-YARN-1011.00.patch, > YARN-8813-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8813) Improve debug messages for NM preemption of OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-8813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642623#comment-16642623 ] Haibo Chen commented on YARN-8813: -- Thanks [~rkanter] for your review. I addressed your comments in a new patch. > Improve debug messages for NM preemption of OPPORTUNISTIC containers > > > Key: YARN-8813 > URL: https://issues.apache.org/jira/browse/YARN-8813 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: YARN-1011 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Major > Attachments: YARN-8813-YARN-1011.00.patch, > YARN-8813-YARN-1011.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8807) FairScheduler crashes RM with oversubscription turned on if an application is killed.
[
https://issues.apache.org/jira/browse/YARN-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642621#comment-16642621
]
Haibo Chen commented on YARN-8807:
--
Thanks for the review, [~rkanter]. I updated the patch to fix the compilation
issue.
> FairScheduler crashes RM with oversubscription turned on if an application is
> killed.
> -
>
> Key: YARN-8807
> URL: https://issues.apache.org/jira/browse/YARN-8807
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, resourcemanager
>Affects Versions: YARN-1011
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8807-YARN-1011.00.patch,
> YARN-8807-YARN-1011.01.patch
>
>
> When an application, that has got opportunistic containers allocated, is
> killed, its containers are not released immediately.
> Fair scheduler would therefore continue to try to promote such orphaned
> containers, which results in NPE.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptToAssignReservedResourcesOrPromoteOpportunisticContainers(FairScheduler.java:1158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1129)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:1001)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testKillingApplicationWithOpportunisticContainersAssigned(TestFairScheduler.java:4019){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8807) FairScheduler crashes RM with oversubscription turned on if an application is killed.
[
https://issues.apache.org/jira/browse/YARN-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated YARN-8807:
-
Attachment: YARN-8807-YARN-1011.01.patch
> FairScheduler crashes RM with oversubscription turned on if an application is
> killed.
> -
>
> Key: YARN-8807
> URL: https://issues.apache.org/jira/browse/YARN-8807
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, resourcemanager
>Affects Versions: YARN-1011
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8807-YARN-1011.00.patch,
> YARN-8807-YARN-1011.01.patch
>
>
> When an application, that has got opportunistic containers allocated, is
> killed, its containers are not released immediately.
> Fair scheduler would therefore continue to try to promote such orphaned
> containers, which results in NPE.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptToAssignReservedResourcesOrPromoteOpportunisticContainers(FairScheduler.java:1158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1129)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:1001)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testKillingApplicationWithOpportunisticContainersAssigned(TestFairScheduler.java:4019){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8448) AM HTTPS Support
[ https://issues.apache.org/jira/browse/YARN-8448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642603#comment-16642603 ] Haibo Chen commented on YARN-8448: -- Thanks [~rkanter] for the the patch. I took a high-level look. The overall approach looks good to me. I do have two comment/questions. 1) We can pull the bounty-castle upgrade into a separate patch 2) Does it make sense to add a check on the AM-RM registeration path? That is, if AM gives a HTTP tracking url when the policy is set to Required, maybe we should fail the AM? Otherwise, REQUIRED is just the same as optional. > AM HTTPS Support > > > Key: YARN-8448 > URL: https://issues.apache.org/jira/browse/YARN-8448 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Major > Attachments: YARN-8448.001.patch, YARN-8448.002.patch, > YARN-8448.003.patch, YARN-8448.004.patch, YARN-8448.005.patch, > YARN-8448.006.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642596#comment-16642596
]
Arun Suresh edited comment on YARN-8827 at 10/8/18 11:29 PM:
-
Updating patch based on [~elgoiri]'s suggestions..
bq. I was talking about making nm1, nm2, nm3, nm4 part of the class as fields.
Ahk.. yup, cleaned it up - I guess it looks better now ?
was (Author: asuresh):
Updating patch based on your suggestions..
bq. I was talking about making nm1, nm2, nm3, nm4 part of the class as fields.
Ahk.. yup, cleaned it up - I guess it looks better now
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch,
> YARN-8827-YARN-1011.02.patch, YARN-8827-YARN-1011.03.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642596#comment-16642596
]
Arun Suresh commented on YARN-8827:
---
Updating patch based on your suggestions..
bq. I was talking about making nm1, nm2, nm3, nm4 part of the class as fields.
Ahk.. yup, cleaned it up - I guess it looks better now
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch,
> YARN-8827-YARN-1011.02.patch, YARN-8827-YARN-1011.03.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-8827:
--
Attachment: YARN-8827-YARN-1011.03.patch
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch,
> YARN-8827-YARN-1011.02.patch, YARN-8827-YARN-1011.03.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7644) NM gets backed up deleting docker containers
[
https://issues.apache.org/jira/browse/YARN-7644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642595#comment-16642595
]
Chandni Singh commented on YARN-7644:
-
[~jlowe] please see my response below
{quote}IIUC the launchContainer method for the executor is a synchronous,
blocking call that won't return until the container completes. For example, see
DefaultContainerExecutor#launchContainer where it invokes
Shell.CommandExecutor#execute. That means the executor lock would be held
continuously while the container is running. Therefore I'm not sure how the
thread running ContainerLaunch#reapContainer is going to obtain the executor
lock to be able to proceed to kill the container. Seems like it would just
hang, but maybe I'm missing something. This may be more of an issue with
YARN-8160 than this one, as it looks like this mostly just refactored existing
code to move it into a ContainerCleanup class.
{quote}
Before {{reapContainer()}}, container term/kill signal is always sent. This is
not blocked. With YARN-8160, we wait for {{launchContainer()}} to complete
after the signal is sent and then perform the {{reapContainer()}}
{quote}To be honest I'm not quite sure what the purpose of the lock is, since
there are many places we invoke the executor without the lock like deactivating
and signalling. The use of the lock seems inconsistent if it's supposed to
guard when we are invoking the executor.
{quote}
This is the comment that describes the issue which the change in YARN-8160
fixed:
https://issues.apache.org/jira/browse/YARN-8160?focusedCommentId=16570774=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16570774
I will summarize it here.
* Container is launched
* Re-init of container is requested
* Re-init triggers container stop and removes the container
* Meanwhile the container launch exits with 255 because the container files
are cleaned up by reap container. This is because after the executor exits the
launch, it performs docker inspect
With the executorLock, we are waiting for the executor.launchContainer to
complete after term/kill signal is sent to it. Once the launch is completed, we
have the correct exit code from the container. Then the reap is performed.
Possibly, the name {{executorLock}} is confusing which I can change?
I will address your other comments in the next patch.
> NM gets backed up deleting docker containers
>
>
> Key: YARN-7644
> URL: https://issues.apache.org/jira/browse/YARN-7644
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Eric Badger
>Assignee: Chandni Singh
>Priority: Major
> Labels: Docker
> Attachments: YARN-7644.001.patch, YARN-7644.002.patch
>
>
> We are sending a {{docker stop}} to the docker container with a timeout of 10
> seconds when we shut down a container. If the container does not stop after
> 10 seconds then we force kill it. However, the {{docker stop}} command is a
> blocking call. So in cases where lots of containers don't go down with the
> initial SIGTERM, we have to wait 10+ seconds for the {{docker stop}} to
> return. This ties up the ContainerLaunch handler and so these kill events
> back up. It also appears to be backing up new container launches as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642579#comment-16642579
]
Íñigo Goiri commented on YARN-8827:
---
bq. Hmmm, not sure what you meant by this. The test case uses the MockNM and
MockRM classes which I think we use throughout most CapacityScheduler tests.
Not sure how we can move it inside the class.
I was talking about making nm1, nm2, nm3, nm4 part of the class as fields.
Then, we could have a method to trigger all heartbeats.
Probably also call the drainEvents and wait for event thread, etc.
bq. Ive added a javadoc before the testcase and some demarcation within the
testcase - to mark begining and end of each step - hope that clears things ?
Looks good. I would probably quote the relevant numbers that are being used
here.
bq. If you don't mind, id like to keep it as 'e' The point was to reduce the
typing and length of the line. Also I don't plan to re-use it ouside the this
testcase, so lets keep it as private. If I do reuse it, I will create a
TestUtil class and put everything there - and probably rename it.
OK.
For the sleep removal, this looks better.
Why do we have to run the heartbeats three times?
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch,
> YARN-8827-YARN-1011.02.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7644) NM gets backed up deleting docker containers
[
https://issues.apache.org/jira/browse/YARN-7644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642576#comment-16642576
]
Jason Lowe commented on YARN-7644:
--
Thanks for the patch! I'm a little concerned about the container executor lock
that was added in YARN-8160 and used as part of this patch. IIUC the
launchContainer method for the executor is a synchronous, blocking call that
won't return until the container completes. For example, see
DefaultContainerExecutor#launchContainer where it invokes
Shell.CommandExecutor#execute. That means the executor lock would be held
continuously while the container is running. Therefore I'm not sure how the
thread running ContainerLaunch#reapContainer is going to obtain the executor
lock to be able to proceed to kill the container. Seems like it would just
hang, but maybe I'm missing something. This may be more of an issue with
YARN-8160 than this one, as it looks like this mostly just refactored existing
code to move it into a ContainerCleanup class. To be honest I'm not quite sure
what the purpose of the lock is, since there are many places we invoke the
executor without the lock like deactivating and signalling. The use of the
lock seems inconsistent if it's supposed to guard when we are invoking the
executor.
Nit: It feels a little off for ContainerCleanup to reach into fields directly,
e.g.: launch.pidFilePath, launch.containerAlreadyLaunched, launch.completed,
etc. It made more sense when this code was part of ContainerLaunch because the
field was private and no code other than the implementation details of
ContainerLaunch needed to know. Now there's another, separate class that is
reaching in to grab all these fields. Seems like either ContainerCleanup
should be a private static class of ContainerLaunch or there should be
accessors so we can keep these fields private.
ContainerLaunch.sleepDelayBeforeSigKill should be final like the other
properties. The assignment to 250 will always be clobbered by the constructor
anyway.
> NM gets backed up deleting docker containers
>
>
> Key: YARN-7644
> URL: https://issues.apache.org/jira/browse/YARN-7644
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Eric Badger
>Assignee: Chandni Singh
>Priority: Major
> Labels: Docker
> Attachments: YARN-7644.001.patch, YARN-7644.002.patch
>
>
> We are sending a {{docker stop}} to the docker container with a timeout of 10
> seconds when we shut down a container. If the container does not stop after
> 10 seconds then we force kill it. However, the {{docker stop}} command is a
> blocking call. So in cases where lots of containers don't go down with the
> initial SIGTERM, we have to wait 10+ seconds for the {{docker stop}} to
> return. This ties up the ContainerLaunch handler and so these kill events
> back up. It also appears to be backing up new container launches as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8807) FairScheduler crashes RM with oversubscription turned on if an application is killed.
[
https://issues.apache.org/jira/browse/YARN-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642549#comment-16642549
]
Robert Kanter edited comment on YARN-8807 at 10/8/18 10:07 PM:
---
Looks good to me, though something must have changed since the patch was
created because it doesn't compile anymore:
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project hadoop-yarn-server-resourcemanager:
Compilation failure: Compilation failure:
[ERROR]
/Users/rkanter/dev/hadoop-git/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java:[4012,11]
cannot find symbol
[ERROR] symbol: method
updateContainersAndNodeUtilization(org.apache.hadoop.yarn.server.resourcemanager.rmnode.UpdatedContainerInfo,org.apache.hadoop.yarn.api.records.ResourceUtilization)
[ERROR] location: variable node of type
org.apache.hadoop.yarn.server.resourcemanager.MockNodes.MockRMNodeImpl
[ERROR]
/Users/rkanter/dev/hadoop-git/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java:[4041,11]
cannot find symbol
[ERROR] symbol: method
updateContainersAndNodeUtilization(org.apache.hadoop.yarn.server.resourcemanager.rmnode.UpdatedContainerInfo,org.apache.hadoop.yarn.api.records.ResourceUtilization)
[ERROR] location: variable node of type
org.apache.hadoop.yarn.server.resourcemanager.MockNodes.MockRMNodeImpl
{noformat}
was (Author: rkanter):
Looks good to me, though something must have changed since the patch was
created because it doesn't compile anymore.
> FairScheduler crashes RM with oversubscription turned on if an application is
> killed.
> -
>
> Key: YARN-8807
> URL: https://issues.apache.org/jira/browse/YARN-8807
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, resourcemanager
>Affects Versions: YARN-1011
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8807-YARN-1011.00.patch
>
>
> When an application, that has got opportunistic containers allocated, is
> killed, its containers are not released immediately.
> Fair scheduler would therefore continue to try to promote such orphaned
> containers, which results in NPE.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptToAssignReservedResourcesOrPromoteOpportunisticContainers(FairScheduler.java:1158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1129)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:1001)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testKillingApplicationWithOpportunisticContainersAssigned(TestFairScheduler.java:4019){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8807) FairScheduler crashes RM with oversubscription turned on if an application is killed.
[
https://issues.apache.org/jira/browse/YARN-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642549#comment-16642549
]
Robert Kanter commented on YARN-8807:
-
Looks good to me, though something must have changed since the patch was
created because it doesn't compile anymore.
> FairScheduler crashes RM with oversubscription turned on if an application is
> killed.
> -
>
> Key: YARN-8807
> URL: https://issues.apache.org/jira/browse/YARN-8807
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: fairscheduler, resourcemanager
>Affects Versions: YARN-1011
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8807-YARN-1011.00.patch
>
>
> When an application, that has got opportunistic containers allocated, is
> killed, its containers are not released immediately.
> Fair scheduler would therefore continue to try to promote such orphaned
> containers, which results in NPE.
> {code:java}
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptToAssignReservedResourcesOrPromoteOpportunisticContainers(FairScheduler.java:1158)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.attemptScheduling(FairScheduler.java:1129)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.nodeUpdate(FairScheduler.java:1001)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1275)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testKillingApplicationWithOpportunisticContainersAssigned(TestFairScheduler.java:4019){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-3890) FairScheduler should show the scheduler health metrics similar to ones added in CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Siegl reassigned YARN-3890: -- Assignee: Zoltan Siegl (was: Gergo Repas) > FairScheduler should show the scheduler health metrics similar to ones added > in CapacityScheduler > - > > Key: YARN-3890 > URL: https://issues.apache.org/jira/browse/YARN-3890 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Reporter: Anubhav Dhoot >Assignee: Zoltan Siegl >Priority: Major > > We should add information displayed in YARN-3293 in FairScheduler as well > possibly sharing the implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7225) Add queue and partition info to RM audit log
[ https://issues.apache.org/jira/browse/YARN-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated YARN-7225: - Affects Version/s: 2.9.1 2.8.4 3.0.2 3.1.1 > Add queue and partition info to RM audit log > > > Key: YARN-7225 > URL: https://issues.apache.org/jira/browse/YARN-7225 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.9.1, 2.8.4, 3.0.2, 3.1.1 >Reporter: Jonathan Hung >Assignee: Eric Payne >Priority: Major > Attachments: YARN-7225.001.patch, YARN-7225.002.patch, > YARN-7225.003.patch > > > Right now RM audit log has fields such as user, ip, resource, etc. Having > queue and partition is useful for resource tracking. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8856) TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError
[
https://issues.apache.org/jira/browse/YARN-8856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642514#comment-16642514
]
Jason Lowe commented on YARN-8856:
--
Sample test failure:
{noformat}
java.io.IOException: Incorrect response from timeline reader. Status=500
at
org.apache.hadoop.yarn.server.timelineservice.reader.AbstractTimelineReaderHBaseTestBase.getResponse(AbstractTimelineReaderHBaseTestBase.java:129)
at
org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage.testGetAppWithoutFlowInfo(TestTimelineReaderWebServicesHBaseStorage.java:1064)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
{noformat}
and this is in the test's stdout:
{noformat}
2018-10-08 09:42:57,722 ERROR [1028538462@qtp-2008619427-0] mortbay.log
(Slf4jLog.java:warn(87)) - Error for
/ws/v2/timeline/clusters/cluster1/apps/application_11_
java.lang.NoClassDefFoundError: Could not initialize class
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
com.sun.jersey.server.spi.component.ResourceComponentConstructor._construct(ResourceComponentConstructor.java:245)
at
com.sun.jersey.server.spi.component.ResourceComponentConstructor.construct(ResourceComponentConstructor.java:233)
at
com.sun.jersey.server.impl.resource.PerRequestFactory$PerRequest._getInstance(PerRequestFactory.java:182)
at
com.sun.jersey.server.impl.resource.PerRequestFactory$AbstractPerRequest.getInstance(PerRequestFactory.java:144)
at
com.sun.jersey.server.impl.application.WebApplicationContext.getResource(WebApplicationContext.java:239)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:83)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
at
[jira] [Created] (YARN-8856) TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError
Jason Lowe created YARN-8856: Summary: TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError Key: YARN-8856 URL: https://issues.apache.org/jira/browse/YARN-8856 Project: Hadoop YARN Issue Type: Bug Reporter: Jason Lowe TestTimelineReaderWebServicesHBaseStorage has been failing in nightly builds with NoClassDefFoundError in the tests. Sample error and stacktrace to follow. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7225) Add queue and partition info to RM audit log
[ https://issues.apache.org/jira/browse/YARN-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated YARN-7225: - Attachment: YARN-7225.003.patch > Add queue and partition info to RM audit log > > > Key: YARN-7225 > URL: https://issues.apache.org/jira/browse/YARN-7225 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Jonathan Hung >Assignee: Eric Payne >Priority: Major > Attachments: YARN-7225.001.patch, YARN-7225.002.patch, > YARN-7225.003.patch > > > Right now RM audit log has fields such as user, ip, resource, etc. Having > queue and partition is useful for resource tracking. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642496#comment-16642496
]
Arun Suresh edited comment on YARN-8827 at 10/8/18 9:13 PM:
Thanks for the review [~elgoiri]
Attached patch addressing most of your comments. save the few below:
bq. .. Given that this class is only used for this, I would make the NMs part
of the class and trigger registrations and so on as part of the class. ...
Hmmm, not sure what you meant by this. The test case uses the {{MockNM}} and
{{MockRM}} classes which I think we use throughout most CapacityScheduler
tests. Not sure how we can move it inside the class.
bq. .. I would also add a couple more comments throughout explaining how the
apps are submitting requests, etc. Specially the numbers after #184. ...
Ive added a javadoc before the testcase and some demarcation within the
testcase - to mark begining and end of each step - hope that clears things ?
bq. mkmap() and e() are an OK pattern but I would probably call the second one
mkmapentry and add javadoc for the two explaining the purpose.
If you don't mind, id like to keep it as 'e' :) The point was to reduce the
typing and length of the line. Also I don't plan to re-use it ouside the this
testcase, so lets keep it as private. If I do reuse it, I will create a
TestUtil class and put everything there - and probably rename it.
I've also cleaned the findbugs. The test case error though seems unrelated.
was (Author: asuresh):
Thanks for the review [~elgoiri]
Attached patch addressing most of your comments. save the few below:
bq. .. Given that this class is only used for this, I would make the NMs part
of the class and trigger registrations and so on as part of the class. ...
Hmmm, not sure what you meant by this. The test case uses the {{MockNM}} and
{{MockRM}} classes which I think we use throughout most CapacityScheduler
tests. Not sure how we can move it inside the class.
bq. .. I would also add a couple more comments throughout explaining how the
apps are submitting requests, etc. Specially the numbers after #184. ...
Ive added a javadoc before the testcase and some demarcation within the
testcase - to mark begining and end of each step - hope that clears things ?
bq. mkmap() and e() are an OK pattern but I would probably call the second one
mkmapentry and add javadoc for the two explaining the purpose.
If you don't mind, id like to keep it as 'e' :) The point was to reduce the
typing and length of the line. Also I don't plan to re-use it ouside the this
testcase, so lets keep it as private. If I d reuse it, I will create a TestUtil
class and put everything there - and probably rename it.
I've also cleaned the findbugs. The test case error though seems unrelated.
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch,
> YARN-8827-YARN-1011.02.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642496#comment-16642496
]
Arun Suresh commented on YARN-8827:
---
Thanks for the review [~elgoiri]
Attached patch addressing most of your comments. save the few below:
bq. .. Given that this class is only used for this, I would make the NMs part
of the class and trigger registrations and so on as part of the class. ...
Hmmm, not sure what you meant by this. The test case uses the {{MockNM}} and
{{MockRM}} classes which I think we use throughout most CapacityScheduler
tests. Not sure how we can move it inside the class.
bq. .. I would also add a couple more comments throughout explaining how the
apps are submitting requests, etc. Specially the numbers after #184. ...
Ive added a javadoc before the testcase and some demarcation within the
testcase - to mark begining and end of each step - hope that clears things ?
bq. mkmap() and e() are an OK pattern but I would probably call the second one
mkmapentry and add javadoc for the two explaining the purpose.
If you don't mind, id like to keep it as 'e' :) The point was to reduce the
typing and length of the line. Also I don't plan to re-use it ouside the this
testcase, so lets keep it as private. If I d reuse it, I will create a TestUtil
class and put everything there - and probably rename it.
I've also cleaned the findbugs. The test case error though seems unrelated.
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch,
> YARN-8827-YARN-1011.02.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated YARN-8827:
--
Attachment: YARN-8827-YARN-1011.02.patch
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch,
> YARN-8827-YARN-1011.02.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8569) Create an interface to provide cluster information to application
[
https://issues.apache.org/jira/browse/YARN-8569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642483#comment-16642483
]
Suma Shivaprasad commented on YARN-8569:
[~eyang] Patch 10 Looks good. A few minor checkstyle issues would need to be
resolved.
> Create an interface to provide cluster information to application
> -
>
> Key: YARN-8569
> URL: https://issues.apache.org/jira/browse/YARN-8569
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
> Attachments: YARN-8569 YARN sysfs interface to provide cluster
> information to application.pdf, YARN-8569.001.patch, YARN-8569.002.patch,
> YARN-8569.003.patch, YARN-8569.004.patch, YARN-8569.005.patch,
> YARN-8569.006.patch, YARN-8569.007.patch, YARN-8569.008.patch,
> YARN-8569.009.patch, YARN-8569.010.patch
>
>
> Some program requires container hostnames to be known for application to run.
> For example, distributed tensorflow requires launch_command that looks like:
> {code}
> # On ps0.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=ps --task_index=0
> # On ps1.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=ps --task_index=1
> # On worker0.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=worker --task_index=0
> # On worker1.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=worker --task_index=1
> {code}
> This is a bit cumbersome to orchestrate via Distributed Shell, or YARN
> services launch_command. In addition, the dynamic parameters do not work
> with YARN flex command. This is the classic pain point for application
> developer attempt to automate system environment settings as parameter to end
> user application.
> It would be great if YARN Docker integration can provide a simple option to
> expose hostnames of the yarn service via a mounted file. The file content
> gets updated when flex command is performed. This allows application
> developer to consume system environment settings via a standard interface.
> It is like /proc/devices for Linux, but for Hadoop. This may involve
> updating a file in distributed cache, and allow mounting of the file via
> container-executor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8569) Create an interface to provide cluster information to application
[
https://issues.apache.org/jira/browse/YARN-8569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642465#comment-16642465
]
Hadoop QA commented on YARN-8569:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 8m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
29s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 25s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 14 new + 146 unchanged - 1 fixed = 160 total (was 147) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
49s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
58s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m
17s{color} | {color:green} hadoop-yarn-services-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
22s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black}
[jira] [Commented] (YARN-8845) hadoop.registry.rm.enabled is not used
[
https://issues.apache.org/jira/browse/YARN-8845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642425#comment-16642425
]
Íñigo Goiri commented on YARN-8845:
---
[^YARN-8845.001.patch] should take care of the last reference to
{{hadoop.registry.rm.enabled}}.
I checked *.java and *.md and nothing there.
> hadoop.registry.rm.enabled is not used
> --
>
> Key: YARN-8845
> URL: https://issues.apache.org/jira/browse/YARN-8845
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.1.1
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: YARN-8845.000.patch, YARN-8845.001.patch
>
>
> YARN-2652 introduced "hadoop.registry.rm.enabled" as YARN-2571 was supposed
> to initialize the registry but that's now gone. We should remove all the
> references to this configuration key.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8845) hadoop.registry.rm.enabled is not used
[ https://issues.apache.org/jira/browse/YARN-8845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-8845: -- Attachment: YARN-8845.001.patch > hadoop.registry.rm.enabled is not used > -- > > Key: YARN-8845 > URL: https://issues.apache.org/jira/browse/YARN-8845 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: YARN-8845.000.patch, YARN-8845.001.patch > > > YARN-2652 introduced "hadoop.registry.rm.enabled" as YARN-2571 was supposed > to initialize the registry but that's now gone. We should remove all the > references to this configuration key. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8813) Improve debug messages for NM preemption of OPPORTUNISTIC containers
[
https://issues.apache.org/jira/browse/YARN-8813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642413#comment-16642413
]
Robert Kanter commented on YARN-8813:
-
The debug messages look fine to me, other than a small grammer nitpick:
- In {{ContainersMonitorImpl}}, instead of "Update latest..." it should be
"Updated latest..." because it happens after the actual update.
And one minor related issue that we may as well take care of while we're here:
- In {{CGroupElasticMemoryController#getDefaultOOMHandler}}, it looks up the
{{oomHandlerClass}} first, but only uses it if {{oomHandlerLocal}} is {{null}}.
We can move that lookup into the if statement to save on the call in the cases
where {{oomHandlerLocal}} isn't {{null}}.
> Improve debug messages for NM preemption of OPPORTUNISTIC containers
>
>
> Key: YARN-8813
> URL: https://issues.apache.org/jira/browse/YARN-8813
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: YARN-1011
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-8813-YARN-1011.00.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8845) hadoop.registry.rm.enabled is not used
[ https://issues.apache.org/jira/browse/YARN-8845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642385#comment-16642385 ] Eric Yang commented on YARN-8845: - [~elgoiri] Thank you for the patch. Registry-configuration.md still has example configuration that use: hadoop.registry.rm.enabled after applying patch 000. > hadoop.registry.rm.enabled is not used > -- > > Key: YARN-8845 > URL: https://issues.apache.org/jira/browse/YARN-8845 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.1 >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: YARN-8845.000.patch > > > YARN-2652 introduced "hadoop.registry.rm.enabled" as YARN-2571 was supposed > to initialize the registry but that's now gone. We should remove all the > references to this configuration key. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8846) Allow Applications to demand Guaranteed Containers with Capacity Scheduler
[
https://issues.apache.org/jira/browse/YARN-8846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated YARN-8846:
-
Summary: Allow Applications to demand Guaranteed Containers with Capacity
Scheduler (was: Allow Applications to demand Guaranteed Containers)
> Allow Applications to demand Guaranteed Containers with Capacity Scheduler
> --
>
> Key: YARN-8846
> URL: https://issues.apache.org/jira/browse/YARN-8846
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
>
> The Capacity Scheduler should ensure that if the {{enforceExecutionType}}
> flag in the resource request is {{true}} and the requested Container is of
> {{GUARANTEED}} type, the Capacity scheduler should not return over-allocated
> containers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8503) Add unit test for FINISHED_CONTAINERS_PULLED_BY_AM event on DECOMMISSIONING
[
https://issues.apache.org/jira/browse/YARN-8503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642358#comment-16642358
]
Hadoop QA commented on YARN-8503:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color}
| {color:red} YARN-8503 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-8503 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942893/YARN-8503.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22100/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add unit test for FINISHED_CONTAINERS_PULLED_BY_AM event on DECOMMISSIONING
> ---
>
> Key: YARN-8503
> URL: https://issues.apache.org/jira/browse/YARN-8503
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Affects Versions: 2.7.2
>Reporter: Amiya Chakraborty
>Priority: Major
> Labels: patch-available, yarn
> Attachments: YARN-8503.001.patch, YARN-8503.001.patch
>
>
> Currently, there is no unit test for testing the functionality -
> FINISHED_CONTAINERS_PULLED_BY_AM event while Decommissioning of node. This
> patch provides the same to check the AM has pulled the containers from the
> RM; then the RM will inform the NM about it and the NM can remove the
> completed container from its list during DECOMMISSIONING.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8763) Add WebSocket logic to the Node Manager web server to establish servlet
[ https://issues.apache.org/jira/browse/YARN-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642268#comment-16642268 ] Hudson commented on YARN-8763: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15141 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15141/]) YARN-8763. Added node manager websocket API for accessing containers. (eyang: rev 347ea385817766a5c418017009728cd8b9959776) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerShellWebSocket.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestLinuxContainerExecutor.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMContainerWebSocket.java * (edit) hadoop-client-modules/hadoop-client-runtime/pom.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerShellWebSocketServlet.java * (edit) hadoop-project/pom.xml * (edit) hadoop-client-modules/hadoop-client-minicluster/pom.xml * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/ContainerShellClientSocketTest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerExecutor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/pom.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/WebServer.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/executor/ContainerExecContext.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitorResourceChange.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java > Add WebSocket logic to the Node Manager web server to establish servlet > --- > > Key: YARN-8763 > URL: https://issues.apache.org/jira/browse/YARN-8763 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zian Chen >Assignee: Zian Chen >Priority: Major > Labels: Docker > Fix For: 3.3.0 > > Attachments: YARN-8763-001.patch, YARN-8763.002.patch, > YARN-8763.003.patch, YARN-8763.004.patch, YARN-8763.005.patch > > > The reason we want to use WebSocket servlet to serve the backend instead of > establishing the connection through HTTP is that WebSocket solves a few > issues with HTTP which needed for our scenario, > # In HTTP, the request is always initiated by the client and the response is > processed by the server — making HTTP a unidirectional protocol, while web > socket provides the Bi-directional protocol which means either client/server > can send a message to the other party. > # Full-duplex communication — client and server can talk to each other > independently at the same time > # Single TCP connection — After upgrading the HTTP connection in the > beginning, client and server communicate over that same TCP connection > throughout the lifecycle of WebSocket connection -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7054) Yarn Service Phase 2
[ https://issues.apache.org/jira/browse/YARN-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642246#comment-16642246 ] Peter Bacsko commented on YARN-7054: [~eyang] thanks for your reply! Yes, unfortunately our branch is Hadoop-2 based, that's bad news. But we do have commits from YARN-3611, albeit not all of them. I'll reach out to our devs to see whether it's worth the effort to backport this JIRA alone or just abandon it. > Yarn Service Phase 2 > > > Key: YARN-7054 > URL: https://issues.apache.org/jira/browse/YARN-7054 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jian He >Assignee: Jian He >Priority: Major > Fix For: yarn-native-services > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8569) Create an interface to provide cluster information to application
[
https://issues.apache.org/jira/browse/YARN-8569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642226#comment-16642226
]
Eric Yang commented on YARN-8569:
-
Patch10 renamed YARN_CONTAINER_RUNTIME_YARN_SYSFS to
YARN_CONTAINER_RUNTIME_YARN_SYSFS_ENABLE.
> Create an interface to provide cluster information to application
> -
>
> Key: YARN-8569
> URL: https://issues.apache.org/jira/browse/YARN-8569
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
> Attachments: YARN-8569 YARN sysfs interface to provide cluster
> information to application.pdf, YARN-8569.001.patch, YARN-8569.002.patch,
> YARN-8569.003.patch, YARN-8569.004.patch, YARN-8569.005.patch,
> YARN-8569.006.patch, YARN-8569.007.patch, YARN-8569.008.patch,
> YARN-8569.009.patch, YARN-8569.010.patch
>
>
> Some program requires container hostnames to be known for application to run.
> For example, distributed tensorflow requires launch_command that looks like:
> {code}
> # On ps0.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=ps --task_index=0
> # On ps1.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=ps --task_index=1
> # On worker0.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=worker --task_index=0
> # On worker1.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=worker --task_index=1
> {code}
> This is a bit cumbersome to orchestrate via Distributed Shell, or YARN
> services launch_command. In addition, the dynamic parameters do not work
> with YARN flex command. This is the classic pain point for application
> developer attempt to automate system environment settings as parameter to end
> user application.
> It would be great if YARN Docker integration can provide a simple option to
> expose hostnames of the yarn service via a mounted file. The file content
> gets updated when flex command is performed. This allows application
> developer to consume system environment settings via a standard interface.
> It is like /proc/devices for Linux, but for Hadoop. This may involve
> updating a file in distributed cache, and allow mounting of the file via
> container-executor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8569) Create an interface to provide cluster information to application
[
https://issues.apache.org/jira/browse/YARN-8569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-8569:
Attachment: YARN-8569.010.patch
> Create an interface to provide cluster information to application
> -
>
> Key: YARN-8569
> URL: https://issues.apache.org/jira/browse/YARN-8569
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Labels: Docker
> Attachments: YARN-8569 YARN sysfs interface to provide cluster
> information to application.pdf, YARN-8569.001.patch, YARN-8569.002.patch,
> YARN-8569.003.patch, YARN-8569.004.patch, YARN-8569.005.patch,
> YARN-8569.006.patch, YARN-8569.007.patch, YARN-8569.008.patch,
> YARN-8569.009.patch, YARN-8569.010.patch
>
>
> Some program requires container hostnames to be known for application to run.
> For example, distributed tensorflow requires launch_command that looks like:
> {code}
> # On ps0.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=ps --task_index=0
> # On ps1.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=ps --task_index=1
> # On worker0.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=worker --task_index=0
> # On worker1.example.com:
> $ python trainer.py \
> --ps_hosts=ps0.example.com:,ps1.example.com: \
> --worker_hosts=worker0.example.com:,worker1.example.com: \
> --job_name=worker --task_index=1
> {code}
> This is a bit cumbersome to orchestrate via Distributed Shell, or YARN
> services launch_command. In addition, the dynamic parameters do not work
> with YARN flex command. This is the classic pain point for application
> developer attempt to automate system environment settings as parameter to end
> user application.
> It would be great if YARN Docker integration can provide a simple option to
> expose hostnames of the yarn service via a mounted file. The file content
> gets updated when flex command is performed. This allows application
> developer to consume system environment settings via a standard interface.
> It is like /proc/devices for Linux, but for Hadoop. This may involve
> updating a file in distributed cache, and allow mounting of the file via
> container-executor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8777) Container Executor C binary change to execute interactive docker command
[ https://issues.apache.org/jira/browse/YARN-8777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642220#comment-16642220 ] Zian Chen commented on YARN-8777: - +1 for patch 7. > Container Executor C binary change to execute interactive docker command > > > Key: YARN-8777 > URL: https://issues.apache.org/jira/browse/YARN-8777 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zian Chen >Assignee: Eric Yang >Priority: Major > Labels: Docker > Attachments: YARN-8777.001.patch, YARN-8777.002.patch, > YARN-8777.003.patch, YARN-8777.004.patch, YARN-8777.005.patch, > YARN-8777.006.patch, YARN-8777.007.patch > > > Since Container Executor provides Container execution using the native > container-executor binary, we also need to make changes to accept new > “dockerExec” method to invoke the corresponding native function to execute > docker exec command to the running container. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8763) Add WebSocket logic to the Node Manager web server to establish servlet
[ https://issues.apache.org/jira/browse/YARN-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642219#comment-16642219 ] Zian Chen commented on YARN-8763: - [~eyang], thanks for +1. Could you help me commit this patch? Thanks > Add WebSocket logic to the Node Manager web server to establish servlet > --- > > Key: YARN-8763 > URL: https://issues.apache.org/jira/browse/YARN-8763 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zian Chen >Assignee: Zian Chen >Priority: Major > Labels: Docker > Attachments: YARN-8763-001.patch, YARN-8763.002.patch, > YARN-8763.003.patch, YARN-8763.004.patch, YARN-8763.005.patch > > > The reason we want to use WebSocket servlet to serve the backend instead of > establishing the connection through HTTP is that WebSocket solves a few > issues with HTTP which needed for our scenario, > # In HTTP, the request is always initiated by the client and the response is > processed by the server — making HTTP a unidirectional protocol, while web > socket provides the Bi-directional protocol which means either client/server > can send a message to the other party. > # Full-duplex communication — client and server can talk to each other > independently at the same time > # Single TCP connection — After upgrading the HTTP connection in the > beginning, client and server communicate over that same TCP connection > throughout the lifecycle of WebSocket connection -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8843) updateNodeResource does not support units for memory
[ https://issues.apache.org/jira/browse/YARN-8843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642214#comment-16642214 ] Íñigo Goiri commented on YARN-8843: --- Cherry-picked to branch-3.2. > updateNodeResource does not support units for memory > > > Key: YARN-8843 > URL: https://issues.apache.org/jira/browse/YARN-8843 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Manikandan R >Priority: Minor > Fix For: 3.2.0, 3.3.0 > > Attachments: YARN-8843.000.patch, YARN-8843.001.patch > > > When doing -updateNodeResource memory-mb=1Gi, it assigns 1MB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8843) updateNodeResource does not support units for memory
[ https://issues.apache.org/jira/browse/YARN-8843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated YARN-8843: -- Fix Version/s: 3.2.0 > updateNodeResource does not support units for memory > > > Key: YARN-8843 > URL: https://issues.apache.org/jira/browse/YARN-8843 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Manikandan R >Priority: Minor > Fix For: 3.2.0, 3.3.0 > > Attachments: YARN-8843.000.patch, YARN-8843.001.patch > > > When doing -updateNodeResource memory-mb=1Gi, it assigns 1MB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7054) Yarn Service Phase 2
[ https://issues.apache.org/jira/browse/YARN-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642213#comment-16642213 ] Eric Yang commented on YARN-7054: - [~pbacsko] There is a lot of intersections between this JIRA and YARN-3611. Commit changes dated back to 2014. It will not be an easy effort to back port this to Hadoop 2.x based branch. > Yarn Service Phase 2 > > > Key: YARN-7054 > URL: https://issues.apache.org/jira/browse/YARN-7054 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jian He >Assignee: Jian He >Priority: Major > Fix For: yarn-native-services > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8763) Add WebSocket logic to the Node Manager web server to establish servlet
[ https://issues.apache.org/jira/browse/YARN-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642203#comment-16642203 ] Eric Yang commented on YARN-8763: - +1 on patch 5. > Add WebSocket logic to the Node Manager web server to establish servlet > --- > > Key: YARN-8763 > URL: https://issues.apache.org/jira/browse/YARN-8763 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zian Chen >Assignee: Zian Chen >Priority: Major > Labels: Docker > Attachments: YARN-8763-001.patch, YARN-8763.002.patch, > YARN-8763.003.patch, YARN-8763.004.patch, YARN-8763.005.patch > > > The reason we want to use WebSocket servlet to serve the backend instead of > establishing the connection through HTTP is that WebSocket solves a few > issues with HTTP which needed for our scenario, > # In HTTP, the request is always initiated by the client and the response is > processed by the server — making HTTP a unidirectional protocol, while web > socket provides the Bi-directional protocol which means either client/server > can send a message to the other party. > # Full-duplex communication — client and server can talk to each other > independently at the same time > # Single TCP connection — After upgrading the HTTP connection in the > beginning, client and server communicate over that same TCP connection > throughout the lifecycle of WebSocket connection -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8855) Application fails if one of the sublcluster is down.
[
https://issues.apache.org/jira/browse/YARN-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642204#comment-16642204
]
Botong Huang commented on YARN-8855:
Thanks [~rahulanand90] for reporting it! Which federation policy
(yarn.federation.policy-manager) and code version are you using? This should
have been fixed in latest trunk and branch-2.
> Application fails if one of the sublcluster is down.
>
>
> Key: YARN-8855
> URL: https://issues.apache.org/jira/browse/YARN-8855
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rahul Anand
>Priority: Major
>
> If one of sub cluster is down then application keeps on trying multiple times
> and then it fails About 30 failover attempts found in the logs. Below is the
> detailed exception.
> {code:java}
> 2018-10-08 14:21:21,245 | INFO | NM ContainerManager dispatcher | Container
> container_e03_1538297667953_0005_01_01 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to DONE | ContainerImpl.java:2093
> 2018-10-08 14:21:21,245 | INFO | NM ContainerManager dispatcher | Removing
> container_e03_1538297667953_0005_01_01 from application
> application_1538297667953_0005 | ApplicationImpl.java:512
> 2018-10-08 14:21:21,246 | INFO | NM ContainerManager dispatcher | Stopping
> resource-monitoring for container_e03_1538297667953_0005_01_01 |
> ContainersMonitorImpl.java:932
> 2018-10-08 14:21:21,246 | INFO | NM ContainerManager dispatcher | Considering
> container container_e03_1538297667953_0005_01_01 for log-aggregation |
> AppLogAggregatorImpl.java:538
> 2018-10-08 14:21:21,246 | INFO | NM ContainerManager dispatcher | Got event
> CONTAINER_STOP for appId application_1538297667953_0005 | AuxServices.java:350
> 2018-10-08 14:21:21,246 | INFO | NM ContainerManager dispatcher | Stopping
> container container_e03_1538297667953_0005_01_01 |
> YarnShuffleService.java:295
> 2018-10-08 14:21:21,247 | WARN | NM Event dispatcher | couldn't find
> container container_e03_1538297667953_0005_01_01 while processing
> FINISH_CONTAINERS event | ContainerManagerImpl.java:1660
> 2018-10-08 14:21:22,248 | INFO | Node Status Updater | Removed completed
> containers from NM context: [container_e03_1538297667953_0005_01_01] |
> NodeStatusUpdaterImpl.java:696
> 2018-10-08 14:21:26,734 | INFO | pool-16-thread-1 | Failing over to the
> ResourceManager for SubClusterId: cluster2 |
> FederationRMFailoverProxyProvider.java:124
> 2018-10-08 14:21:26,735 | INFO | pool-16-thread-1 | Flushing subClusters from
> cache and rehydrating from store, most likely on account of RM failover. |
> FederationStateStoreFacade.java:258
> 2018-10-08 14:21:26,738 | INFO | pool-16-thread-1 | Connecting to
> /192.168.0.25:8032 subClusterId cluster2 with protocol
> ApplicationClientProtocol as user root (auth:SIMPLE) |
> FederationRMFailoverProxyProvider.java:145
> 2018-10-08 14:21:26,741 | INFO | pool-16-thread-1 |
> java.net.ConnectException: Call From node-core-jIKcN/192.168.0.64 to
> node-master1-IYTxR:8032 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
> ApplicationClientProtocolPBClientImpl.submitApplication over cluster2 after
> 28 failover attempts. Trying to failover after sleeping for 15261ms. |
> RetryInvocationHandler.java:411
> 2018-10-08 14:21:42,002 | INFO | pool-16-thread-1 | Failing over to the
> ResourceManager for SubClusterId: cluster2 |
> FederationRMFailoverProxyProvider.java:124
> 2018-10-08 14:21:42,003 | INFO | pool-16-thread-1 | Flushing subClusters from
> cache and rehydrating from store, most likely on account of RM failover. |
> FederationStateStoreFacade.java:258
> 2018-10-08 14:21:42,005 | INFO | pool-16-thread-1 | Connecting to
> /192.168.0.25:8032 subClusterId cluster2 with protocol
> ApplicationClientProtocol as user root (auth:SIMPLE) |
> FederationRMFailoverProxyProvider.java:145
> 2018-10-08 14:21:42,007 | INFO | pool-16-thread-1 |
> java.net.ConnectException: Call From node-core-jIKcN/192.168.0.64 to
> node-master1-IYTxR:8032 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
> ApplicationClientProtocolPBClientImpl.submitApplication over cluster2 after
> 29 failover attempts. Trying to failover after sleeping for 21175ms. |
> RetryInvocationHandler.java:411
> 2018-10-08 14:22:03,183 | INFO | pool-16-thread-1 | Failing over to the
> ResourceManager for SubClusterId: cluster2 |
> FederationRMFailoverProxyProvider.java:124
> 2018-10-08 14:22:03,183 | INFO | pool-16-thread-1 | Flushing subClusters from
> cache and rehydrating from store, most likely on account of RM
[jira] [Commented] (YARN-6900) ZooKeeper based implementation of the FederationStateStore
[
https://issues.apache.org/jira/browse/YARN-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642188#comment-16642188
]
Íñigo Goiri commented on YARN-6900:
---
Thanks [~rahulanand90] for the comment.
Actually your question is the reason why I created a separate policy to send to
a local cluster.
Currently, it is not very easy to add policies as they use PB.
[~subru] do you guys have any easy interface to set the policies, etc?
Otherwise we should add it to something like {{yarn routeradmin}}.
> ZooKeeper based implementation of the FederationStateStore
> --
>
> Key: YARN-6900
> URL: https://issues.apache.org/jira/browse/YARN-6900
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation, nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Íñigo Goiri
>Priority: Major
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6900-002.patch, YARN-6900-003.patch,
> YARN-6900-004.patch, YARN-6900-005.patch, YARN-6900-006.patch,
> YARN-6900-007.patch, YARN-6900-008.patch, YARN-6900-009.patch,
> YARN-6900-010.patch, YARN-6900-011.patch, YARN-6900-YARN-2915-000.patch,
> YARN-6900-YARN-2915-001.patch
>
>
> YARN-5408 defines the unified {{FederationStateStore}} API. Currently we only
> support SQL based stores, this JIRA tracks adding a ZooKeeper based
> implementation for simplifying deployment as it's already popularly used for
> {{RMStateStore}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7652) Handle AM register requests asynchronously in FederationInterceptor
[
https://issues.apache.org/jira/browse/YARN-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642179#comment-16642179
]
Íñigo Goiri commented on YARN-7652:
---
[^YARN-7652.v2.patch] LGTM.
+1
> Handle AM register requests asynchronously in FederationInterceptor
> ---
>
> Key: YARN-7652
> URL: https://issues.apache.org/jira/browse/YARN-7652
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: amrmproxy, federation
>Affects Versions: 2.9.0, 3.0.0
>Reporter: Subru Krishnan
>Assignee: Botong Huang
>Priority: Major
> Attachments: YARN-7652.v1.patch, YARN-7652.v2.patch
>
>
> We (cc [~goiri]/[~botong]) observed that the {{FederationInterceptor}} in
> {{AMRMProxy}} (and consequently the AM) is blocked if the _StateStore_ has
> outdated info about a _SubCluster_. This is because we handle AM register
> requests synchronously. This jira proposes to move to async similar to how we
> operate with allocate invocations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8843) updateNodeResource does not support units for memory
[ https://issues.apache.org/jira/browse/YARN-8843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642170#comment-16642170 ] Sunil Govindan commented on YARN-8843: -- Yes. Lets get this in for 3.2 as well. > updateNodeResource does not support units for memory > > > Key: YARN-8843 > URL: https://issues.apache.org/jira/browse/YARN-8843 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Manikandan R >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-8843.000.patch, YARN-8843.001.patch > > > When doing -updateNodeResource memory-mb=1Gi, it assigns 1MB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8843) updateNodeResource does not support units for memory
[ https://issues.apache.org/jira/browse/YARN-8843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642168#comment-16642168 ] Hudson commented on YARN-8843: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #15140 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15140/]) YARN-8843. updateNodeResource does not support units for memory. (inigoiri: rev 745f64012a2a912d5f0a36bbda89dc638e1715cb) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestRMAdminCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RMAdminCLI.java > updateNodeResource does not support units for memory > > > Key: YARN-8843 > URL: https://issues.apache.org/jira/browse/YARN-8843 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Manikandan R >Priority: Minor > Fix For: 3.3.0 > > Attachments: YARN-8843.000.patch, YARN-8843.001.patch > > > When doing -updateNodeResource memory-mb=1Gi, it assigns 1MB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[ https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642167#comment-16642167 ] Hudson commented on YARN-8659: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15139 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15139/]) YARN-8659. RMWebServices returns only RUNNING apps when filtered with (haibochen: rev 7c13872cbbb6f1b0b1c2dde894885b41186b3797) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java > RMWebServices returns only RUNNING apps when filtered with queue > > > Key: YARN-8659 > URL: https://issues.apache.org/jira/browse/YARN-8659 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.3.0 > > Attachments: Screen Shot 2018-08-13 at 8.01.29 PM.png, Screen Shot > 2018-08-13 at 8.01.52 PM.png, YARN-8659.001.patch, YARN-8659.002.patch, > YARN-8659.003.patch, YARN-8659.004.patch, YARN-8659.005.patch > > > RMWebServices returns only RUNNING apps when filtered with queue and returns > empty apps > when filtered with both FINISHED states and queue. > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?queue=default > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?states=FINISHED=default -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8843) updateNodeResource does not support units for memory
[ https://issues.apache.org/jira/browse/YARN-8843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642154#comment-16642154 ] Íñigo Goiri commented on YARN-8843: --- Assigned it to [~maniraj...@gmail.com], committing to trunk and branch-3.2. Do you guys want this anywhere else? > updateNodeResource does not support units for memory > > > Key: YARN-8843 > URL: https://issues.apache.org/jira/browse/YARN-8843 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Manikandan R >Priority: Minor > Attachments: YARN-8843.000.patch, YARN-8843.001.patch > > > When doing -updateNodeResource memory-mb=1Gi, it assigns 1MB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8843) updateNodeResource does not support units for memory
[ https://issues.apache.org/jira/browse/YARN-8843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri reassigned YARN-8843: - Assignee: Manikandan R (was: Íñigo Goiri) > updateNodeResource does not support units for memory > > > Key: YARN-8843 > URL: https://issues.apache.org/jira/browse/YARN-8843 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Manikandan R >Priority: Minor > Attachments: YARN-8843.000.patch, YARN-8843.001.patch > > > When doing -updateNodeResource memory-mb=1Gi, it assigns 1MB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8827) Plumb per app, per user and per queue resource utilization from the NM to RM
[
https://issues.apache.org/jira/browse/YARN-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642148#comment-16642148
]
Íñigo Goiri commented on YARN-8827:
---
Thanks [~asuresh] for the patch.
I think we can fix the findbugs, checkstyles and so on from Yetus.
Other minor comments:
* I think that ContainersMonitorImpl#run is single threaded and all the
accesses to {{perAppResourceUtilization}} are made here, no need for the
thread-safe comment; nothing is here.
* In ContainersMonitorImpl, we are doing the change of units twice (#825), can
we refactor that into a proper ResourceUtilization and use it in both places?
* ContainersMonitorImpl#1106, what does it mean?
* TestResourceUtilizationAggregator#line66 fits in one single line. (There are
a few more places like #171, #174, etc).
* TestResourceUtilizationAggregator for values like 8192, you can do 8*GB as
the rest of the class.
* Can we do a more informed wait in TestResourceUtilizationAggregator instead
of sleep(1000)?
* mkmap() and e() are an OK pattern but I would probably call the second one
mkmapentry and add javadoc for the two explaining the purpose.
* Can we use an static import for {{Assert.assertEquals()}}?
* The code in {{testResourceUtilizationAggregation}} is a little repetitive.
Given that this class is only used for this, I would make the NMs part of the
class and trigger registrations and so on as part of the class. I would also
add a couple more comments throughout explaining how the apps are submitting
requests, etc. Specially the numbers after #184.
* TestResourceUtilizationAggregator#184 can it be made calculated from some
assignment we track?
> Plumb per app, per user and per queue resource utilization from the NM to RM
>
>
> Key: YARN-8827
> URL: https://issues.apache.org/jira/browse/YARN-8827
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>Priority: Major
> Attachments: YARN-8827-YARN-1011.01.patch
>
>
> Opportunistic Containers for OverAllocation need to be allocated to pending
> applications in some fair manner. Rather than evaluating queue and user
> resource usage (allocated resource usage) and comparing against queue and
> user limits to decide the allocation, it might make more sense to use a
> snapshot of actual resource utilization of the queue and user.
> To facilitate this, this JIRA proposes to aggregate per user, per app (and
> maybe per queue) resource utilization in addition to aggregated Container and
> Node Utilization and send it along with the NM heartbeat. It should be fairly
> inexpensive to aggregate - since it can be performed in the same loop of the
> {{ContainersMonitorImpl}}'s Monitoring thread.
> A snapshot aggregate can be made every couple of seconds in the RM. This
> instantaneous resource utilization should be used to decide if Opportunistic
> containers can be allocated to an App, Queue or User.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8843) updateNodeResource does not support units for memory
[ https://issues.apache.org/jira/browse/YARN-8843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642130#comment-16642130 ] Sunil Govindan commented on YARN-8843: -- Looks good to me as well. [~elgoiri] pls feel free to get it in. > updateNodeResource does not support units for memory > > > Key: YARN-8843 > URL: https://issues.apache.org/jira/browse/YARN-8843 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Attachments: YARN-8843.000.patch, YARN-8843.001.patch > > > When doing -updateNodeResource memory-mb=1Gi, it assigns 1MB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7225) Add queue and partition info to RM audit log
[
https://issues.apache.org/jira/browse/YARN-7225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642125#comment-16642125
]
Eric Payne commented on YARN-7225:
--
bq. I think we can remove these right? Since this will be checked by the
isEmpty() checks preceding it.
[~jhung], thanks for the review and comments. Good point. I'll make the changes.
I have a somewhat related question. I'm not very familiar with the Fair
Scheduler, but AFAICT, the FS doesn't seem to support labelled queues. Is that
correct? If so, we can just always use null for the partition when calling
{{logSuccess}} and {{logFailure}} in the Fair Scheduler.
> Add queue and partition info to RM audit log
>
>
> Key: YARN-7225
> URL: https://issues.apache.org/jira/browse/YARN-7225
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Jonathan Hung
>Assignee: Eric Payne
>Priority: Major
> Attachments: YARN-7225.001.patch, YARN-7225.002.patch
>
>
> Right now RM audit log has fields such as user, ip, resource, etc. Having
> queue and partition is useful for resource tracking.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[ https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642118#comment-16642118 ] Haibo Chen commented on YARN-8659: -- Thanks [~Prabhu Joseph] for the initial report and [~snemeth] for the fix! > RMWebServices returns only RUNNING apps when filtered with queue > > > Key: YARN-8659 > URL: https://issues.apache.org/jira/browse/YARN-8659 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Szilard Nemeth >Priority: Major > Fix For: 3.3.0 > > Attachments: Screen Shot 2018-08-13 at 8.01.29 PM.png, Screen Shot > 2018-08-13 at 8.01.52 PM.png, YARN-8659.001.patch, YARN-8659.002.patch, > YARN-8659.003.patch, YARN-8659.004.patch, YARN-8659.005.patch > > > RMWebServices returns only RUNNING apps when filtered with queue and returns > empty apps > when filtered with both FINISHED states and queue. > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?queue=default > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?states=FINISHED=default -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7644) NM gets backed up deleting docker containers
[
https://issues.apache.org/jira/browse/YARN-7644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642119#comment-16642119
]
Chandni Singh commented on YARN-7644:
-
[~jlowe] do you have any comments?
> NM gets backed up deleting docker containers
>
>
> Key: YARN-7644
> URL: https://issues.apache.org/jira/browse/YARN-7644
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Eric Badger
>Assignee: Chandni Singh
>Priority: Major
> Labels: Docker
> Attachments: YARN-7644.001.patch, YARN-7644.002.patch
>
>
> We are sending a {{docker stop}} to the docker container with a timeout of 10
> seconds when we shut down a container. If the container does not stop after
> 10 seconds then we force kill it. However, the {{docker stop}} command is a
> blocking call. So in cases where lots of containers don't go down with the
> initial SIGTERM, we have to wait 10+ seconds for the {{docker stop}} to
> return. This ties up the ContainerLaunch handler and so these kill events
> back up. It also appears to be backing up new container launches as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8659) RMWebServices returns only RUNNING apps when filtered with queue
[ https://issues.apache.org/jira/browse/YARN-8659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642114#comment-16642114 ] Haibo Chen commented on YARN-8659: -- +1 on the latest patch. Will commit it to trunk shortly. > RMWebServices returns only RUNNING apps when filtered with queue > > > Key: YARN-8659 > URL: https://issues.apache.org/jira/browse/YARN-8659 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Affects Versions: 2.7.3 >Reporter: Prabhu Joseph >Assignee: Szilard Nemeth >Priority: Major > Attachments: Screen Shot 2018-08-13 at 8.01.29 PM.png, Screen Shot > 2018-08-13 at 8.01.52 PM.png, YARN-8659.001.patch, YARN-8659.002.patch, > YARN-8659.003.patch, YARN-8659.004.patch, YARN-8659.005.patch > > > RMWebServices returns only RUNNING apps when filtered with queue and returns > empty apps > when filtered with both FINISHED states and queue. > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?queue=default > http://pjoseph-script-llap3.openstacklocal:8088/ws/v1/cluster/apps?states=FINISHED=default -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8843) updateNodeResource does not support units for memory
[ https://issues.apache.org/jira/browse/YARN-8843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642093#comment-16642093 ] Íñigo Goiri commented on YARN-8843: --- The fix in [^YARN-8843.001.patch] LGTM. > updateNodeResource does not support units for memory > > > Key: YARN-8843 > URL: https://issues.apache.org/jira/browse/YARN-8843 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Minor > Attachments: YARN-8843.000.patch, YARN-8843.001.patch > > > When doing -updateNodeResource memory-mb=1Gi, it assigns 1MB. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7054) Yarn Service Phase 2
[ https://issues.apache.org/jira/browse/YARN-7054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641993#comment-16641993 ] Peter Bacsko commented on YARN-7054: Hi all, we'd like to backport YARN-7605 to our internal repository. Problem is, there are a lot of merge conflicts. I don't think we have any commits from this particular JIRA. I assume that YARN-7605 depends on several other commits, mainly from other subtasks listed here. Does anyhone have a rough idea what those might be? Also, how risky it is to backport that feature alone? I'm not expecting a perfect, conflict-free cherry-pick - minimal amount of conflicts are okay. But currently it's too much. > Yarn Service Phase 2 > > > Key: YARN-7054 > URL: https://issues.apache.org/jira/browse/YARN-7054 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jian He >Assignee: Jian He >Priority: Major > Fix For: yarn-native-services > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5742) Serve aggregated logs of historical apps from timeline service
[
https://issues.apache.org/jira/browse/YARN-5742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641944#comment-16641944
]
Hadoop QA commented on YARN-5742:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
48s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 0s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 13 new
+ 1 unchanged - 0 fixed = 14 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
45s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
19s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 72m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-5742 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942822/YARN-5742.01.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 89babf44772a 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 228dc19 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
[jira] [Commented] (YARN-8852) Add documentation for submarine installation details
[
https://issues.apache.org/jira/browse/YARN-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641904#comment-16641904
]
Hadoop QA commented on YARN-8852:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
35m 50s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8852 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942806/YARN-8852.003.patch |
| Optional Tests | dupname asflicense mvnsite |
| uname | Linux 1c52de12686c 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 228dc19 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 306 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22098/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add documentation for submarine installation details
>
>
> Key: YARN-8852
> URL: https://issues.apache.org/jira/browse/YARN-8852
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zac Zhou
>Assignee: Zac Zhou
>Priority: Major
> Attachments: YARN-8852.001.patch, YARN-8852.002.patch,
> YARN-8852.003.patch
>
>
> To help the beginners to install and use the submarine, A detailed guide is
> needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8853) Application Attempts tab not shown correctly when there are no attempts
[
https://issues.apache.org/jira/browse/YARN-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641902#comment-16641902
]
Hadoop QA commented on YARN-8853:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
31m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8853 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942808/YARN-8853.001.patch |
| Optional Tests | dupname asflicense shadedclient |
| uname | Linux fea7fd705cc4 3.13.0-144-generic #193-Ubuntu SMP Thu Mar 15
17:03:53 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 228dc19 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 307 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22097/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Application Attempts tab not shown correctly when there are no attempts
>
>
> Key: YARN-8853
> URL: https://issues.apache.org/jira/browse/YARN-8853
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Charan Hebri
>Assignee: Akhil PB
>Priority: Major
> Attachments: Application_Attempts.png, YARN-8853.001.patch
>
>
> When there are no attempts registered for an application the 'Application
> Attempts' tab overlaps the Attempts List tab. Attached screenshot.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8468) Enable the use of queue based maximum container allocation limit and implement it in FairScheduler
[
https://issues.apache.org/jira/browse/YARN-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641824#comment-16641824
]
Weiwei Yang commented on YARN-8468:
---
Hi [~bsteinbach]
The UT failure \{{TestAllocationFileLoaderService}} on branch-3.1 seems to be
related to this patch, could you please take a look?
> Enable the use of queue based maximum container allocation limit and
> implement it in FairScheduler
> --
>
> Key: YARN-8468
> URL: https://issues.apache.org/jira/browse/YARN-8468
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Affects Versions: 3.1.0
>Reporter: Antal Bálint Steinbach
>Assignee: Antal Bálint Steinbach
>Priority: Critical
> Fix For: 3.2.0
>
> Attachments: YARN-8468-branch-3.1.018.patch, YARN-8468.000.patch,
> YARN-8468.001.patch, YARN-8468.002.patch, YARN-8468.003.patch,
> YARN-8468.004.patch, YARN-8468.005.patch, YARN-8468.006.patch,
> YARN-8468.007.patch, YARN-8468.008.patch, YARN-8468.009.patch,
> YARN-8468.010.patch, YARN-8468.011.patch, YARN-8468.012.patch,
> YARN-8468.013.patch, YARN-8468.014.patch, YARN-8468.015.patch,
> YARN-8468.016.patch, YARN-8468.017.patch, YARN-8468.018.patch
>
>
> When using any scheduler, you can use "yarn.scheduler.maximum-allocation-mb"
> to limit the overall size of a container. This applies globally to all
> containers and cannot be limited by queue or and is not scheduler dependent.
> The goal of this ticket is to allow this value to be set on a per queue basis.
> The use case: User has two pools, one for ad hoc jobs and one for enterprise
> apps. User wants to limit ad hoc jobs to small containers but allow
> enterprise apps to request as many resources as needed. Setting
> yarn.scheduler.maximum-allocation-mb sets a default value for maximum
> container size for all queues and setting maximum resources per queue with
> “maxContainerResources” queue config value.
> Suggested solution:
> All the infrastructure is already in the code. We need to do the following:
> * add the setting to the queue properties for all queue types (parent and
> leaf), this will cover dynamically created queues.
> * if we set it on the root we override the scheduler setting and we should
> not allow that.
> * make sure that queue resource cap can not be larger than scheduler max
> resource cap in the config.
> * implement getMaximumResourceCapability(String queueName) in the
> FairScheduler
> * implement getMaximumResourceCapability(String queueName) in both
> FSParentQueue and FSLeafQueue as follows
> * expose the setting in the queue information in the RM web UI.
> * expose the setting in the metrics etc for the queue.
> * Enforce the use of queue based maximum allocation limit if it is
> available, if not use the general scheduler level setting
> ** Use it during validation and normalization of requests in
> scheduler.allocate, app submit and resource request
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7747) YARN UI is broken in the minicluster
[
https://issues.apache.org/jira/browse/YARN-7747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641818#comment-16641818
]
Steve Loughran commented on YARN-7747:
--
bq. Steve Loughran we definitely need tests to prevent this kind of regression
in the future. We could make sure that all web/http address keys are properly
reflected in MiniYARNCLuster#getConfig implementation and then probe all of
them through easy-to-validate REST api. RM URI should respond to the
RM-specific REST, and so on and so forth
please. I've just been trying to this with the RM ports in a
kerberized-cluster, and things didn't work out right w.r.t port bindings,
kerberos principals and locahost vs hostname registration.
Maybe the probes should go the MiniYARNCluster itself, some
miniYarnCluster.get("/") call, so you could do things in tests like
eventually() calls waiting for it to come up, etc.
> YARN UI is broken in the minicluster
> -
>
> Key: YARN-7747
> URL: https://issues.apache.org/jira/browse/YARN-7747
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Gera Shegalov
>Assignee: Gera Shegalov
>Priority: Major
> Attachments: YARN-7747.001.patch, YARN-7747.002.patch
>
>
> YARN web apps use non-injected instances of GuiceFilter, i.e. instances
> created by Jetty as opposed by Guice itself. This triggers the [call
> path|https://github.com/google/guice/blob/master/extensions/servlet/src/com/google/inject/servlet/GuiceFilter.java#L251]
> where the static field {{pipeline}} is used instead of the instance field
> {{injectedPipeline}}. However, besides GuiceFilter instances created by
> Jetty, each Guice module generates them as well. On the injection call path
> this static variable is updated by each instance. Thus if there are multiple
> modules as it happens to be the case in the minicluster the one loaded last
> ends up defining the filter pipeline for all Jetty instances. In the
> minicluster case this is the nodemanager UI
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7592) yarn.federation.failover.enabled missing in yarn-default.xml
[
https://issues.apache.org/jira/browse/YARN-7592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641793#comment-16641793
]
Bibin A Chundatt commented on YARN-7592:
+1 for this change.
[~subru] thoughts??
> yarn.federation.failover.enabled missing in yarn-default.xml
>
>
> Key: YARN-7592
> URL: https://issues.apache.org/jira/browse/YARN-7592
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.0.0-beta1
>Reporter: Gera Shegalov
>Priority: Major
> Attachments: IssueReproduce.patch
>
>
> yarn.federation.failover.enabled should be documented in yarn-default.xml. I
> am also not sure why it should be true by default and force the HA retry
> policy in {{RMProxy#createRMProxy}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8855) Application fails if one of the sublcluster is down.
[
https://issues.apache.org/jira/browse/YARN-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rahul Anand updated YARN-8855:
--
Summary: Application fails if one of the sublcluster is down. (was:
Application submission fails if one of the sublcluster is down.)
> Application fails if one of the sublcluster is down.
>
>
> Key: YARN-8855
> URL: https://issues.apache.org/jira/browse/YARN-8855
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Rahul Anand
>Priority: Major
>
> If one of sub cluster is down then application keeps on trying multiple times
> and then it fails About 30 failover attempts found in the logs. Below is the
> detailed exception.
> {code:java}
> 2018-10-08 14:21:21,245 | INFO | NM ContainerManager dispatcher | Container
> container_e03_1538297667953_0005_01_01 transitioned from
> CONTAINER_CLEANEDUP_AFTER_KILL to DONE | ContainerImpl.java:2093
> 2018-10-08 14:21:21,245 | INFO | NM ContainerManager dispatcher | Removing
> container_e03_1538297667953_0005_01_01 from application
> application_1538297667953_0005 | ApplicationImpl.java:512
> 2018-10-08 14:21:21,246 | INFO | NM ContainerManager dispatcher | Stopping
> resource-monitoring for container_e03_1538297667953_0005_01_01 |
> ContainersMonitorImpl.java:932
> 2018-10-08 14:21:21,246 | INFO | NM ContainerManager dispatcher | Considering
> container container_e03_1538297667953_0005_01_01 for log-aggregation |
> AppLogAggregatorImpl.java:538
> 2018-10-08 14:21:21,246 | INFO | NM ContainerManager dispatcher | Got event
> CONTAINER_STOP for appId application_1538297667953_0005 | AuxServices.java:350
> 2018-10-08 14:21:21,246 | INFO | NM ContainerManager dispatcher | Stopping
> container container_e03_1538297667953_0005_01_01 |
> YarnShuffleService.java:295
> 2018-10-08 14:21:21,247 | WARN | NM Event dispatcher | couldn't find
> container container_e03_1538297667953_0005_01_01 while processing
> FINISH_CONTAINERS event | ContainerManagerImpl.java:1660
> 2018-10-08 14:21:22,248 | INFO | Node Status Updater | Removed completed
> containers from NM context: [container_e03_1538297667953_0005_01_01] |
> NodeStatusUpdaterImpl.java:696
> 2018-10-08 14:21:26,734 | INFO | pool-16-thread-1 | Failing over to the
> ResourceManager for SubClusterId: cluster2 |
> FederationRMFailoverProxyProvider.java:124
> 2018-10-08 14:21:26,735 | INFO | pool-16-thread-1 | Flushing subClusters from
> cache and rehydrating from store, most likely on account of RM failover. |
> FederationStateStoreFacade.java:258
> 2018-10-08 14:21:26,738 | INFO | pool-16-thread-1 | Connecting to
> /192.168.0.25:8032 subClusterId cluster2 with protocol
> ApplicationClientProtocol as user root (auth:SIMPLE) |
> FederationRMFailoverProxyProvider.java:145
> 2018-10-08 14:21:26,741 | INFO | pool-16-thread-1 |
> java.net.ConnectException: Call From node-core-jIKcN/192.168.0.64 to
> node-master1-IYTxR:8032 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
> ApplicationClientProtocolPBClientImpl.submitApplication over cluster2 after
> 28 failover attempts. Trying to failover after sleeping for 15261ms. |
> RetryInvocationHandler.java:411
> 2018-10-08 14:21:42,002 | INFO | pool-16-thread-1 | Failing over to the
> ResourceManager for SubClusterId: cluster2 |
> FederationRMFailoverProxyProvider.java:124
> 2018-10-08 14:21:42,003 | INFO | pool-16-thread-1 | Flushing subClusters from
> cache and rehydrating from store, most likely on account of RM failover. |
> FederationStateStoreFacade.java:258
> 2018-10-08 14:21:42,005 | INFO | pool-16-thread-1 | Connecting to
> /192.168.0.25:8032 subClusterId cluster2 with protocol
> ApplicationClientProtocol as user root (auth:SIMPLE) |
> FederationRMFailoverProxyProvider.java:145
> 2018-10-08 14:21:42,007 | INFO | pool-16-thread-1 |
> java.net.ConnectException: Call From node-core-jIKcN/192.168.0.64 to
> node-master1-IYTxR:8032 failed on connection exception:
> java.net.ConnectException: Connection refused; For more details see:
> http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
> ApplicationClientProtocolPBClientImpl.submitApplication over cluster2 after
> 29 failover attempts. Trying to failover after sleeping for 21175ms. |
> RetryInvocationHandler.java:411
> 2018-10-08 14:22:03,183 | INFO | pool-16-thread-1 | Failing over to the
> ResourceManager for SubClusterId: cluster2 |
> FederationRMFailoverProxyProvider.java:124
> 2018-10-08 14:22:03,183 | INFO | pool-16-thread-1 | Flushing subClusters from
> cache and rehydrating from store, most likely on account of RM failover. |
> FederationStateStoreFacade.java:258
> 2018-10-08 14:22:03,186 | INFO |
[jira] [Created] (YARN-8855) Application submission fails if one of the sublcluster is down.
Rahul Anand created YARN-8855:
-
Summary: Application submission fails if one of the sublcluster is
down.
Key: YARN-8855
URL: https://issues.apache.org/jira/browse/YARN-8855
Project: Hadoop YARN
Issue Type: Bug
Reporter: Rahul Anand
If one of sub cluster is down then application keeps on trying multiple times
and then it fails About 30 failover attempts found in the logs. Below is the
detailed exception.
{code:java}
2018-10-08 14:21:21,245 | INFO | NM ContainerManager dispatcher | Container
container_e03_1538297667953_0005_01_01 transitioned from
CONTAINER_CLEANEDUP_AFTER_KILL to DONE | ContainerImpl.java:2093
2018-10-08 14:21:21,245 | INFO | NM ContainerManager dispatcher | Removing
container_e03_1538297667953_0005_01_01 from application
application_1538297667953_0005 | ApplicationImpl.java:512
2018-10-08 14:21:21,246 | INFO | NM ContainerManager dispatcher | Stopping
resource-monitoring for container_e03_1538297667953_0005_01_01 |
ContainersMonitorImpl.java:932
2018-10-08 14:21:21,246 | INFO | NM ContainerManager dispatcher | Considering
container container_e03_1538297667953_0005_01_01 for log-aggregation |
AppLogAggregatorImpl.java:538
2018-10-08 14:21:21,246 | INFO | NM ContainerManager dispatcher | Got event
CONTAINER_STOP for appId application_1538297667953_0005 | AuxServices.java:350
2018-10-08 14:21:21,246 | INFO | NM ContainerManager dispatcher | Stopping
container container_e03_1538297667953_0005_01_01 |
YarnShuffleService.java:295
2018-10-08 14:21:21,247 | WARN | NM Event dispatcher | couldn't find container
container_e03_1538297667953_0005_01_01 while processing FINISH_CONTAINERS
event | ContainerManagerImpl.java:1660
2018-10-08 14:21:22,248 | INFO | Node Status Updater | Removed completed
containers from NM context: [container_e03_1538297667953_0005_01_01] |
NodeStatusUpdaterImpl.java:696
2018-10-08 14:21:26,734 | INFO | pool-16-thread-1 | Failing over to the
ResourceManager for SubClusterId: cluster2 |
FederationRMFailoverProxyProvider.java:124
2018-10-08 14:21:26,735 | INFO | pool-16-thread-1 | Flushing subClusters from
cache and rehydrating from store, most likely on account of RM failover. |
FederationStateStoreFacade.java:258
2018-10-08 14:21:26,738 | INFO | pool-16-thread-1 | Connecting to
/192.168.0.25:8032 subClusterId cluster2 with protocol
ApplicationClientProtocol as user root (auth:SIMPLE) |
FederationRMFailoverProxyProvider.java:145
2018-10-08 14:21:26,741 | INFO | pool-16-thread-1 | java.net.ConnectException:
Call From node-core-jIKcN/192.168.0.64 to node-master1-IYTxR:8032 failed on
connection exception: java.net.ConnectException: Connection refused; For more
details see: http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
ApplicationClientProtocolPBClientImpl.submitApplication over cluster2 after 28
failover attempts. Trying to failover after sleeping for 15261ms. |
RetryInvocationHandler.java:411
2018-10-08 14:21:42,002 | INFO | pool-16-thread-1 | Failing over to the
ResourceManager for SubClusterId: cluster2 |
FederationRMFailoverProxyProvider.java:124
2018-10-08 14:21:42,003 | INFO | pool-16-thread-1 | Flushing subClusters from
cache and rehydrating from store, most likely on account of RM failover. |
FederationStateStoreFacade.java:258
2018-10-08 14:21:42,005 | INFO | pool-16-thread-1 | Connecting to
/192.168.0.25:8032 subClusterId cluster2 with protocol
ApplicationClientProtocol as user root (auth:SIMPLE) |
FederationRMFailoverProxyProvider.java:145
2018-10-08 14:21:42,007 | INFO | pool-16-thread-1 | java.net.ConnectException:
Call From node-core-jIKcN/192.168.0.64 to node-master1-IYTxR:8032 failed on
connection exception: java.net.ConnectException: Connection refused; For more
details see: http://wiki.apache.org/hadoop/ConnectionRefused, while invoking
ApplicationClientProtocolPBClientImpl.submitApplication over cluster2 after 29
failover attempts. Trying to failover after sleeping for 21175ms. |
RetryInvocationHandler.java:411
2018-10-08 14:22:03,183 | INFO | pool-16-thread-1 | Failing over to the
ResourceManager for SubClusterId: cluster2 |
FederationRMFailoverProxyProvider.java:124
2018-10-08 14:22:03,183 | INFO | pool-16-thread-1 | Flushing subClusters from
cache and rehydrating from store, most likely on account of RM failover. |
FederationStateStoreFacade.java:258
2018-10-08 14:22:03,186 | INFO | pool-16-thread-1 | Connecting to
/192.168.0.25:8032 subClusterId cluster2 with protocol
ApplicationClientProtocol as user root (auth:SIMPLE) |
FederationRMFailoverProxyProvider.java:145
2018-10-08 14:22:03,189 | ERROR | pool-16-thread-1 | Failed to register
application master: cluster2 Application: appattempt_1538297667953_0005_01
| FederationInterceptor.java:1106
java.net.ConnectException: Call From node-core-jIKcN/192.168.0.64 to
node-master1-IYTxR:8032
[jira] [Commented] (YARN-5742) Serve aggregated logs of historical apps from timeline service
[ https://issues.apache.org/jira/browse/YARN-5742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641677#comment-16641677 ] Rohith Sharma K S commented on YARN-5742: - Updated the patch by adding basic test for newly added code. > Serve aggregated logs of historical apps from timeline service > -- > > Key: YARN-5742 > URL: https://issues.apache.org/jira/browse/YARN-5742 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Rohith Sharma K S >Priority: Critical > Attachments: YARN-5742-POC-v0.patch, YARN-5742.01.patch, > YARN-5742.v0.patch > > > ATSv1.5 daemon has servlet to serve aggregated logs. But enabling only ATSv2, > does not serve logs from CLI and UI for completed application. Log serving > story has completely broken in ATSv2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5742) Serve aggregated logs of historical apps from timeline service
[ https://issues.apache.org/jira/browse/YARN-5742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-5742: Attachment: YARN-5742.01.patch > Serve aggregated logs of historical apps from timeline service > -- > > Key: YARN-5742 > URL: https://issues.apache.org/jira/browse/YARN-5742 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Rohith Sharma K S >Priority: Critical > Attachments: YARN-5742-POC-v0.patch, YARN-5742.01.patch, > YARN-5742.v0.patch > > > ATSv1.5 daemon has servlet to serve aggregated logs. But enabling only ATSv2, > does not serve logs from CLI and UI for completed application. Log serving > story has completely broken in ATSv2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8854) [Hadoop YARN Common] Update JQuery version references
[ https://issues.apache.org/jira/browse/YARN-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akhil PB updated YARN-8854: --- Summary: [Hadoop YARN Common] Update JQuery version references (was: [Hadoop Yarn Common] Update JQuery version references) > [Hadoop YARN Common] Update JQuery version references > - > > Key: YARN-8854 > URL: https://issues.apache.org/jira/browse/YARN-8854 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Critical > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8854) [Hadoop Yarn Common] Update JQuery version references
[ https://issues.apache.org/jira/browse/YARN-8854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akhil PB updated YARN-8854: --- Priority: Critical (was: Major) > [Hadoop Yarn Common] Update JQuery version references > - > > Key: YARN-8854 > URL: https://issues.apache.org/jira/browse/YARN-8854 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Critical > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8854) [Hadoop Yarn Common] Update JQuery version references
Akhil PB created YARN-8854: -- Summary: [Hadoop Yarn Common] Update JQuery version references Key: YARN-8854 URL: https://issues.apache.org/jira/browse/YARN-8854 Project: Hadoop YARN Issue Type: Bug Reporter: Akhil PB Assignee: Akhil PB -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8853) Application Attempts tab not shown correctly when there are no attempts
[ https://issues.apache.org/jira/browse/YARN-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641587#comment-16641587 ] Akhil PB commented on YARN-8853: [~sunilg] Please review the patch. > Application Attempts tab not shown correctly when there are no attempts > > > Key: YARN-8853 > URL: https://issues.apache.org/jira/browse/YARN-8853 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Akhil PB >Priority: Major > Attachments: Application_Attempts.png, YARN-8853.001.patch > > > When there are no attempts registered for an application the 'Application > Attempts' tab overlaps the Attempts List tab. Attached screenshot. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8853) Application Attempts tab not shown correctly when there are no attempts
[ https://issues.apache.org/jira/browse/YARN-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akhil PB reassigned YARN-8853: -- Assignee: Akhil PB > Application Attempts tab not shown correctly when there are no attempts > > > Key: YARN-8853 > URL: https://issues.apache.org/jira/browse/YARN-8853 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Assignee: Akhil PB >Priority: Major > Attachments: Application_Attempts.png > > > When there are no attempts registered for an application the 'Application > Attempts' tab overlaps the Attempts List tab. Attached screenshot. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8853) Application Attempts tab not shown correctly when there are no attempts
[ https://issues.apache.org/jira/browse/YARN-8853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charan Hebri updated YARN-8853: --- Attachment: Application_Attempts.png > Application Attempts tab not shown correctly when there are no attempts > > > Key: YARN-8853 > URL: https://issues.apache.org/jira/browse/YARN-8853 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Charan Hebri >Priority: Major > Attachments: Application_Attempts.png > > > When there are no attempts registered for an application the 'Application > Attempts' tab overlaps the Attempts List tab. Attached screenshot. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8853) Application Attempts tab not shown correctly when there are no attempts
Charan Hebri created YARN-8853: -- Summary: Application Attempts tab not shown correctly when there are no attempts Key: YARN-8853 URL: https://issues.apache.org/jira/browse/YARN-8853 Project: Hadoop YARN Issue Type: Bug Reporter: Charan Hebri When there are no attempts registered for an application the 'Application Attempts' tab overlaps the Attempts List tab. Attached screenshot. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8852) Add documentation for submarine installation details
[ https://issues.apache.org/jira/browse/YARN-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641577#comment-16641577 ] Zac Zhou commented on YARN-8852: Fix whitespace errors > Add documentation for submarine installation details > > > Key: YARN-8852 > URL: https://issues.apache.org/jira/browse/YARN-8852 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zac Zhou >Assignee: Zac Zhou >Priority: Major > Attachments: YARN-8852.001.patch, YARN-8852.002.patch, > YARN-8852.003.patch > > > To help the beginners to install and use the submarine, A detailed guide is > needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8852) Add documentation for submarine installation details
[ https://issues.apache.org/jira/browse/YARN-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zac Zhou updated YARN-8852: --- Attachment: YARN-8852.003.patch > Add documentation for submarine installation details > > > Key: YARN-8852 > URL: https://issues.apache.org/jira/browse/YARN-8852 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zac Zhou >Assignee: Zac Zhou >Priority: Major > Attachments: YARN-8852.001.patch, YARN-8852.002.patch, > YARN-8852.003.patch > > > To help the beginners to install and use the submarine, A detailed guide is > needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8843) updateNodeResource does not support units for memory
[
https://issues.apache.org/jira/browse/YARN-8843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641477#comment-16641477
]
Hadoop QA commented on YARN-8843:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 23s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 41s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 24m
55s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8843 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942771/YARN-8843.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 99b318c0d6d2 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 228dc19 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22095/testReport/ |
| Max. process+thread count | 678 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22095/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> updateNodeResource does not support units for memory
>
>
> Key:
[jira] [Commented] (YARN-8852) Add documentation for submarine installation details
[
https://issues.apache.org/jira/browse/YARN-8852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641451#comment-16641451
]
Hadoop QA commented on YARN-8852:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
31m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 29 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 4 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:4b8c2b1 |
| JIRA Issue | YARN-8852 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12942786/YARN-8852.002.patch |
| Optional Tests | dupname asflicense mvnsite |
| uname | Linux e427e84452a3 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 228dc19 |
| maven | version: Apache Maven 3.3.9 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/22094/artifact/out/whitespace-eol.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/22094/artifact/out/whitespace-tabs.txt
|
| Max. process+thread count | 295 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22094/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add documentation for submarine installation details
>
>
> Key: YARN-8852
> URL: https://issues.apache.org/jira/browse/YARN-8852
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zac Zhou
>Assignee: Zac Zhou
>Priority: Major
> Attachments: YARN-8852.001.patch, YARN-8852.002.patch
>
>
> To help the beginners to install and use the submarine, A detailed guide is
> needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
