[jira] [Commented] (YARN-7966) Remove AllocationConfiguration#getQueueAcl and related unit test
[ https://issues.apache.org/jira/browse/YARN-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440395#comment-16440395 ] Sen Zhao commented on YARN-7966: I just submit a patch. Please review it. Thanks, [~yufeigu]. > Remove AllocationConfiguration#getQueueAcl and related unit test > > > Key: YARN-7966 > URL: https://issues.apache.org/jira/browse/YARN-7966 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Sen Zhao >Priority: Minor > Labels: newbie++ > Attachments: YARN-7966.001.patch > > > AllocationConfiguration#getQueueAcl isn't needed any more after YARN-4997. We > should remove it and its related unit test. All its logic is reimplemented in > AllocationFileLoaderService#getDefaultPermissions. class > AllocationConfiguration doesn't have any API annotation, it is considered as > private. So it is OK to remove its public method. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7966) Remove AllocationConfiguration#getQueueAcl and related unit test
[ https://issues.apache.org/jira/browse/YARN-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sen Zhao updated YARN-7966: --- Attachment: YARN-7966.001.patch > Remove AllocationConfiguration#getQueueAcl and related unit test > > > Key: YARN-7966 > URL: https://issues.apache.org/jira/browse/YARN-7966 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Priority: Minor > Labels: newbie++ > Attachments: YARN-7966.001.patch > > > AllocationConfiguration#getQueueAcl isn't needed any more after YARN-4997. We > should remove it and its related unit test. All its logic is reimplemented in > AllocationFileLoaderService#getDefaultPermissions. class > AllocationConfiguration doesn't have any API annotation, it is considered as > private. So it is OK to remove its public method. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7966) Remove AllocationConfiguration#getQueueAcl and related unit test
[ https://issues.apache.org/jira/browse/YARN-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sen Zhao reassigned YARN-7966: -- Assignee: Sen Zhao > Remove AllocationConfiguration#getQueueAcl and related unit test > > > Key: YARN-7966 > URL: https://issues.apache.org/jira/browse/YARN-7966 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Yufei Gu >Assignee: Sen Zhao >Priority: Minor > Labels: newbie++ > Attachments: YARN-7966.001.patch > > > AllocationConfiguration#getQueueAcl isn't needed any more after YARN-4997. We > should remove it and its related unit test. All its logic is reimplemented in > AllocationFileLoaderService#getDefaultPermissions. class > AllocationConfiguration doesn't have any API annotation, it is considered as > private. So it is OK to remove its public method. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8060) Create default readiness check for service components
[ https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8060: Fix Version/s: 3.1.1 3.2.0 > Create default readiness check for service components > - > > Key: YARN-8060 > URL: https://issues.apache.org/jira/browse/YARN-8060 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Major > Fix For: 3.2.0, 3.1.1 > > Attachments: YARN-8060.1.patch, YARN-8060.2.patch, YARN-8060.3.patch, > YARN-8060.4.patch, YARN-8060.5.patch, YARN-8060.6.patch > > > It is currently possible for a component instance to have READY status before > the AM retrieves an IP for the container. We should make sure the IP has been > retrieved before marking the instance as READY. > This default probe could also have an option to check for a DNS entry for the > instance's hostname if a DNS address is provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8041) Federation: Implement multiple interfaces(14 interfaces), routing REST invocations transparently to multiple RMs
[
https://issues.apache.org/jira/browse/YARN-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yiran Wu updated YARN-8041:
---
Description:
Implement routing
getAppStatistics/getAppState/getNodeToLabels/getLabelsOnNode/updateApplicationPriority/getAppQueue/updateAppQueue/getAppTimeout/getAppTimeouts/updateApplicationTimeout/getAppAttempts/getAppAttempt/getContainers/getContainer
REST invocations transparently to multiple RMs
was:
Implement routing
getAppStatistics/getAppState/getNodeToLabels/getLabelsOnNode/updateApplicationPriority/getAppQueue/updateAppQueue/getAppTimeout/getAppTimeouts/updateApplicationTimeout/getAppAttempts/getAppAttempt/getContainers/getContainer
REST invocations transparently to multiple RMs
I think we need add a new Web Protocol for Router, like is
{code:java}
public interface RouterWebServiceProtocol extends RMWebServiceProtocol {
List getAllSubClusterInfo();
ClusterInfo getSubClusterInfo(clusterId);
SchedulerInfoType getSchedulerInfo(subClusterId);
}
{code}
cause the Router needed some protocol, such is getAllSubClusterInfo():
List 、 getSubClusterInfo(clusterId): ClusterInfo
、getSchedulerInfo(subClusterId): SchedulerInfo 。
> Federation: Implement multiple interfaces(14 interfaces), routing REST
> invocations transparently to multiple RMs
> -
>
> Key: YARN-8041
> URL: https://issues.apache.org/jira/browse/YARN-8041
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 2.9.0, 3.0.0
>Reporter: Yiran Wu
>Priority: Major
> Labels: patch
> Attachments: YARN-8041.001.patch, YARN-8041.002.patch,
> YARN-8041.003.patch
>
>
> Implement routing
> getAppStatistics/getAppState/getNodeToLabels/getLabelsOnNode/updateApplicationPriority/getAppQueue/updateAppQueue/getAppTimeout/getAppTimeouts/updateApplicationTimeout/getAppAttempts/getAppAttempt/getContainers/getContainer
> REST invocations transparently to multiple RMs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8095) Allow disable non-exclusive allocation
[
https://issues.apache.org/jira/browse/YARN-8095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440283#comment-16440283
]
kyungwan nam commented on YARN-8095:
I attached a patch, which respect the accessible-node-lables even though
non-exclusive allocation.
when the capacity-scheduler configs are as follows
{code:java}
yarn.scheduler.capacity.root.batch.accessible-node-labels=label1
yarn.scheduler.capacity.root.longlived.accessible-node-labels= {code}
I’ve seen that apps submitted to root.longlived does not allocated on ‘label1’
partition.
> Allow disable non-exclusive allocation
> --
>
> Key: YARN-8095
> URL: https://issues.apache.org/jira/browse/YARN-8095
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
>Affects Versions: 2.8.3
>Reporter: kyungwan nam
>Priority: Major
> Attachments: YARN-8095-branch-2.8.001.patch
>
>
> We have 'longlived' Queue, which is used for long-lived apps.
> In situation where default Partition resources are not enough, containers for
> long-lived app can be allocated to sharable Partition.
> Since then, containers for long-lived app can be easily preempted.
> We don’t want long-lived apps to be killed abruptly.
> Currently, non-exclusive allocation can happen regardless of whether the
> queue is accessible to the sharable Partition.
> It would be good if non-exclusive allocation can be disabled at queue level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8095) Allow disable non-exclusive allocation
[ https://issues.apache.org/jira/browse/YARN-8095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kyungwan nam updated YARN-8095: --- Attachment: YARN-8095-branch-2.8.001.patch > Allow disable non-exclusive allocation > -- > > Key: YARN-8095 > URL: https://issues.apache.org/jira/browse/YARN-8095 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Affects Versions: 2.8.3 >Reporter: kyungwan nam >Priority: Major > Attachments: YARN-8095-branch-2.8.001.patch > > > We have 'longlived' Queue, which is used for long-lived apps. > In situation where default Partition resources are not enough, containers for > long-lived app can be allocated to sharable Partition. > Since then, containers for long-lived app can be easily preempted. > We don’t want long-lived apps to be killed abruptly. > Currently, non-exclusive allocation can happen regardless of whether the > queue is accessible to the sharable Partition. > It would be good if non-exclusive allocation can be disabled at queue level. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7088) Fix application start time and add submit time to UIs
[ https://issues.apache.org/jira/browse/YARN-7088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440278#comment-16440278 ] Haibo Chen commented on YARN-7088: -- Will check it in tomorrow if there is no objection. > Fix application start time and add submit time to UIs > - > > Key: YARN-7088 > URL: https://issues.apache.org/jira/browse/YARN-7088 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Abdullah Yousufi >Assignee: Kanwaljeet Sachdev >Priority: Major > Attachments: YARN-7088.001.patch, YARN-7088.002.patch, > YARN-7088.003.patch, YARN-7088.004.patch, YARN-7088.005.patch, > YARN-7088.006.patch, YARN-7088.007.patch, YARN-7088.008.patch, > YARN-7088.009.patch, YARN-7088.010.patch, YARN-7088.011.patch, > YARN-7088.012.patch, YARN-7088.013.patch, YARN-7088.014.patch, > YARN-7088.015.patch, YARN-7088.016.patch, YARN-7088.017.patch > > > Currently, the start time in the old and new UI actually shows the app > submission time. There should actually be two different fields; one for the > app's submission and one for its start, as well as the elapsed pending time > between the two. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8165) Incorrect queue name logging in AbstractContainerAllocator
[
https://issues.apache.org/jira/browse/YARN-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440267#comment-16440267
]
Weiwei Yang commented on YARN-8165:
---
Thanks [~elgoiri], [~giovanni.fumarola].
> Incorrect queue name logging in AbstractContainerAllocator
> --
>
> Key: YARN-8165
> URL: https://issues.apache.org/jira/browse/YARN-8165
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 2.9.0, 3.0.0
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Trivial
> Fix For: 2.10.0, 3.2.0
>
> Attachments: YARN-8165.001.patch
>
>
> Found some incorrect logging messages in RM log
> {noformat}
> INFO Reserved container application=application_1523849397637_0006
> resource=
> queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@38224385
> cluster=
> ...
> INFO AbstractContainerAllocator:131 - assignedContainer application
> attempt=appattempt_1523849397637_0006_01 container=null
> queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@38224385
> clusterResource= type=OFF_SWITCH
> requestedPartition=
> {noformat}
> Note, value for queue is incorrect.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7654) Support ENTRY_POINT for docker container
[
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440218#comment-16440218
]
genericqa commented on YARN-7654:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
29s{color} | {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} hadoop-yarn-services-core in the patch failed. {color}
|
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 9s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 110 unchanged - 0 fixed = 113 total (was 110) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
30s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
29s{color} | {color:red} hadoop-yarn-services-core in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
27s{color} | {color:red} hadoop-yarn-services-core in the patch failed. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
43s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 30s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 30s{color}
| {color:red} hadoop-yarn-services-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
|
[jira] [Commented] (YARN-8108) RM metrics rest API throws GSSException in kerberized environment
[
https://issues.apache.org/jira/browse/YARN-8108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440210#comment-16440210
]
Eric Yang commented on YARN-8108:
-
Here is the stack trace to better illustrate the problem:
{code}
2018-04-16 22:56:34,208 WARN
org.apache.hadoop.security.authentication.server.AuthenticationFilter: error
at: {}
org.apache.hadoop.security.authentication.client.AuthenticationException:
GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is
a replay (34))
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:303)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.authenticate(DelegationTokenAuthenticationHandler.java:413)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:536)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:82)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:645)
at
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.doFilter(DelegationTokenAuthenticationFilter.java:304)
at
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592)
at
org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.doFilter(RMAuthenticationFilter.java:82)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1601)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: GSSException: Failure unspecified at GSS-API level (Mechanism level:
Request is a replay (34))
at
sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:856)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)
at
sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.runWithPrincipal(KerberosAuthenticationHandler.java:329)
at
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.access$000(KerberosAuthenticationHandler.java:64)
at
[jira] [Updated] (YARN-8164) Fix a potential NPE in AbstractSchedulerPlanFollower
[
https://issues.apache.org/jira/browse/YARN-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-8164:
--
Summary: Fix a potential NPE in AbstractSchedulerPlanFollower (was: Fix a
Potential NPE )
> Fix a potential NPE in AbstractSchedulerPlanFollower
>
>
> Key: YARN-8164
> URL: https://issues.apache.org/jira/browse/YARN-8164
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
>Assignee: lujie
>Priority: Major
> Attachments: YARN-8146_1.patch, YARN-8146_2.patch
>
>
> We have developed a static analysis tool
> [NPEDetector|https://github.com/lujiefsi/NPEDetector] to find some potential
> NPE. Our analysis shows that some callees may return null in corner case(e.g.
> node crash , IO exception), some of their callers have _!=null_ check but
> some do not have.
> Callee FairScheduler#getAppsInQueue can return null
> {code:java}
> public List getAppsInQueue(String queueName) {
> FSQueue queue = queueMgr.getQueue(queueName);
>if (queue == null) {
> return null;//here
> }
> }
> {code}
> it has 4 callers, three of them have null checker, one does not have. In this
> issue we post a patch which can add !=null based on existed !=null check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8168) Add support in Winutils for reporting CPU cores in all CPU groups, and aggregate kernel time, idle time and user time for all CPU groups
Xiao Liang created YARN-8168: Summary: Add support in Winutils for reporting CPU cores in all CPU groups, and aggregate kernel time, idle time and user time for all CPU groups Key: YARN-8168 URL: https://issues.apache.org/jira/browse/YARN-8168 Project: Hadoop YARN Issue Type: Bug Reporter: Xiao Liang Currently winutils can only report the CPU cores of the CPU group that it's running in, and the cpuTimeMs calculated from kernel time, idle time and user time is also for that CPU group only, which is not complete and incorrect for systems with multiple CPU groups (NUMA system for example). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4781) Support intra-queue preemption for fairness ordering policy.
[
https://issues.apache.org/jira/browse/YARN-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440199#comment-16440199
]
genericqa commented on YARN-4781:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 37s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 33s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 8 new + 61 unchanged - 0 fixed = 69 total (was 61) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 32s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}125m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-4781 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919284/YARN-4781.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e03674741f83 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 49f9aca |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/20362/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
[jira] [Commented] (YARN-8164) Fix a Potential NPE
[
https://issues.apache.org/jira/browse/YARN-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440194#comment-16440194
]
Giovanni Matteo Fumarola commented on YARN-8164:
Thanks [~xiaoheipangzi] for the patch and for the tool. I am going to try it
out.
LGTM +1.
CapacityScheduler, FifoScheduler and FairScheduler can return null for
getAppsInQueue.
[~elgoiri]. Can please you push it on trunk and branch-2?
> Fix a Potential NPE
>
>
> Key: YARN-8164
> URL: https://issues.apache.org/jira/browse/YARN-8164
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
>Assignee: lujie
>Priority: Major
> Attachments: YARN-8146_1.patch, YARN-8146_2.patch
>
>
> We have developed a static analysis tool
> [NPEDetector|https://github.com/lujiefsi/NPEDetector] to find some potential
> NPE. Our analysis shows that some callees may return null in corner case(e.g.
> node crash , IO exception), some of their callers have _!=null_ check but
> some do not have.
> Callee FairScheduler#getAppsInQueue can return null
> {code:java}
> public List getAppsInQueue(String queueName) {
> FSQueue queue = queueMgr.getQueue(queueName);
>if (queue == null) {
> return null;//here
> }
> }
> {code}
> it has 4 callers, three of them have null checker, one does not have. In this
> issue we post a patch which can add !=null based on existed !=null check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7088) Fix application start time and add submit time to UIs
[
https://issues.apache.org/jira/browse/YARN-7088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440191#comment-16440191
]
genericqa commented on YARN-7088:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 29m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
16s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 27m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 27m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 27m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 31s{color} | {color:orange} root: The patch generated 1 new + 1115 unchanged
- 2 fixed = 1116 total (was 1117) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 5m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 35s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
47s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
14s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
14s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
7s{color} | {color:green} hadoop-yarn-server-applicationhistoryservice in the
patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 19s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 6s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 44s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
|
[jira] [Updated] (YARN-8164) Fix a Potential NPE
[
https://issues.apache.org/jira/browse/YARN-8164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Íñigo Goiri updated YARN-8164:
--
Summary: Fix a Potential NPE (was: Fix an Potential NPE )
> Fix a Potential NPE
>
>
> Key: YARN-8164
> URL: https://issues.apache.org/jira/browse/YARN-8164
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-beta1
>Reporter: lujie
>Assignee: lujie
>Priority: Major
> Attachments: YARN-8146_1.patch, YARN-8146_2.patch
>
>
> We have developed a static analysis tool
> [NPEDetector|https://github.com/lujiefsi/NPEDetector] to find some potential
> NPE. Our analysis shows that some callees may return null in corner case(e.g.
> node crash , IO exception), some of their callers have _!=null_ check but
> some do not have.
> Callee FairScheduler#getAppsInQueue can return null
> {code:java}
> public List getAppsInQueue(String queueName) {
> FSQueue queue = queueMgr.getQueue(queueName);
>if (queue == null) {
> return null;//here
> }
> }
> {code}
> it has 4 callers, three of them have null checker, one does not have. In this
> issue we post a patch which can add !=null based on existed !=null check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7654) Support ENTRY_POINT for docker container
[
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440187#comment-16440187
]
genericqa commented on YARN-7654:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
32s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
16s{color} | {color:red} hadoop-yarn-services-core in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m
12s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 3m 12s{color} |
{color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 3m 12s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 0s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 110 unchanged - 0 fixed = 113 total (was 110) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
19s{color} | {color:red} hadoop-yarn-services-core in the patch failed. {color}
|
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 20s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
43s{color} | {color:red} hadoop-yarn-services-core in the patch failed. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 20s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 20s{color}
| {color:red} hadoop-yarn-services-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-7654 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919292/YARN-7654.012.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient
[jira] [Commented] (YARN-8165) Incorrect queue name logging in AbstractContainerAllocator
[
https://issues.apache.org/jira/browse/YARN-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440174#comment-16440174
]
Giovanni Matteo Fumarola commented on YARN-8165:
Thanks [~cheersyang] for the patch.
LGTM +1.
[~elgoiri] can you push it to trunk and branch-2?
The wrong log is also in branch-2.9.
> Incorrect queue name logging in AbstractContainerAllocator
> --
>
> Key: YARN-8165
> URL: https://issues.apache.org/jira/browse/YARN-8165
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 2.9.0, 3.0.0
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Trivial
> Attachments: YARN-8165.001.patch
>
>
> Found some incorrect logging messages in RM log
> {noformat}
> INFO Reserved container application=application_1523849397637_0006
> resource=
> queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@38224385
> cluster=
> ...
> INFO AbstractContainerAllocator:131 - assignedContainer application
> attempt=appattempt_1523849397637_0006_01 container=null
> queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@38224385
> clusterResource= type=OFF_SWITCH
> requestedPartition=
> {noformat}
> Note, value for queue is incorrect.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7773) YARN Federation used Mysql as state store throw exception, Unknown column 'homeSubCluster' in 'field list'
[
https://issues.apache.org/jira/browse/YARN-7773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440172#comment-16440172
]
genericqa commented on YARN-7773:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
25s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 7s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-7773 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906659/YARN-7773.001.patch |
| Optional Tests | asflicense |
| uname | Linux f38b4ee30aa4 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 827d859 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 407 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/20364/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> YARN Federation used Mysql as state store throw exception, Unknown column
> 'homeSubCluster' in 'field list'
> --
>
> Key: YARN-7773
> URL: https://issues.apache.org/jira/browse/YARN-7773
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 2.9.0, 3.0.0-alpha1, 3.0.0-alpha2, 3.0.0-beta1,
> 3.0.0-alpha4, 3.0.0-alpha3, 3.0.0
> Environment: Hadoop 3.0.0
>Reporter: Yiran Wu
>Assignee: Yiran Wu
>Priority: Blocker
> Labels: patch
> Fix For: 3.2.0
>
> Attachments: YARN-7773.001.patch
>
>
> An error occurred when YARN Federation used Mysql as state store. The reason
> I found it was because the field used to create the
> applicationsHomeSubCluster table was 'subClusterId' and the stored procedure
> used 'homeSubCluster'. I fixed this problem.
>
> submitApplication appIdapplication_1516277664083_0014 try #0 on SubCluster
> cluster1 , queue: root.bdp_federation
> [2018-01-18T23:25:29.325+08:00] [ERROR]
> store.impl.SQLFederationStateStore.logAndThrowRetriableException(FederationStateStoreUtils.java
> 158) [IPC Server handler 44 on 8050] : Unable to insert the newly generated
> application application_1516277664083_0014
> com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column
> 'homeSubCluster' in 'field list'
> at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
> at com.mysql.jdbc.Util.getInstance(Util.java:408)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:944)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858)
> at
>
[jira] [Commented] (YARN-7773) YARN Federation used Mysql as state store throw exception, Unknown column 'homeSubCluster' in 'field list'
[ https://issues.apache.org/jira/browse/YARN-7773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440161#comment-16440161 ] Íñigo Goiri commented on YARN-7773: --- thank you [~yiran]. Committed to trunk. > YARN Federation used Mysql as state store throw exception, Unknown column > 'homeSubCluster' in 'field list' > -- > > Key: YARN-7773 > URL: https://issues.apache.org/jira/browse/YARN-7773 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Affects Versions: 2.9.0, 3.0.0-alpha1, 3.0.0-alpha2, 3.0.0-beta1, > 3.0.0-alpha4, 3.0.0-alpha3, 3.0.0 > Environment: Hadoop 3.0.0 >Reporter: Yiran Wu >Assignee: Yiran Wu >Priority: Blocker > Labels: patch > Fix For: 3.2.0 > > Attachments: YARN-7773.001.patch > > > An error occurred when YARN Federation used Mysql as state store. The reason > I found it was because the field used to create the > applicationsHomeSubCluster table was 'subClusterId' and the stored procedure > used 'homeSubCluster'. I fixed this problem. > > submitApplication appIdapplication_1516277664083_0014 try #0 on SubCluster > cluster1 , queue: root.bdp_federation > [2018-01-18T23:25:29.325+08:00] [ERROR] > store.impl.SQLFederationStateStore.logAndThrowRetriableException(FederationStateStoreUtils.java > 158) [IPC Server handler 44 on 8050] : Unable to insert the newly generated > application application_1516277664083_0014 > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column > 'homeSubCluster' in 'field list' > at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) > at com.mysql.jdbc.Util.getInstance(Util.java:408) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:944) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909) > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484) > at > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858) > at > com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2079) > at > com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2013) > at > com.mysql.jdbc.PreparedStatement.executeLargeUpdate(PreparedStatement.java:5104) > at > com.mysql.jdbc.CallableStatement.executeLargeUpdate(CallableStatement.java:2418) > at com.mysql.jdbc.CallableStatement.executeUpdate(CallableStatement.java:887) > at > com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) > at > com.zaxxer.hikari.pool.HikariProxyCallableStatement.executeUpdate(HikariProxyCallableStatement.java) > at > org.apache.hadoop.yarn.server.federation.store.impl.SQLFederationStateStore.addApplicationHomeSubCluster(SQLFederationStateStore.java:547) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy31.addApplicationHomeSubCluster(Unknown Source) > at > org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade.addApplicationHomeSubCluster(FederationStateStoreFacade.java:345) > at > org.apache.hadoop.yarn.server.router.clientrm.JDFederationClientInterceptor.submitApplication(JDFederationClientInterceptor.java:334) > at > org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService.submitApplication(RouterClientRMService.java:196) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:218) > at > org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:419) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) > at
[jira] [Updated] (YARN-7654) Support ENTRY_POINT for docker container
[
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7654:
Attachment: YARN-7654.013.patch
> Support ENTRY_POINT for docker container
>
>
> Key: YARN-7654
> URL: https://issues.apache.org/jira/browse/YARN-7654
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Blocker
> Attachments: YARN-7654.001.patch, YARN-7654.002.patch,
> YARN-7654.003.patch, YARN-7654.004.patch, YARN-7654.005.patch,
> YARN-7654.006.patch, YARN-7654.007.patch, YARN-7654.008.patch,
> YARN-7654.009.patch, YARN-7654.010.patch, YARN-7654.011.patch,
> YARN-7654.012.patch, YARN-7654.013.patch
>
>
> Docker image may have ENTRY_POINT predefined, but this is not supported in
> the current implementation. It would be nice if we can detect existence of
> {{launch_command}} and base on this variable launch docker container in
> different ways:
> h3. Launch command exists
> {code}
> docker run [image]:[version]
> docker exec [container_id] [launch_command]
> {code}
> h3. Use ENTRY_POINT
> {code}
> docker run [image]:[version]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7773) YARN Federation used Mysql as state store throw exception, Unknown column 'homeSubCluster' in 'field list'
[ https://issues.apache.org/jira/browse/YARN-7773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440156#comment-16440156 ] Íñigo Goiri commented on YARN-7773: --- [~giovanni.fumarola] which branches you want this in? > YARN Federation used Mysql as state store throw exception, Unknown column > 'homeSubCluster' in 'field list' > -- > > Key: YARN-7773 > URL: https://issues.apache.org/jira/browse/YARN-7773 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Affects Versions: 2.9.0, 3.0.0-alpha1, 3.0.0-alpha2, 3.0.0-beta1, > 3.0.0-alpha4, 3.0.0-alpha3, 3.0.0 > Environment: Hadoop 3.0.0 >Reporter: Yiran Wu >Priority: Blocker > Labels: patch > Attachments: YARN-7773.001.patch > > > An error occurred when YARN Federation used Mysql as state store. The reason > I found it was because the field used to create the > applicationsHomeSubCluster table was 'subClusterId' and the stored procedure > used 'homeSubCluster'. I fixed this problem. > > submitApplication appIdapplication_1516277664083_0014 try #0 on SubCluster > cluster1 , queue: root.bdp_federation > [2018-01-18T23:25:29.325+08:00] [ERROR] > store.impl.SQLFederationStateStore.logAndThrowRetriableException(FederationStateStoreUtils.java > 158) [IPC Server handler 44 on 8050] : Unable to insert the newly generated > application application_1516277664083_0014 > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column > 'homeSubCluster' in 'field list' > at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) > at com.mysql.jdbc.Util.getInstance(Util.java:408) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:944) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909) > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484) > at > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858) > at > com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2079) > at > com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2013) > at > com.mysql.jdbc.PreparedStatement.executeLargeUpdate(PreparedStatement.java:5104) > at > com.mysql.jdbc.CallableStatement.executeLargeUpdate(CallableStatement.java:2418) > at com.mysql.jdbc.CallableStatement.executeUpdate(CallableStatement.java:887) > at > com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) > at > com.zaxxer.hikari.pool.HikariProxyCallableStatement.executeUpdate(HikariProxyCallableStatement.java) > at > org.apache.hadoop.yarn.server.federation.store.impl.SQLFederationStateStore.addApplicationHomeSubCluster(SQLFederationStateStore.java:547) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy31.addApplicationHomeSubCluster(Unknown Source) > at > org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade.addApplicationHomeSubCluster(FederationStateStoreFacade.java:345) > at > org.apache.hadoop.yarn.server.router.clientrm.JDFederationClientInterceptor.submitApplication(JDFederationClientInterceptor.java:334) > at > org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService.submitApplication(RouterClientRMService.java:196) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:218) > at > org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:419) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2076) > at
[jira] [Assigned] (YARN-7773) YARN Federation used Mysql as state store throw exception, Unknown column 'homeSubCluster' in 'field list'
[ https://issues.apache.org/jira/browse/YARN-7773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri reassigned YARN-7773: - Assignee: Yiran Wu > YARN Federation used Mysql as state store throw exception, Unknown column > 'homeSubCluster' in 'field list' > -- > > Key: YARN-7773 > URL: https://issues.apache.org/jira/browse/YARN-7773 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Affects Versions: 2.9.0, 3.0.0-alpha1, 3.0.0-alpha2, 3.0.0-beta1, > 3.0.0-alpha4, 3.0.0-alpha3, 3.0.0 > Environment: Hadoop 3.0.0 >Reporter: Yiran Wu >Assignee: Yiran Wu >Priority: Blocker > Labels: patch > Attachments: YARN-7773.001.patch > > > An error occurred when YARN Federation used Mysql as state store. The reason > I found it was because the field used to create the > applicationsHomeSubCluster table was 'subClusterId' and the stored procedure > used 'homeSubCluster'. I fixed this problem. > > submitApplication appIdapplication_1516277664083_0014 try #0 on SubCluster > cluster1 , queue: root.bdp_federation > [2018-01-18T23:25:29.325+08:00] [ERROR] > store.impl.SQLFederationStateStore.logAndThrowRetriableException(FederationStateStoreUtils.java > 158) [IPC Server handler 44 on 8050] : Unable to insert the newly generated > application application_1516277664083_0014 > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column > 'homeSubCluster' in 'field list' > at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) > at com.mysql.jdbc.Util.getInstance(Util.java:408) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:944) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909) > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484) > at > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858) > at > com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2079) > at > com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2013) > at > com.mysql.jdbc.PreparedStatement.executeLargeUpdate(PreparedStatement.java:5104) > at > com.mysql.jdbc.CallableStatement.executeLargeUpdate(CallableStatement.java:2418) > at com.mysql.jdbc.CallableStatement.executeUpdate(CallableStatement.java:887) > at > com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) > at > com.zaxxer.hikari.pool.HikariProxyCallableStatement.executeUpdate(HikariProxyCallableStatement.java) > at > org.apache.hadoop.yarn.server.federation.store.impl.SQLFederationStateStore.addApplicationHomeSubCluster(SQLFederationStateStore.java:547) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy31.addApplicationHomeSubCluster(Unknown Source) > at > org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade.addApplicationHomeSubCluster(FederationStateStoreFacade.java:345) > at > org.apache.hadoop.yarn.server.router.clientrm.JDFederationClientInterceptor.submitApplication(JDFederationClientInterceptor.java:334) > at > org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService.submitApplication(RouterClientRMService.java:196) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:218) > at > org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:419) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2076) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2072) > at
[jira] [Commented] (YARN-7773) YARN Federation used Mysql as state store throw exception, Unknown column 'homeSubCluster' in 'field list'
[ https://issues.apache.org/jira/browse/YARN-7773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440155#comment-16440155 ] Giovanni Matteo Fumarola commented on YARN-7773: Thanks [~yiran] for catching this. LGTM +1. [~elgoiri] can you push it on trunk? > YARN Federation used Mysql as state store throw exception, Unknown column > 'homeSubCluster' in 'field list' > -- > > Key: YARN-7773 > URL: https://issues.apache.org/jira/browse/YARN-7773 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Affects Versions: 2.9.0, 3.0.0-alpha1, 3.0.0-alpha2, 3.0.0-beta1, > 3.0.0-alpha4, 3.0.0-alpha3, 3.0.0 > Environment: Hadoop 3.0.0 >Reporter: Yiran Wu >Priority: Blocker > Labels: patch > Attachments: YARN-7773.001.patch > > > An error occurred when YARN Federation used Mysql as state store. The reason > I found it was because the field used to create the > applicationsHomeSubCluster table was 'subClusterId' and the stored procedure > used 'homeSubCluster'. I fixed this problem. > > submitApplication appIdapplication_1516277664083_0014 try #0 on SubCluster > cluster1 , queue: root.bdp_federation > [2018-01-18T23:25:29.325+08:00] [ERROR] > store.impl.SQLFederationStateStore.logAndThrowRetriableException(FederationStateStoreUtils.java > 158) [IPC Server handler 44 on 8050] : Unable to insert the newly generated > application application_1516277664083_0014 > com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Unknown column > 'homeSubCluster' in 'field list' > at sun.reflect.GeneratedConstructorAccessor15.newInstance(Unknown Source) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) > at com.mysql.jdbc.Util.getInstance(Util.java:408) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:944) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909) > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484) > at > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858) > at > com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2079) > at > com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2013) > at > com.mysql.jdbc.PreparedStatement.executeLargeUpdate(PreparedStatement.java:5104) > at > com.mysql.jdbc.CallableStatement.executeLargeUpdate(CallableStatement.java:2418) > at com.mysql.jdbc.CallableStatement.executeUpdate(CallableStatement.java:887) > at > com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) > at > com.zaxxer.hikari.pool.HikariProxyCallableStatement.executeUpdate(HikariProxyCallableStatement.java) > at > org.apache.hadoop.yarn.server.federation.store.impl.SQLFederationStateStore.addApplicationHomeSubCluster(SQLFederationStateStore.java:547) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) > at com.sun.proxy.$Proxy31.addApplicationHomeSubCluster(Unknown Source) > at > org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade.addApplicationHomeSubCluster(FederationStateStoreFacade.java:345) > at > org.apache.hadoop.yarn.server.router.clientrm.JDFederationClientInterceptor.submitApplication(JDFederationClientInterceptor.java:334) > at > org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService.submitApplication(RouterClientRMService.java:196) > at > org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:218) > at > org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:419) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) > at
[jira] [Commented] (YARN-7654) Support ENTRY_POINT for docker container
[
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440120#comment-16440120
]
Eric Yang commented on YARN-7654:
-
Patch 12 rebased to current trunk, and removed some unnecessary changes.
> Support ENTRY_POINT for docker container
>
>
> Key: YARN-7654
> URL: https://issues.apache.org/jira/browse/YARN-7654
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Blocker
> Attachments: YARN-7654.001.patch, YARN-7654.002.patch,
> YARN-7654.003.patch, YARN-7654.004.patch, YARN-7654.005.patch,
> YARN-7654.006.patch, YARN-7654.007.patch, YARN-7654.008.patch,
> YARN-7654.009.patch, YARN-7654.010.patch, YARN-7654.011.patch,
> YARN-7654.012.patch
>
>
> Docker image may have ENTRY_POINT predefined, but this is not supported in
> the current implementation. It would be nice if we can detect existence of
> {{launch_command}} and base on this variable launch docker container in
> different ways:
> h3. Launch command exists
> {code}
> docker run [image]:[version]
> docker exec [container_id] [launch_command]
> {code}
> h3. Use ENTRY_POINT
> {code}
> docker run [image]:[version]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7654) Support ENTRY_POINT for docker container
[
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7654:
Attachment: YARN-7654.012.patch
> Support ENTRY_POINT for docker container
>
>
> Key: YARN-7654
> URL: https://issues.apache.org/jira/browse/YARN-7654
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Blocker
> Attachments: YARN-7654.001.patch, YARN-7654.002.patch,
> YARN-7654.003.patch, YARN-7654.004.patch, YARN-7654.005.patch,
> YARN-7654.006.patch, YARN-7654.007.patch, YARN-7654.008.patch,
> YARN-7654.009.patch, YARN-7654.010.patch, YARN-7654.011.patch,
> YARN-7654.012.patch
>
>
> Docker image may have ENTRY_POINT predefined, but this is not supported in
> the current implementation. It would be nice if we can detect existence of
> {{launch_command}} and base on this variable launch docker container in
> different ways:
> h3. Launch command exists
> {code}
> docker run [image]:[version]
> docker exec [container_id] [launch_command]
> {code}
> h3. Use ENTRY_POINT
> {code}
> docker run [image]:[version]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8137) Parallelize node addition in SLS
[
https://issues.apache.org/jira/browse/YARN-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440103#comment-16440103
]
Giovanni Matteo Fumarola edited comment on YARN-8137 at 4/16/18 10:21 PM:
--
Thanks [~abmodi] for the patch.
Few comments:
1) You have this instruction {{int threadPoolSize = Math.max(poolSize, 10);}}
instead of 10 you should have {{SLSConfiguration.RUNNER_POOL_SIZE_DEFAULT;}}
{{2)executorService.awaitTermination(1, TimeUnit.HOURS);}} I think the awaiting
time is too long. How did you come up with this value?
was (Author: giovanni.fumarola):
Thanks [~abmodi] for the patch.
Few comments:
1) You have this instruction {{int threadPoolSize = Math.max(poolSize, 10);}}
instead of 10 you should have {{SLSConfiguration.RUNNER_POOL_SIZE_DEFAULT;}}
{{2) {{executorService.awaitTermination(1, TimeUnit.HOURS); I think the
awaiting time is too long. How did you come up with this value?
> Parallelize node addition in SLS
>
>
> Key: YARN-8137
> URL: https://issues.apache.org/jira/browse/YARN-8137
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-8137.001.patch
>
>
> Right now, nodes are added sequentially and it can take a long time if there
> are large number of nodes. With this change nodes will be added in parallel
> and thus reduce the node addition time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8137) Parallelize node addition in SLS
[
https://issues.apache.org/jira/browse/YARN-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440103#comment-16440103
]
Giovanni Matteo Fumarola edited comment on YARN-8137 at 4/16/18 10:20 PM:
--
Thanks [~abmodi] for the patch.
Few comments:
1) You have this instruction {{int threadPoolSize = Math.max(poolSize, 10);}}
instead of 10 you should have {{SLSConfiguration.RUNNER_POOL_SIZE_DEFAULT;}}
{{2) {{executorService.awaitTermination(1, TimeUnit.HOURS); I think the
awaiting time is too long. How did you come up with this value?
was (Author: giovanni.fumarola):
Thanks [~abmodi] for the patch.
Few comments:
1) You have this instruction {{int threadPoolSize = Math.max(poolSize, 10);}}
instead of 10 you should have {{SLSConfiguration.RUNNER_POOL_SIZE_DEFAULT;}}
{{2) }}{{executorService.awaitTermination(1, TimeUnit.HOURS);}} I think the
awaiting time is too long. How did you come up with this value?
> Parallelize node addition in SLS
>
>
> Key: YARN-8137
> URL: https://issues.apache.org/jira/browse/YARN-8137
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-8137.001.patch
>
>
> Right now, nodes are added sequentially and it can take a long time if there
> are large number of nodes. With this change nodes will be added in parallel
> and thus reduce the node addition time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8137) Parallelize node addition in SLS
[
https://issues.apache.org/jira/browse/YARN-8137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440103#comment-16440103
]
Giovanni Matteo Fumarola commented on YARN-8137:
Thanks [~abmodi] for the patch.
Few comments:
1) You have this instruction {{int threadPoolSize = Math.max(poolSize, 10);}}
instead of 10 you should have {{SLSConfiguration.RUNNER_POOL_SIZE_DEFAULT;}}
{{2) }}{{executorService.awaitTermination(1, TimeUnit.HOURS);}} I think the
awaiting time is too long. How did you come up with this value?
> Parallelize node addition in SLS
>
>
> Key: YARN-8137
> URL: https://issues.apache.org/jira/browse/YARN-8137
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Abhishek Modi
>Assignee: Abhishek Modi
>Priority: Major
> Attachments: YARN-8137.001.patch
>
>
> Right now, nodes are added sequentially and it can take a long time if there
> are large number of nodes. With this change nodes will be added in parallel
> and thus reduce the node addition time.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8134) Support specifying node resources in SLS
[ https://issues.apache.org/jira/browse/YARN-8134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440099#comment-16440099 ] Giovanni Matteo Fumarola commented on YARN-8134: Thanks [~abmodi] for the patch. LGTM +1. NIT: why do you have only one node in nodes-with-resources? > Support specifying node resources in SLS > > > Key: YARN-8134 > URL: https://issues.apache.org/jira/browse/YARN-8134 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8134.002.patch, YARN-8134.003.patch, YARN-8134.patch > > > At present, all nodes have same resources in SLS. We need to add capability > to add different resources to different nodes in SLS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8151) Yarn RM Epoch should wrap around
[ https://issues.apache.org/jira/browse/YARN-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440091#comment-16440091 ] Giovanni Matteo Fumarola commented on YARN-8151: Thanks [~youchen] for the patch. Few comments: 1) RM_EPOCH_RANGE is the same as RM_EPOCH; 2) Add better comments on nextEpoch(); 3) Refactor the instruction in newxtEpoch() as: moduleEpoch = (epoch - baseEpoch + 1) % epochRange ; return baseEpoch + moduleEpoch; This will simply the understanding of the method. 4) Add a simple unit test. > Yarn RM Epoch should wrap around > > > Key: YARN-8151 > URL: https://issues.apache.org/jira/browse/YARN-8151 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Young Chen >Assignee: Young Chen >Priority: Major > Attachments: YARN-8151.01.patch, YARN-8151.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8162) Remove Method DirectoryCollection#verifyDirUsingMkdir
[
https://issues.apache.org/jira/browse/YARN-8162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440086#comment-16440086
]
Haibo Chen commented on YARN-8162:
--
Checking this in.
> Remove Method DirectoryCollection#verifyDirUsingMkdir
> -
>
> Key: YARN-8162
> URL: https://issues.apache.org/jira/browse/YARN-8162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Major
> Attachments: YARN-8162.001.patch, YARN-8162.002.patch
>
>
> First of all, DirectoryCollection#verifyDirUsingMkdir doesn't do what it
> claims to do, it creates a file but its Java doc says it create a dir.
> Because of that, it doesn't compatible with {{ReadWriteDiskValidator}}
> because the validator will throw an exception if it isn't dir. What's more,
> it is no longer needed after HADOOP-13738 (DiskChecker should perform some
> disk IO). It duplicates what HADOOP-13738 does. We should remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7878) Docker container IP detail missing when service is in STABLE state
[
https://issues.apache.org/jira/browse/YARN-7878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yesha Vora resolved YARN-7878.
--
Resolution: Duplicate
> Docker container IP detail missing when service is in STABLE state
> --
>
> Key: YARN-7878
> URL: https://issues.apache.org/jira/browse/YARN-7878
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-native-services
>Reporter: Yesha Vora
>Priority: Critical
>
> Scenario
> 1) Launch Hbase on docker app
> 2) Validate yarn service status using cli
> {code:java}
> {"name":"hbase-app-with-docker","id":"application_1517516543573_0012","artifact":{"id":"hbase-centos","type":"DOCKER"},"lifetime":3519,"components":[{"name":"hbasemaster","dependencies":[],"artifact":{"id":"hbase-centos","type":"DOCKER"},"resource":{"cpus":1,"memory":"2048"},"state":"STABLE","configuration":{"properties":{"docker.network":"hadoop"},"env":{"HBASE_MASTER_OPTS":"-Xmx2048m
>
> -Xms1024m","HBASE_LOG_DIR":""},"files":[{"type":"XML","properties":{"hbase.zookeeper.quorum":"${CLUSTER_ZK_QUORUM}","zookeeper.znode.parent":"${SERVICE_ZK_PATH}","hbase.rootdir":"${SERVICE_HDFS_DIR}/hbase","hbase.master.hostname":"hbasemaster-0.${SERVICE_NAME}.${USER}.${DOMAIN}","hbase.master.info.port":"16010","hbase.cluster.distributed":"true"},"dest_file":"/etc/hbase/conf/hbase-site.xml"},{"type":"TEMPLATE","properties":{},"dest_file":"/etc/hadoop/conf/core-site.xml","src_file":"core-site.xml"},{"type":"TEMPLATE","properties":{},"dest_file":"/etc/hadoop/conf/hdfs-site.xml","src_file":"hdfs-site.xml"}]},"quicklinks":[],"containers":[{"id":"container_e02_1517516543573_0012_01_02","ip":"10.0.0.9","hostname":"hbasemaster-0.hbase-app-with-docker.hrt-qa.test.com","state":"READY","launch_time":1517533029963,"bare_host":"xxx","component_name":"hbasemaster-0"}],"launch_command":"sleep
> 15; /usr/hdp/current/hbase-master/bin/hbase master
> start","number_of_containers":1,"run_privileged_container":false},{"name":"regionserver","dependencies":["hbasemaster"],"artifact":{"id":"hbase-centos","type":"DOCKER"},"resource":{"cpus":1,"memory":"2048"},"state":"STABLE","configuration":{"properties":{"docker.network":"hadoop"},"env":{"HBASE_REGIONSERVER_OPTS":"-XX:CMSInitiatingOccupancyFraction=70
> -Xmx2048m
> -Xms1024m","HBASE_LOG_DIR":""},"files":[{"type":"XML","properties":{"hbase.regionserver.hostname":"${COMPONENT_INSTANCE_NAME}.${SERVICE_NAME}.${USER}.${DOMAIN}","hbase.zookeeper.quorum":"${CLUSTER_ZK_QUORUM}","zookeeper.znode.parent":"${SERVICE_ZK_PATH}","hbase.rootdir":"${SERVICE_HDFS_DIR}/hbase","hbase.master.hostname":"hbasemaster-0.${SERVICE_NAME}.${USER}.${DOMAIN}","hbase.master.info.port":"16010","hbase.cluster.distributed":"true"},"dest_file":"/etc/hbase/conf/hbase-site.xml"},{"type":"TEMPLATE","properties":{},"dest_file":"/etc/hadoop/conf/core-site.xml","src_file":"core-site.xml"},{"type":"TEMPLATE","properties":{},"dest_file":"/etc/hadoop/conf/hdfs-site.xml","src_file":"hdfs-site.xml"}]},"quicklinks":[],"containers":[{"id":"container_e02_1517516543573_0012_01_05","state":"READY","launch_time":1517533059022,"bare_host":"xxx","component_name":"regionserver-0"}],"launch_command":"sleep
> 15; /usr/hdp/current/hbase-regionserver/bin/hbase regionserver
> start","number_of_containers":1,"run_privileged_container":false},{"name":"hbaseclient","dependencies":[],"artifact":{"id":"hbase-centos","type":"DOCKER"},"resource":{"cpus":1,"memory":"1024"},"state":"STABLE","configuration":{"properties":{"docker.network":"hadoop"},"env":{"HBASE_LOG_DIR":""},"files":[{"type":"XML","properties":{"hbase.zookeeper.quorum":"${CLUSTER_ZK_QUORUM}","zookeeper.znode.parent":"${SERVICE_ZK_PATH}","hbase.rootdir":"${SERVICE_HDFS_DIR}/hbase","hbase.master.hostname":"hbasemaster-0.${SERVICE_NAME}.${USER}.${DOMAIN}","hbase.master.info.port":"16010","hbase.cluster.distributed":"true"},"dest_file":"/etc/hbase/conf/hbase-site.xml"},{"type":"TEMPLATE","properties":{},"dest_file":"/etc/hadoop/conf/core-site.xml","src_file":"core-site.xml"},{"type":"TEMPLATE","properties":{},"dest_file":"/etc/hadoop/conf/hdfs-site.xml","src_file":"hdfs-site.xml"}]},"quicklinks":[],"containers":[{"id":"container_e02_1517516543573_0012_01_03","ip":"10.0.0.8","hostname":"hbaseclient-0.hbase-app-with-docker.hrt-qa.test.com","state":"READY","launch_time":1517533029964,"bare_host":"xxx","component_name":"hbaseclient-0"}],"launch_command":"sleep
>
>
[jira] [Commented] (YARN-8162) Remove Method DirectoryCollection#verifyDirUsingMkdir
[
https://issues.apache.org/jira/browse/YARN-8162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440068#comment-16440068
]
genericqa commented on YARN-8162:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
17s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8162 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919262/YARN-8162.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 11da58ed4ac7 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2d0662c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/20361/testReport/ |
| Max. process+thread count | 341 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/20361/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT
[jira] [Commented] (YARN-8060) Create default readiness check for service components
[ https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440066#comment-16440066 ] Vinod Kumar Vavilapalli commented on YARN-8060: --- [~eyang], please set the fix-version(s) for this JIRA. > Create default readiness check for service components > - > > Key: YARN-8060 > URL: https://issues.apache.org/jira/browse/YARN-8060 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Major > Attachments: YARN-8060.1.patch, YARN-8060.2.patch, YARN-8060.3.patch, > YARN-8060.4.patch, YARN-8060.5.patch, YARN-8060.6.patch > > > It is currently possible for a component instance to have READY status before > the AM retrieves an IP for the container. We should make sure the IP has been > retrieved before marking the instance as READY. > This default probe could also have an option to check for a DNS entry for the > instance's hostname if a DNS address is provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8162) Remove Method DirectoryCollection#verifyDirUsingMkdir
[
https://issues.apache.org/jira/browse/YARN-8162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440064#comment-16440064
]
Yufei Gu commented on YARN-8162:
Thanks [~haibochen]. These unit test failures are unrelated.
> Remove Method DirectoryCollection#verifyDirUsingMkdir
> -
>
> Key: YARN-8162
> URL: https://issues.apache.org/jira/browse/YARN-8162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Major
> Attachments: YARN-8162.001.patch, YARN-8162.002.patch
>
>
> First of all, DirectoryCollection#verifyDirUsingMkdir doesn't do what it
> claims to do, it creates a file but its Java doc says it create a dir.
> Because of that, it doesn't compatible with {{ReadWriteDiskValidator}}
> because the validator will throw an exception if it isn't dir. What's more,
> it is no longer needed after HADOOP-13738 (DiskChecker should perform some
> disk IO). It duplicates what HADOOP-13738 does. We should remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7810) TestDockerContainerRuntime test failures due to UID lookup of a non-existent user
[
https://issues.apache.org/jira/browse/YARN-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7810:
Target Version/s: 2.10.0, 2.9.1
Fix Version/s: (was: 2.10.0)
> TestDockerContainerRuntime test failures due to UID lookup of a non-existent
> user
> -
>
> Key: YARN-7810
> URL: https://issues.apache.org/jira/browse/YARN-7810
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
>Priority: Major
> Fix For: 3.1.0, 3.0.2
>
> Attachments: YARN-7810-branch-2.001.patch,
> YARN-7810-branch-2.002.patch, YARN-7810-branch-3.0.001.patch,
> YARN-7810.001.patch, YARN-7810.002.patch
>
>
> YARN-7782 enabled the Docker runtime feature to remap the username to uid:gid
> form for launching Docker containers. The feature does an {{id -u}} and {{id
> -G}} to get the UID and GIDs. This fails with the test user, as that user
> doesn't actually exist on the host.
> {code:java}
> [ERROR]
> testContainerLaunchWithCustomNetworks(org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime)
> Time elapsed: 0.411 s <<< ERROR!
> org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException:
>
> ExitCodeException exitCode=1: id: 'run_as_user': no such user
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.getUserIdInfo(DockerLinuxContainerRuntime.java:711)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:757)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks(TestDockerContainerRuntime.java:599){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4781) Support intra-queue preemption for fairness ordering policy.
[ https://issues.apache.org/jira/browse/YARN-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated YARN-4781: - Attachment: YARN-4781.004.patch > Support intra-queue preemption for fairness ordering policy. > > > Key: YARN-4781 > URL: https://issues.apache.org/jira/browse/YARN-4781 > Project: Hadoop YARN > Issue Type: Sub-task > Components: scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: YARN-4781.001.patch, YARN-4781.002.patch, > YARN-4781.003.patch, YARN-4781.004.patch > > > We introduced fairness queue policy since YARN-3319, which will let large > applications make progresses and not starve small applications. However, if a > large application takes the queue’s resources, and containers of the large > app has long lifespan, small applications could still wait for resources for > long time and SLAs cannot be guaranteed. > Instead of wait for application release resources on their own, we need to > preempt resources of queue with fairness policy enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8151) Yarn RM Epoch should wrap around
[
https://issues.apache.org/jira/browse/YARN-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16440046#comment-16440046
]
genericqa commented on YARN-8151:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
1s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 22s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 2 new + 290 unchanged - 0 fixed = 292 total (was 290) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
10s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 40s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
49s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNodeLabelUpdate
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerSchedulingRequestUpdate
|
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestContinuousScheduling |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8151 |
| JIRA
[jira] [Created] (YARN-8167) Improve Diagonstic message when a user without privileged permission deploys a privileged app
Yesha Vora created YARN-8167:
Summary: Improve Diagonstic message when a user without privileged
permission deploys a privileged app
Key: YARN-8167
URL: https://issues.apache.org/jira/browse/YARN-8167
Project: Hadoop YARN
Issue Type: Bug
Components: yarn-native-services
Affects Versions: 3.1.0
Reporter: Yesha Vora
Steps:
1) Validate hrt_qa user is not mentioned in
yarn.nodemanager.runtime.linux.docker.privileged-containers.acl
2) launch a dshell app with
YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER=true
{code}
/usr/hdp/current/hadoop-yarn-client/bin/yarn jar
/usr/hdp/3.0.0.0/hadoop-yarn/hadoop-yarn-applications-distributedshell-3.0.0.jar
-shell_command "sleep 30" -num_containers 1 -shell_env
YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=centos/httpd-24-centos7:latest -shell_env
YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER=true -jar
/usr/hdp/3.0.0.0/hadoop-yarn/hadoop-yarn-applications-distributedshell-3.0.0.jar{code}
The application is failing to launch the container. However, diagnostic message
of the app is not proper.
It returns "Diagnostics., total=1, completed=1, allocated=1, failed=1"
The AM log also does not have proper error message
{code:title=AppMaster.stderr}
18/04/16 20:45:56 INFO distributedshell.ApplicationMaster:
appattempt_1523387473707_0049_01 got container status for
containerID=container_e24_1523387473707_0049_01_02, state=COMPLETE,
exitStatus=-1, diagnostics=[2018-04-16 20:45:49.062]Exception from
container-launch.
Container id: container_e24_1523387473707_0049_01_02
Exit code: -1
Exception message:
Shell output:
[2018-04-16 20:45:49.085]Container exited with a non-zero exit code -1.
[2018-04-16 20:45:49.085]Container exited with a non-zero exit code -1.
18/04/16 20:45:56 INFO distributedshell.ApplicationMaster: Application
completed. Stopping running containers{code}
Diagonstic message should be improved to explicitly mention that "hrt_qa user
does not have permission to launch privilege containers"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7088) Fix application start time and add submit time to UIs
[ https://issues.apache.org/jira/browse/YARN-7088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439979#comment-16439979 ] Haibo Chen commented on YARN-7088: -- +1 pending Jenkins. > Fix application start time and add submit time to UIs > - > > Key: YARN-7088 > URL: https://issues.apache.org/jira/browse/YARN-7088 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Abdullah Yousufi >Assignee: Kanwaljeet Sachdev >Priority: Major > Attachments: YARN-7088.001.patch, YARN-7088.002.patch, > YARN-7088.003.patch, YARN-7088.004.patch, YARN-7088.005.patch, > YARN-7088.006.patch, YARN-7088.007.patch, YARN-7088.008.patch, > YARN-7088.009.patch, YARN-7088.010.patch, YARN-7088.011.patch, > YARN-7088.012.patch, YARN-7088.013.patch, YARN-7088.014.patch, > YARN-7088.015.patch, YARN-7088.016.patch, YARN-7088.017.patch > > > Currently, the start time in the old and new UI actually shows the app > submission time. There should actually be two different fields; one for the > app's submission and one for its start, as well as the elapsed pending time > between the two. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7088) Fix application start time and add submit time to UIs
[ https://issues.apache.org/jira/browse/YARN-7088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kanwaljeet Sachdev updated YARN-7088: - Attachment: YARN-7088.017.patch > Fix application start time and add submit time to UIs > - > > Key: YARN-7088 > URL: https://issues.apache.org/jira/browse/YARN-7088 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Abdullah Yousufi >Assignee: Kanwaljeet Sachdev >Priority: Major > Attachments: YARN-7088.001.patch, YARN-7088.002.patch, > YARN-7088.003.patch, YARN-7088.004.patch, YARN-7088.005.patch, > YARN-7088.006.patch, YARN-7088.007.patch, YARN-7088.008.patch, > YARN-7088.009.patch, YARN-7088.010.patch, YARN-7088.011.patch, > YARN-7088.012.patch, YARN-7088.013.patch, YARN-7088.014.patch, > YARN-7088.015.patch, YARN-7088.016.patch, YARN-7088.017.patch > > > Currently, the start time in the old and new UI actually shows the app > submission time. There should actually be two different fields; one for the > app's submission and one for its start, as well as the elapsed pending time > between the two. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8162) Remove Method DirectoryCollection#verifyDirUsingMkdir
[
https://issues.apache.org/jira/browse/YARN-8162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439924#comment-16439924
]
genericqa commented on YARN-8162:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 21s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 23m 25s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.nodemanager.TestLocalDirsHandlerService |
| | hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks |
| | hadoop.yarn.server.nodemanager.TestNodeManagerReboot |
| |
hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor |
| | hadoop.yarn.server.nodemanager.TestNodeHealthService |
| |
hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
|
| |
hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing
|
| | hadoop.yarn.server.nodemanager.webapp.TestNMWebServices |
| | hadoop.yarn.server.nodemanager.containermanager.TestContainerManager |
| |
hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService
|
| | hadoop.yarn.server.nodemanager.TestNodeManagerResync |
| | hadoop.yarn.server.nodemanager.TestNodeManagerShutdown |
| |
hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8162 |
| JIRA Patch URL |
[jira] [Commented] (YARN-8163) Add support for Node Labels in opportunistic scheduling.
[
https://issues.apache.org/jira/browse/YARN-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439874#comment-16439874
]
genericqa commented on YARN-8163:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
35s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
8s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 66m
32s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8163 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919247/YARN-8163.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle cc |
| uname | Linux 1493fc1ab770 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 896b473 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| Test Results |
[jira] [Updated] (YARN-8032) Yarn service should expose failuresValidityInterval to users and use it for launching containers
[ https://issues.apache.org/jira/browse/YARN-8032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated YARN-8032: Target Version/s: 3.2.0, 3.1.1 (was: 3.2.0) Fix Version/s: 3.1.1 > Yarn service should expose failuresValidityInterval to users and use it for > launching containers > > > Key: YARN-8032 > URL: https://issues.apache.org/jira/browse/YARN-8032 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.1.0 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Fix For: 3.2.0, 3.1.1 > > Attachments: YARN-8032.001.patch, YARN-8032.002.patch, > YARN-8032.003.patch > > > With YARN-5015 the support for sliding window retry policy was added. Yarn > service should expose it via the api for the users to take advantage of it. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8162) Remove Method DirectoryCollection#verifyDirUsingMkdir
[
https://issues.apache.org/jira/browse/YARN-8162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439862#comment-16439862
]
Haibo Chen commented on YARN-8162:
--
+1 pending jenkins.
> Remove Method DirectoryCollection#verifyDirUsingMkdir
> -
>
> Key: YARN-8162
> URL: https://issues.apache.org/jira/browse/YARN-8162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Major
> Attachments: YARN-8162.001.patch, YARN-8162.002.patch
>
>
> First of all, DirectoryCollection#verifyDirUsingMkdir doesn't do what it
> claims to do, it creates a file but its Java doc says it create a dir.
> Because of that, it doesn't compatible with {{ReadWriteDiskValidator}}
> because the validator will throw an exception if it isn't dir. What's more,
> it is no longer needed after HADOOP-13738 (DiskChecker should perform some
> disk IO). It duplicates what HADOOP-13738 does. We should remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8162) Remove Method DirectoryCollection#verifyDirUsingMkdir
[
https://issues.apache.org/jira/browse/YARN-8162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439837#comment-16439837
]
Yufei Gu commented on YARN-8162:
Thanks for the review. Oh, that's even worse. I uploaded the second patch to
remove the unused imports.
> Remove Method DirectoryCollection#verifyDirUsingMkdir
> -
>
> Key: YARN-8162
> URL: https://issues.apache.org/jira/browse/YARN-8162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Major
> Attachments: YARN-8162.001.patch, YARN-8162.002.patch
>
>
> First of all, DirectoryCollection#verifyDirUsingMkdir doesn't do what it
> claims to do, it creates a file but its Java doc says it create a dir.
> Because of that, it doesn't compatible with {{ReadWriteDiskValidator}}
> because the validator will throw an exception if it isn't dir. What's more,
> it is no longer needed after HADOOP-13738 (DiskChecker should perform some
> disk IO). It duplicates what HADOOP-13738 does. We should remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8161) ServiceState FLEX should be removed
[
https://issues.apache.org/jira/browse/YARN-8161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439811#comment-16439811
]
Eric Yang edited comment on YARN-8161 at 4/16/18 6:18 PM:
--
[~gsaha] Thank you for filing this issue. The current REST API performs
operations such as:
{code}
curl -i --negotiate -u : -X PUT -H "Content-Type: application/json"
-d@/tmp/flex.json
http://eyang-2.openstacklocal:8088/app/v1/services/q1/components/ping
{code}
The payload of JSON shows:
{code}
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "FLEXING"
}
{code}
The state is in progressive tense. There is no clear indicator when querying
GET status API, whether AM is actually performing the operation, or the state
was set by user who issued the operation request.
It would be nice to enhance the API to be more responsive by separating user
requested operation and state change.
For example, the pay load could be:
{code}
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "FLEX"
}
{code}
Application Master code will change to FLEXING when operation is being worked
on. GET status API is invoked.
{code}
...
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "FLEXING"
}
...
{code}
When operation is finished, it reaches STABLE state:
{code}
...
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "STABLE"
}
...
{code}
The state transition provides better user feedback between state being
triggered, or server is currently working on the operation.
The original code was written such that -flex operation is in cli code, which
switch to FLEXING when it reaches REST API. For third party developer that
does not rely on cli code base, this can be confusing. I think more feedback
from the community, can help to decide to simplify code base by removing
present tense state transition, or adding present tense transition for better
user feedback.
was (Author: eyang):
[~gsaha] Thank you for filing this issue. The current REST API performs
operations such as:
{code}
curl -i --negotiate -u : -X PUT -H "Content-Type: application/json"
-d@/tmp/flex.json
http://eyang-2.openstacklocal:8088/app/v1/services/q1/components/ping
{code}
The payload of JSON shows:
{code}
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "FLEXING"
}
{code}
The state is in progressive tense. There is no clear indicator when querying
GET status API, whether AM is actually performing the operation, or the state
was set by user who issued the operation request.
It would be nice to enhance the API to be more responsive by separating user
requested operation and state change.
For example, the pay load could be:
{code}
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "FLEX"
}
{code}
Application Master code will change to FLEXING when operation is being worked
on. GET status API is invoked.
{code}
...
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "FLEXING"
}
...
{code}
When operation is finished, it reaches STABLE state:
{code}
...
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "STABLE"
}
...
{code}
The state transition provides better user feedback between state being
triggered, or server is currently working on the operation.
The original code was written such that -flex operation is in cli code, which
switch to FLEXING when it reaches REST API. For third party developer that
does not rely on cli code base, this can be confusing. I think more feedback
from the community, can help to decide to simply code base by removing present
tense state transition, or adding present tense transition for better user
feedback.
> ServiceState FLEX should be removed
> ---
>
> Key: YARN-8161
> URL: https://issues.apache.org/jira/browse/YARN-8161
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-native-services
>Affects Versions: 3.1.0
>Reporter: Gour Saha
>Priority: Major
>
> ServiceState FLEX is not required to trigger flex up/down of containers and
> should be removed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8161) ServiceState FLEX should be removed
[
https://issues.apache.org/jira/browse/YARN-8161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439811#comment-16439811
]
Eric Yang edited comment on YARN-8161 at 4/16/18 6:16 PM:
--
[~gsaha] Thank you for filing this issue. The current REST API performs
operations such as:
{code}
curl -i --negotiate -u : -X PUT -H "Content-Type: application/json"
-d@/tmp/flex.json
http://eyang-2.openstacklocal:8088/app/v1/services/q1/components/ping
{code}
The payload of JSON shows:
{code}
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "FLEXING"
}
{code}
The state is in progressive tense. There is no clear indicator when querying
GET status API, whether AM is actually performing the operation, or the state
was set by user who issued the operation request.
It would be nice to enhance the API to be more responsive by separating user
requested operation and state change.
For example, the pay load could be:
{code}
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "FLEX"
}
{code}
Application Master code will change to FLEXING when operation is being worked
on. GET status API is invoked.
{code}
...
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "FLEXING"
}
...
{code}
When operation is finished, it reaches STABLE state:
{code}
...
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "STABLE"
}
...
{code}
The state transition provides better user feedback between state being
triggered, or server is currently working on the operation.
The original code was written such that -flex operation is in cli code, which
switch to FLEXING when it reaches REST API. For third party developer that
does not rely on cli code base, this can be confusing. I think more feedback
from the community, can help to decide to simply code base by removing present
tense state transition, or adding present tense transition for better user
feedback.
was (Author: eyang):
[~gsaha] Thank you for filing this issue. The current REST API performs
operations such as:
{quote}
curl -i --negotiate -u : -X PUT -H "Content-Type: application/json"
-d@/tmp/flex.json
http://eyang-2.openstacklocal:8088/app/v1/services/q1/components/ping
{quote}
The payload of JSON shows:
{quote}
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "FLEXING"
}
{quote}
The state is in progressive tense. There is no clear indicator when querying
GET status API, whether AM is actually performing the operation, or the state
was set by user who issued the operation request.
It would be nice to enhance the API to be more responsive by separating user
requested operation and state change.
For example, the pay load could be:
{quote}
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "FLEX"
}
{quote}
Application Master code will change to FLEXING when operation is being worked
on. GET status API is invoked.
{quote}
...
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "FLEXING"
}
...
{quote}
When operation is finished, it reaches STABLE state:
{quote}
...
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "STABLE"
}
...
{quote}
The state transition provides better user feedback between state being
triggered, or server is currently working on the operation.
The original code was written such that -flex operation is in cli code, which
switch to FLEXING when it reaches REST API. For third party developer that
does not rely on cli code base, this can be confusing. I think more feedback
from the community, can help to decide to simply code base by removing present
tense state transition, or adding present tense transition for better user
feedback.
> ServiceState FLEX should be removed
> ---
>
> Key: YARN-8161
> URL: https://issues.apache.org/jira/browse/YARN-8161
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-native-services
>Affects Versions: 3.1.0
>Reporter: Gour Saha
>Priority: Major
>
> ServiceState FLEX is not required to trigger flex up/down of containers and
> should be removed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8161) ServiceState FLEX should be removed
[
https://issues.apache.org/jira/browse/YARN-8161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439811#comment-16439811
]
Eric Yang commented on YARN-8161:
-
[~gsaha] Thank you for filing this issue. The current REST API performs
operations such as:
{quote}
curl -i --negotiate -u : -X PUT -H "Content-Type: application/json"
-d@/tmp/flex.json
http://eyang-2.openstacklocal:8088/app/v1/services/q1/components/ping
{quote}
The payload of JSON shows:
{quote}
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "FLEXING"
}
{quote}
The state is in progressive tense. There is no clear indicator when querying
GET status API, whether AM is actually performing the operation, or the state
was set by user who issued the operation request.
It would be nice to enhance the API to be more responsive by separating user
requested operation and state change.
For example, the pay load could be:
{quote}
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "FLEX"
}
{quote}
Application Master code will change to FLEXING when operation is being worked
on. GET status API is invoked.
{quote}
...
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "FLEXING"
}
...
{quote}
When operation is finished, it reaches STABLE state:
{quote}
...
{
"name" : "ping",
"number_of_containers" : 3,
"run_privileged_container" : false,
"state" : "STABLE"
}
...
{quote}
The state transition provides better user feedback between state being
triggered, or server is currently working on the operation.
The original code was written such that -flex operation is in cli code, which
switch to FLEXING when it reaches REST API. For third party developer that
does not rely on cli code base, this can be confusing. I think more feedback
from the community, can help to decide to simply code base by removing present
tense state transition, or adding present tense transition for better user
feedback.
> ServiceState FLEX should be removed
> ---
>
> Key: YARN-8161
> URL: https://issues.apache.org/jira/browse/YARN-8161
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-native-services
>Affects Versions: 3.1.0
>Reporter: Gour Saha
>Priority: Major
>
> ServiceState FLEX is not required to trigger flex up/down of containers and
> should be removed
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8151) Yarn RM Epoch should wrap around
[ https://issues.apache.org/jira/browse/YARN-8151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Young Chen updated YARN-8151: - Attachment: YARN-8151.01.patch > Yarn RM Epoch should wrap around > > > Key: YARN-8151 > URL: https://issues.apache.org/jira/browse/YARN-8151 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Young Chen >Assignee: Young Chen >Priority: Major > Attachments: YARN-8151.01.patch, YARN-8151.01.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8166) Service AppId page throws HTTP Error 401
[
https://issues.apache.org/jira/browse/YARN-8166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yesha Vora reassigned YARN-8166:
Assignee: Yesha Vora
> Service AppId page throws HTTP Error 401
>
>
> Key: YARN-8166
> URL: https://issues.apache.org/jira/browse/YARN-8166
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-ui-v2
>Reporter: Yesha Vora
>Assignee: Yesha Vora
>Priority: Major
>
> Steps:
> 1) Launch a yarn service in unsecure cluster
> 2) Go to component info page for sleeper-0
> 3) click on sleeper link
> http://xxx:8088/ui2/#/yarn-component-instances/sleeper/components?service=yesha-sleeper&=application_1518804855867_0002
> Above url fails with HTTP Error 401
> {code}
> 401, Authorization required.
> Please check your security settings.
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-8166) Service AppId page throws HTTP Error 401
Yesha Vora created YARN-8166:
Summary: Service AppId page throws HTTP Error 401
Key: YARN-8166
URL: https://issues.apache.org/jira/browse/YARN-8166
Project: Hadoop YARN
Issue Type: Bug
Components: yarn-ui-v2
Reporter: Yesha Vora
Steps:
1) Launch a yarn service in unsecure cluster
2) Go to component info page for sleeper-0
3) click on sleeper link
http://xxx:8088/ui2/#/yarn-component-instances/sleeper/components?service=yesha-sleeper&=application_1518804855867_0002
Above url fails with HTTP Error 401
{code}
401, Authorization required.
Please check your security settings.
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-1014) Configure OOM Killer to kill OPPORTUNISTIC containers first
[ https://issues.apache.org/jira/browse/YARN-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-1014: - Issue Type: Improvement (was: Sub-task) Parent: (was: YARN-1011) > Configure OOM Killer to kill OPPORTUNISTIC containers first > --- > > Key: YARN-1014 > URL: https://issues.apache.org/jira/browse/YARN-1014 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.0.0-alpha1 >Reporter: Arun C Murthy >Assignee: Miklos Szegedi >Priority: Major > Attachments: YARN-1014.00.patch, YARN-1014.01.patch, > YARN-1014.02.patch > > > YARN-2882 introduces the notion of OPPORTUNISTIC containers. These containers > should be killed first should the system run out of memory. > - > Previous description: > Once RM allocates 'speculative containers' we need to get LCE to schedule > them at lower priorities via cgroups. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8071) Add ability to specify nodemanager environment variables individually
[
https://issues.apache.org/jira/browse/YARN-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439791#comment-16439791
]
Jim Brennan commented on YARN-8071:
---
[~jlowe] thanks for the review.
{quote}The original code passed the current environment map, allowing admin
variables to reference other variables defined so far in the environment. The
new code passes an empty map which would seem to preclude this and could be a
backwards compatibility issue.
{quote}
Thanks for pointing this out. I meant to ask specifically about this change.
This was intentional. I agree it is a change in functionality, but it seemed to
me that the current behavior may actually be a bug, not the intended behavior.
I based this on the documentation for {{yarn.nodemanager.admin-env}}, the
comment that precedes this code ({{variables here will be forced in, even if
the container has specified them.}}, and the fact that everything else in this
function overrides any user-specified variable (with the exception of the
windows-specific classpath stuff).
That said, I don't have a good idea of how likely this change would be to break
something, so I am definitely willing to change it if it is considered too
dangerous.
{quote}The changes to TestContainerLaunch#testPrependDistcache appear to be
unnecessary?
{quote}
They were intentional. When I was testing my new test case, I realized that
passing the empty set for the {{nmVars}} argument leads to exceptions in
{{addToEnvMap()}}, so I fixed the testPrependDistcache() cases as well - I
assume this windows-only test must be failing without this fix.
> Add ability to specify nodemanager environment variables individually
> -
>
> Key: YARN-8071
> URL: https://issues.apache.org/jira/browse/YARN-8071
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-8071.001.patch, YARN-8071.002.patch
>
>
> YARN-6830 describes a problem where environment variables that contain commas
> cannot be specified via {{-Dmapreduce.map.env}}.
> For example:
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> will set {{MOUNTS}} to {{/tmp/foo}}
> In that Jira, [~aw] suggested that we change the API to provide a way to
> specify environment variables individually, the same way that Spark does.
> {quote}Rather than fight with a regex why not redefine the API instead?
>
> -Dmapreduce.map.env.MODE=bar
> -Dmapreduce.map.env.IMAGE_NAME=foo
> -Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar
> ...
> e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> This greatly simplifies the input validation needed and makes it clear what
> is actually being defined.
> {quote}
> The mapreduce properties were dealt with in [MAPREDUCE-7069]. This Jira will
> address the YARN properties.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8162) Remove Method DirectoryCollection#verifyDirUsingMkdir
[
https://issues.apache.org/jira/browse/YARN-8162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-8162:
---
Attachment: YARN-8162.002.patch
> Remove Method DirectoryCollection#verifyDirUsingMkdir
> -
>
> Key: YARN-8162
> URL: https://issues.apache.org/jira/browse/YARN-8162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Major
> Attachments: YARN-8162.001.patch, YARN-8162.002.patch
>
>
> First of all, DirectoryCollection#verifyDirUsingMkdir doesn't do what it
> claims to do, it creates a file but its Java doc says it create a dir.
> Because of that, it doesn't compatible with {{ReadWriteDiskValidator}}
> because the validator will throw an exception if it isn't dir. What's more,
> it is no longer needed after HADOOP-13738 (DiskChecker should perform some
> disk IO). It duplicates what HADOOP-13738 does. We should remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8162) Remove Method DirectoryCollection#verifyDirUsingMkdir
[
https://issues.apache.org/jira/browse/YARN-8162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439780#comment-16439780
]
Haibo Chen commented on YARN-8162:
--
Thanks [~yufeigu] for the patch! I agree with you on that it is redundant.
Hence, removing the method. But in the meantime, I don't think in
verifyDirUsingMkdir() files are created.
> Remove Method DirectoryCollection#verifyDirUsingMkdir
> -
>
> Key: YARN-8162
> URL: https://issues.apache.org/jira/browse/YARN-8162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.1.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>Priority: Major
> Attachments: YARN-8162.001.patch
>
>
> First of all, DirectoryCollection#verifyDirUsingMkdir doesn't do what it
> claims to do, it creates a file but its Java doc says it create a dir.
> Because of that, it doesn't compatible with {{ReadWriteDiskValidator}}
> because the validator will throw an exception if it isn't dir. What's more,
> it is no longer needed after HADOOP-13738 (DiskChecker should perform some
> disk IO). It duplicates what HADOOP-13738 does. We should remove it.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7810) TestDockerContainerRuntime test failures due to UID lookup of a non-existent user
[
https://issues.apache.org/jira/browse/YARN-7810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang resolved YARN-7810.
-
Resolution: Fixed
Thank you [~jojochuang] for catching branch-2 issues.
Thank you [~shaneku...@gmail.com] for the branch-2 patch. I verified that it
works with JDK7. The addendum patch is applied to branch-2.9 and branch-2.
> TestDockerContainerRuntime test failures due to UID lookup of a non-existent
> user
> -
>
> Key: YARN-7810
> URL: https://issues.apache.org/jira/browse/YARN-7810
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
>Priority: Major
> Fix For: 3.1.0, 2.10.0, 3.0.2
>
> Attachments: YARN-7810-branch-2.001.patch,
> YARN-7810-branch-2.002.patch, YARN-7810-branch-3.0.001.patch,
> YARN-7810.001.patch, YARN-7810.002.patch
>
>
> YARN-7782 enabled the Docker runtime feature to remap the username to uid:gid
> form for launching Docker containers. The feature does an {{id -u}} and {{id
> -G}} to get the UID and GIDs. This fails with the test user, as that user
> doesn't actually exist on the host.
> {code:java}
> [ERROR]
> testContainerLaunchWithCustomNetworks(org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime)
> Time elapsed: 0.411 s <<< ERROR!
> org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerExecutionException:
>
> ExitCodeException exitCode=1: id: 'run_as_user': no such user
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.getUserIdInfo(DockerLinuxContainerRuntime.java:711)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime.launchContainer(DockerLinuxContainerRuntime.java:757)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks(TestDockerContainerRuntime.java:599){code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7088) Fix application start time and add submit time to UIs
[ https://issues.apache.org/jira/browse/YARN-7088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439762#comment-16439762 ] Haibo Chen commented on YARN-7088: -- Thanks for the update! Can you please clean up schedulingwaitTime and totalRunTime in AppInfo.java? Also, the log statement in AttemptLaunchedTransition should be enclosed in the if clause when we actually update the launch time stamp. > Fix application start time and add submit time to UIs > - > > Key: YARN-7088 > URL: https://issues.apache.org/jira/browse/YARN-7088 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Abdullah Yousufi >Assignee: Kanwaljeet Sachdev >Priority: Major > Attachments: YARN-7088.001.patch, YARN-7088.002.patch, > YARN-7088.003.patch, YARN-7088.004.patch, YARN-7088.005.patch, > YARN-7088.006.patch, YARN-7088.007.patch, YARN-7088.008.patch, > YARN-7088.009.patch, YARN-7088.010.patch, YARN-7088.011.patch, > YARN-7088.012.patch, YARN-7088.013.patch, YARN-7088.014.patch, > YARN-7088.015.patch, YARN-7088.016.patch > > > Currently, the start time in the old and new UI actually shows the app > submission time. There should actually be two different fields; one for the > app's submission and one for its start, as well as the elapsed pending time > between the two. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7781) Update YARN-Services-Examples.md to be in sync with the latest code
[
https://issues.apache.org/jira/browse/YARN-7781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439736#comment-16439736
]
Eric Yang commented on YARN-7781:
-
[~gsaha] Thank you for updating the example. YARN still has performance
limitations in heterogeneous environment. I don't think we want to mix Linux
and Windows in nodes attributes example. It is probably better to list all
nodes as the same OS type to prevent misleading the reader at this time. Small
nitpick, There is no explanation for fault_domain. I did not understand this
initially. I guess it's arbitrary key value pair. I wonder if tagging node
with "cpu":"intel" or "cpu":"amd" for node attributes can be easier for reader
to consume?
> Update YARN-Services-Examples.md to be in sync with the latest code
> ---
>
> Key: YARN-7781
> URL: https://issues.apache.org/jira/browse/YARN-7781
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Gour Saha
>Assignee: Gour Saha
>Priority: Major
> Attachments: YARN-7781.01.patch, YARN-7781.02.patch,
> YARN-7781.03.patch, YARN-7781.04.patch, YARN-7781.05.patch
>
>
> Update YARN-Services-Examples.md to make the following additions/changes:
> 1. Add an additional URL and PUT Request JSON to support flex:
> Update to flex up/down the no of containers (instances) of a component of a
> service
> PUT URL – http://localhost:8088/app/v1/services/hello-world
> PUT Request JSON
> {code}
> {
> "components" : [ {
> "name" : "hello",
> "number_of_containers" : 3
> } ]
> }
> {code}
> 2. Modify all occurrences of /ws/ to /app/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8163) Add support for Node Labels in opportunistic scheduling.
[ https://issues.apache.org/jira/browse/YARN-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-8163: Attachment: YARN-8163.002.patch > Add support for Node Labels in opportunistic scheduling. > > > Key: YARN-8163 > URL: https://issues.apache.org/jira/browse/YARN-8163 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-8163.002.patch, YARN-8163.patch > > > Currently, the Opportunistic Scheduler doesn't honor node labels constraints > and schedule containers based on locality and load constraints. This Jira is > to add support in opportunistic scheduling to honor node labels in resource > requests. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6495) check docker container's exit code when writing to cgroup task files
[
https://issues.apache.org/jira/browse/YARN-6495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439690#comment-16439690
]
Eric Badger edited comment on YARN-6495 at 4/16/18 4:33 PM:
Hey [~Jaeboo], thanks for the patch update. The patch doesn't apply for me on
trunk. I believe a rebase is required. However, here are my comments looking at
the patch
{noformat}
+ // write pid to cgroups
+ char* const* cgroup_ptr;
+ int docker_exit_code = 0;
+ for (cgroup_ptr = resources_values; cgroup_ptr != NULL &&
+ *cgroup_ptr != NULL; ++cgroup_ptr) {
+if (strcmp(*cgroup_ptr, "none") != 0 &&
+ write_pid_to_cgroup_as_root(*cgroup_ptr, pid) != 0) {
+ docker_exit_code = check_docker_exit_code(docker_binary,
container_id);
+ if (docker_exit_code != 0) {
+exit_code = docker_exit_code;
+goto cleanup;
+ } else {
+exit_code = WRITE_CGROUP_FAILED;
+goto cleanup;
+ }
+}
+ }
{noformat}
This is semantically different from the previous version of the patch in that
now failed cgroup writes will always cause an error. When the cgroup write
fails due to {{no such process}}, but the docker exit code is 0, we want to
continue on without error.
Additionally, as of now, there is currently no support in
{{write_pid_to_cgroup_as_root()}} to differentiate between an error due to {{no
such process}} or a different type of error (opening the files or changing
effective user). On the former, we want to ignore the cgroup write error so
long as the docker exit code is 0. On the latter, we want to fail regardless of
the docker outcome.
was (Author: ebadger):
Hey [~Jaeboo], thanks for the patch update
{noformat}
+ // write pid to cgroups
+ char* const* cgroup_ptr;
+ int docker_exit_code = 0;
+ for (cgroup_ptr = resources_values; cgroup_ptr != NULL &&
+ *cgroup_ptr != NULL; ++cgroup_ptr) {
+if (strcmp(*cgroup_ptr, "none") != 0 &&
+ write_pid_to_cgroup_as_root(*cgroup_ptr, pid) != 0) {
+ docker_exit_code = check_docker_exit_code(docker_binary,
container_id);
+ if (docker_exit_code != 0) {
+exit_code = docker_exit_code;
+goto cleanup;
+ } else {
+exit_code = WRITE_CGROUP_FAILED;
+goto cleanup;
+ }
+}
+ }
{noformat}
This is semantically different from the previous version of the patch in that
now failed cgroup writes will always cause an error. When the cgroup write
fails due to {{no such process}}, but the docker exit code is 0, we want to
continue on without error.
Additionally, as of now, there is currently no support in
{{write_pid_to_cgroup_as_root()}} to differentiate between an error due to {{no
such process}} or a different type of error (opening the files or changing
effective user). On the former, we want to ignore the cgroup write error so
long as the docker exit code is 0. On the latter, we want to fail regardless of
the docker outcome.
> check docker container's exit code when writing to cgroup task files
>
>
> Key: YARN-6495
> URL: https://issues.apache.org/jira/browse/YARN-6495
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
>Priority: Major
> Attachments: YARN-6495.001.patch, YARN-6495.002.patch
>
>
> If I execute simple command like date on docker container, the application
> failed to complete successfully.
> for example,
> {code}
> $ yarn jar
> $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=hadoop-docker -shell_command "date" -jar
> $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -num_containers 1 -timeout 360
> …
> 17/04/12 00:16:40 INFO distributedshell.Client: Application did finished
> unsuccessfully. YarnState=FINISHED, DSFinalStatus=FAILED. Breaking monitoring
> loop
> 17/04/12 00:16:40 ERROR distributedshell.Client: Application failed to
> complete successfully
> {code}
> The error log is like below.
> {code}
> ...
> Failed to write pid to file
> /cgroup_parent/cpu/hadoop-yarn/container_/tasks - No such process
> ...
> {code}
> When writing pid to cgroup tasks, container-executor doesn’t check docker
> container’s status.
> If the container finished very quickly, we can’t write pid to cgroup tasks,
> and it is not problem.
> So container-executor needs to check docker container’s exit code during
> writing pid to cgroup tasks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (YARN-6495) check docker container's exit code when writing to cgroup task files
[
https://issues.apache.org/jira/browse/YARN-6495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439690#comment-16439690
]
Eric Badger commented on YARN-6495:
---
Hey [~Jaeboo], thanks for the patch update
{noformat}
+ // write pid to cgroups
+ char* const* cgroup_ptr;
+ int docker_exit_code = 0;
+ for (cgroup_ptr = resources_values; cgroup_ptr != NULL &&
+ *cgroup_ptr != NULL; ++cgroup_ptr) {
+if (strcmp(*cgroup_ptr, "none") != 0 &&
+ write_pid_to_cgroup_as_root(*cgroup_ptr, pid) != 0) {
+ docker_exit_code = check_docker_exit_code(docker_binary,
container_id);
+ if (docker_exit_code != 0) {
+exit_code = docker_exit_code;
+goto cleanup;
+ } else {
+exit_code = WRITE_CGROUP_FAILED;
+goto cleanup;
+ }
+}
+ }
{noformat}
This is semantically different from the previous version of the patch in that
now failed cgroup writes will always cause an error. When the cgroup write
fails due to {{no such process}}, but the docker exit code is 0, we want to
continue on without error.
Additionally, as of now, there is currently no support in
{{write_pid_to_cgroup_as_root()}} to differentiate between an error due to {{no
such process}} or a different type of error (opening the files or changing
effective user). On the former, we want to ignore the cgroup write error so
long as the docker exit code is 0. On the latter, we want to fail regardless of
the docker outcome.
> check docker container's exit code when writing to cgroup task files
>
>
> Key: YARN-6495
> URL: https://issues.apache.org/jira/browse/YARN-6495
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Jaeboo Jeong
>Assignee: Jaeboo Jeong
>Priority: Major
> Attachments: YARN-6495.001.patch, YARN-6495.002.patch
>
>
> If I execute simple command like date on docker container, the application
> failed to complete successfully.
> for example,
> {code}
> $ yarn jar
> $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env
> YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=hadoop-docker -shell_command "date" -jar
> $HADOOP_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.1.jar
> -num_containers 1 -timeout 360
> …
> 17/04/12 00:16:40 INFO distributedshell.Client: Application did finished
> unsuccessfully. YarnState=FINISHED, DSFinalStatus=FAILED. Breaking monitoring
> loop
> 17/04/12 00:16:40 ERROR distributedshell.Client: Application failed to
> complete successfully
> {code}
> The error log is like below.
> {code}
> ...
> Failed to write pid to file
> /cgroup_parent/cpu/hadoop-yarn/container_/tasks - No such process
> ...
> {code}
> When writing pid to cgroup tasks, container-executor doesn’t check docker
> container’s status.
> If the container finished very quickly, we can’t write pid to cgroup tasks,
> and it is not problem.
> So container-executor needs to check docker container’s exit code during
> writing pid to cgroup tasks.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8060) Create default readiness check for service components
[ https://issues.apache.org/jira/browse/YARN-8060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439689#comment-16439689 ] Gour Saha commented on YARN-8060: - Thanks [~billie.rinaldi]. +1 for patch 6. > Create default readiness check for service components > - > > Key: YARN-8060 > URL: https://issues.apache.org/jira/browse/YARN-8060 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-native-services >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi >Priority: Major > Attachments: YARN-8060.1.patch, YARN-8060.2.patch, YARN-8060.3.patch, > YARN-8060.4.patch, YARN-8060.5.patch, YARN-8060.6.patch > > > It is currently possible for a component instance to have READY status before > the AM retrieves an IP for the container. We should make sure the IP has been > retrieved before marking the instance as READY. > This default probe could also have an option to check for a DNS entry for the > instance's hostname if a DNS address is provided. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8071) Add ability to specify nodemanager environment variables individually
[
https://issues.apache.org/jira/browse/YARN-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439678#comment-16439678
]
Jason Lowe commented on YARN-8071:
--
Thanks for the patch!
The original code passed the current environment map, allowing admin variables
to reference other variables defined so far in the environment. The new code
passes an empty map which would seem to preclude this and could be a backwards
compatibility issue.
The changes to TestContainerLaunch#testPrependDistcache appear to be
unnecessary?
> Add ability to specify nodemanager environment variables individually
> -
>
> Key: YARN-8071
> URL: https://issues.apache.org/jira/browse/YARN-8071
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.0.0
>Reporter: Jim Brennan
>Assignee: Jim Brennan
>Priority: Major
> Attachments: YARN-8071.001.patch, YARN-8071.002.patch
>
>
> YARN-6830 describes a problem where environment variables that contain commas
> cannot be specified via {{-Dmapreduce.map.env}}.
> For example:
> {{-Dmapreduce.map.env="MODE=bar,IMAGE_NAME=foo,MOUNTS=/tmp/foo,/tmp/bar"}}
> will set {{MOUNTS}} to {{/tmp/foo}}
> In that Jira, [~aw] suggested that we change the API to provide a way to
> specify environment variables individually, the same way that Spark does.
> {quote}Rather than fight with a regex why not redefine the API instead?
>
> -Dmapreduce.map.env.MODE=bar
> -Dmapreduce.map.env.IMAGE_NAME=foo
> -Dmapreduce.map.env.MOUNTS=/tmp/foo,/tmp/bar
> ...
> e.g, mapreduce.map.env.[foo]=bar gets turned into foo=bar
> This greatly simplifies the input validation needed and makes it clear what
> is actually being defined.
> {quote}
> The mapreduce properties were dealt with in [MAPREDUCE-7069]. This Jira will
> address the YARN properties.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5306) Yarn should detect and fail fast on duplicate resources in container request
[ https://issues.apache.org/jira/browse/YARN-5306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned YARN-5306: Assignee: (was: Junping Du) > Yarn should detect and fail fast on duplicate resources in container request > > > Key: YARN-5306 > URL: https://issues.apache.org/jira/browse/YARN-5306 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Yesha Vora >Priority: Critical > > In some cases, Yarn gets duplicate copies of resources in resource-list. > In this case, you end up with a resource list which contains two copies of > resource JAR, with the timestamps of the two separate uploads —only one of > which (the later one) is correct. At download time, the NM goes through the > list and fails the download when it gets to the one with the older timestamp. > We need some utility class to do a scan & check could be used by the NM at > download time (so fail with meaningful errors), and the yarn client could > perhaps do the check before launch. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5888) [YARN-3368] Improve unit tests for YARN UI
[
https://issues.apache.org/jira/browse/YARN-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439585#comment-16439585
]
genericqa commented on YARN-5888:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
3s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 42m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
57m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 747 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
45s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-5888 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919226/YARN-5888.003.patch |
| Optional Tests | asflicense shadedclient |
| uname | Linux 05df80679bc6 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 896b473 |
| maven | version: Apache Maven 3.3.9 |
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/20356/artifact/out/whitespace-tabs.txt
|
| asflicense |
https://builds.apache.org/job/PreCommit-YARN-Build/20356/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 345 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/20356/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> [YARN-3368] Improve unit tests for YARN UI
> --
>
> Key: YARN-5888
> URL: https://issues.apache.org/jira/browse/YARN-5888
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn-ui-v2
>Reporter: Akhil PB
>Assignee: Akhil PB
>Priority: Minor
> Attachments: YARN-5888.001.patch, YARN-5888.002.patch,
> YARN-5888.003.patch
>
>
> - Add missing test cases in new YARN UI
> - Fix test cases errors in new YARN UI
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7189) Container-executor doesn't remove Docker containers that error out early
[
https://issues.apache.org/jira/browse/YARN-7189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439556#comment-16439556
]
Jason Lowe commented on YARN-7189:
--
Thanks for updating the patch! +1 lgtm. I'll commit this later today if there
are no objections.
> Container-executor doesn't remove Docker containers that error out early
>
>
> Key: YARN-7189
> URL: https://issues.apache.org/jira/browse/YARN-7189
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 2.9.0, 2.8.3, 3.0.1
>Reporter: Eric Badger
>Assignee: Eric Badger
>Priority: Major
> Attachments: YARN-7189-b3.0.001.patch,
> YARN-7189-branch-3.0.001.patch, YARN-7189-branch-3.0.002.patch,
> YARN-7189-branch-3.0.003.patch, YARN-7189-branch-3.0.004.patch
>
>
> Once the docker run command is executed, the docker container is created
> unless the return code is 125 meaning that the run command itself failed
> (https://docs.docker.com/engine/reference/run/#exit-status). Any error that
> happens after the docker run needs to remove the container during cleanup.
> {noformat:title=container-executor.c:launch_docker_container_as_user}
> snprintf(docker_command_with_binary, command_size, "%s %s", docker_binary,
> docker_command);
> fprintf(LOGFILE, "Launching docker container...\n");
> FILE* start_docker = popen(docker_command_with_binary, "r");
> {noformat}
> This is fixed by YARN-5366, which changes how we remove containers. However,
> that was committed into 3.1.0. 2.8, 2.9, and 3.0 are all affected
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7996) Allow user supplied Docker client configurations with YARN native services
[ https://issues.apache.org/jira/browse/YARN-7996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439543#comment-16439543 ] Jason Lowe commented on YARN-7996: -- Thanks for updating the patch! +1 lgtm. I'll commit this later today if there are no objections. > Allow user supplied Docker client configurations with YARN native services > -- > > Key: YARN-7996 > URL: https://issues.apache.org/jira/browse/YARN-7996 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Shane Kumpf >Assignee: Shane Kumpf >Priority: Major > Attachments: YARN-7996.001.patch, YARN-7996.002.patch, > YARN-7996.003.patch, YARN-7996.004.patch, YARN-7996.005.patch, > YARN-7996.006.patch > > > YARN-5428 added support to distributed shell for supplying a Docker client > configuration at application submission time. The auth tokens within the > client configuration are then used to pull images from private Docker > repositories/registries. Add the same support to the YARN Native Services > framework. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5888) [YARN-3368] Improve unit tests for YARN UI
[ https://issues.apache.org/jira/browse/YARN-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439485#comment-16439485 ] Sunil G commented on YARN-5888: --- Rebased [~akhilpb]'s latest patch to trunk. Patch looks fine and good amount of tests are there. Pending jenkins for commit/ > [YARN-3368] Improve unit tests for YARN UI > -- > > Key: YARN-5888 > URL: https://issues.apache.org/jira/browse/YARN-5888 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Minor > Attachments: YARN-5888.001.patch, YARN-5888.002.patch, > YARN-5888.003.patch > > > - Add missing test cases in new YARN UI > - Fix test cases errors in new YARN UI -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5888) [YARN-3368] Improve unit tests for YARN UI
[ https://issues.apache.org/jira/browse/YARN-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-5888: -- Attachment: YARN-5888.003.patch > [YARN-3368] Improve unit tests for YARN UI > -- > > Key: YARN-5888 > URL: https://issues.apache.org/jira/browse/YARN-5888 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Minor > Attachments: YARN-5888.001.patch, YARN-5888.002.patch, > YARN-5888.003.patch > > > - Add missing test cases in new YARN UI > - Fix test cases errors in new YARN UI -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8165) Incorrect queue name logging in AbstractContainerAllocator
[
https://issues.apache.org/jira/browse/YARN-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Weiwei Yang updated YARN-8165:
--
Description:
Found some incorrect logging messages in RM log
{noformat}
INFO Reserved container application=application_1523849397637_0006
resource=
queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@38224385
cluster=
...
INFO AbstractContainerAllocator:131 - assignedContainer application
attempt=appattempt_1523849397637_0006_01 container=null
queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@38224385
clusterResource= type=OFF_SWITCH
requestedPartition=
{noformat}
Note, value for queue is incorrect.
was:
Found some incorrect logging messages in RM log
{noformat}
Reserved container application=application_1523849397637_0006
resource=
queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@38224385
cluster=
...
INFO AbstractContainerAllocator:131 - assignedContainer application
attempt=appattempt_1523849397637_0006_01 container=null
queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@38224385
clusterResource= type=OFF_SWITCH
requestedPartition=
{noformat}
Note, value for queue is incorrect.
> Incorrect queue name logging in AbstractContainerAllocator
> --
>
> Key: YARN-8165
> URL: https://issues.apache.org/jira/browse/YARN-8165
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 2.9.0, 3.0.0
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Trivial
> Attachments: YARN-8165.001.patch
>
>
> Found some incorrect logging messages in RM log
> {noformat}
> INFO Reserved container application=application_1523849397637_0006
> resource=
> queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@38224385
> cluster=
> ...
> INFO AbstractContainerAllocator:131 - assignedContainer application
> attempt=appattempt_1523849397637_0006_01 container=null
> queue=org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.allocator.RegularContainerAllocator@38224385
> clusterResource= type=OFF_SWITCH
> requestedPartition=
> {noformat}
> Note, value for queue is incorrect.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8165) Incorrect queue name logging in AbstractContainerAllocator
[
https://issues.apache.org/jira/browse/YARN-8165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439018#comment-16439018
]
genericqa commented on YARN-8165:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
1s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 42m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 12s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 65m
12s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}146m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8165 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919148/YARN-8165.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e9912b7000da 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 896b473 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/20355/testReport/ |
| Max. process+thread count | 830 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/20355/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT
