[jira] [Commented] (YARN-10340) HsWebServices getContainerReport uses loginUser instead of remoteUser to access ApplicationClientProtocol
[
https://issues.apache.org/jira/browse/YARN-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153240#comment-17153240
]
Prabhu Joseph commented on YARN-10340:
--
Thanks [~tarunparimi] for the analysis.
bq. This creates a separate rpc client instance every time though which is not
efficient.
This won't be a problem as these newly added WebServices (YARN-10028) are used
only by Yarn UI2 unless user opens huge number of UI2 pages at a time. And also
this is the right way for achieving doAs for RPC calls.
> HsWebServices getContainerReport uses loginUser instead of remoteUser to
> access ApplicationClientProtocol
> -
>
> Key: YARN-10340
> URL: https://issues.apache.org/jira/browse/YARN-10340
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Prabhu Joseph
>Assignee: Tarun Parimi
>Priority: Major
>
> HsWebServices getContainerReport uses loginUser instead of remoteUser to
> access ApplicationClientProtocol
>
> [http://:19888/ws/v1/history/containers/container_e03_1594030808801_0002_01_03/logs|http://pjoseph-secure-1.pjoseph-secure.root.hwx.site:19888/ws/v1/history/containers/container_e03_1594030808801_0002_01_03/logs]
> While accessing above link using systest user, the request fails saying
> mapred user does not have access to the job
>
> {code:java}
> 2020-07-06 14:02:59,178 WARN org.apache.hadoop.yarn.server.webapp.LogServlet:
> Could not obtain node HTTP address from provider.
> javax.ws.rs.WebApplicationException:
> org.apache.hadoop.yarn.exceptions.YarnException: User mapred does not have
> privilege to see this application application_1593997842459_0214
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getContainerReport(ClientRMService.java:516)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getContainerReport(ApplicationClientProtocolPBServiceImpl.java:466)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:639)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:985)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:913)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2882)
> at
> org.apache.hadoop.yarn.server.webapp.WebServices.rewrapAndThrowThrowable(WebServices.java:544)
> at
> org.apache.hadoop.yarn.server.webapp.WebServices.rewrapAndThrowException(WebServices.java:530)
> at
> org.apache.hadoop.yarn.server.webapp.WebServices.getContainer(WebServices.java:405)
> at
> org.apache.hadoop.yarn.server.webapp.WebServices.getNodeHttpAddress(WebServices.java:373)
> at
> org.apache.hadoop.yarn.server.webapp.LogServlet.getContainerLogsInfo(LogServlet.java:268)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getContainerLogs(HsWebServices.java:461)
>
> {code}
> On Analyzing, found WebServices#getContainer uses doAs using UGI created by
> createRemoteUser(end user) to access RM#ApplicationClientProtocol which does
> not work. Need to use createProxyUser to do the same.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10335) Improve scheduling of containers based on node health
[
https://issues.apache.org/jira/browse/YARN-10335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153234#comment-17153234
]
Bibin Chundatt commented on YARN-10335:
---
Thank you [~cyrusjackson25] for working in this
Few comments:
# Refer NodeHealthStatus for how the records needs to implemented. Define as
abstract and also add comments.
# setNodeResources -> setNodeResourceScore also rename the variables too.
# Finding addition description detail why did we add this ??
{noformat}
optional string node_health_description = 4;
{noformat}
# NodeHealthService instead of *getNodeHealthDetails* we could add
updateNodeHealthDetails
# Add Visibility Annotation as private
> Improve scheduling of containers based on node health
> -
>
> Key: YARN-10335
> URL: https://issues.apache.org/jira/browse/YARN-10335
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Bibin Chundatt
>Assignee: Cyrus Jackson
>Priority: Major
> Attachments: YARN-10335.001.patch
>
>
> YARN-7494 supports providing interface to choose nodeset for scheduler
> allocation.
> We could leverage the same to support allocation of containers based on node
> health value send from nodemanagers
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10340) HsWebServices getContainerReport uses loginUser instead of remoteUser to access ApplicationClientProtocol
[
https://issues.apache.org/jira/browse/YARN-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153225#comment-17153225
]
Tarun Parimi commented on YARN-10340:
-
[~prabhujoseph], The issue is because the HistoryClientService#initializeWebApp
instantiates the RPC client connection when creating the WebApp .
{code:java}
ApplicationClientProtocol appClientProtocol =
ClientRMProxy.createRMProxy(conf, ApplicationClientProtocol.class);
{code}
This RPC client proxy instance will only use the mapred ugi at the time of
creation and even for subsequent calls irrespective of doAs.
I made a code change to check by adding below method in HSWebServices and it
works with the correct ugi fixing the issue.
{code:java}
@Override
protected ContainerReport getContainerReport(
GetContainerReportRequest request) throws YarnException, IOException {
return ClientRMProxy.createRMProxy(conf,
ApplicationClientProtocol.class).getContainerReport(request).getContainerReport();
}
{code}
This creates a separate rpc client instance every time though which is not
efficient.
> HsWebServices getContainerReport uses loginUser instead of remoteUser to
> access ApplicationClientProtocol
> -
>
> Key: YARN-10340
> URL: https://issues.apache.org/jira/browse/YARN-10340
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Prabhu Joseph
>Assignee: Tarun Parimi
>Priority: Major
>
> HsWebServices getContainerReport uses loginUser instead of remoteUser to
> access ApplicationClientProtocol
>
> [http://:19888/ws/v1/history/containers/container_e03_1594030808801_0002_01_03/logs|http://pjoseph-secure-1.pjoseph-secure.root.hwx.site:19888/ws/v1/history/containers/container_e03_1594030808801_0002_01_03/logs]
> While accessing above link using systest user, the request fails saying
> mapred user does not have access to the job
>
> {code:java}
> 2020-07-06 14:02:59,178 WARN org.apache.hadoop.yarn.server.webapp.LogServlet:
> Could not obtain node HTTP address from provider.
> javax.ws.rs.WebApplicationException:
> org.apache.hadoop.yarn.exceptions.YarnException: User mapred does not have
> privilege to see this application application_1593997842459_0214
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getContainerReport(ClientRMService.java:516)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getContainerReport(ApplicationClientProtocolPBServiceImpl.java:466)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:639)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:985)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:913)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2882)
> at
> org.apache.hadoop.yarn.server.webapp.WebServices.rewrapAndThrowThrowable(WebServices.java:544)
> at
> org.apache.hadoop.yarn.server.webapp.WebServices.rewrapAndThrowException(WebServices.java:530)
> at
> org.apache.hadoop.yarn.server.webapp.WebServices.getContainer(WebServices.java:405)
> at
> org.apache.hadoop.yarn.server.webapp.WebServices.getNodeHttpAddress(WebServices.java:373)
> at
> org.apache.hadoop.yarn.server.webapp.LogServlet.getContainerLogsInfo(LogServlet.java:268)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getContainerLogs(HsWebServices.java:461)
>
> {code}
> On Analyzing, found WebServices#getContainer uses doAs using UGI created by
> createRemoteUser(end user) to access RM#ApplicationClientProtocol which does
> not work. Need to use createProxyUser to do the same.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10344) Sync netty versions in hadoop-yarn-csi
[
https://issues.apache.org/jira/browse/YARN-10344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka reassigned YARN-10344:
Assignee: Akira Ajisaka
> Sync netty versions in hadoop-yarn-csi
> --
>
> Key: YARN-10344
> URL: https://issues.apache.org/jira/browse/YARN-10344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>
> netty-all is now 4.1.50.Final but the other netty libraries are 4.1.42.Final:
> {noformat}
> [INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ hadoop-yarn-csi
> ---
> [INFO] org.apache.hadoop:hadoop-yarn-csi:jar:3.3.0
> [INFO] +- com.google.guava:guava:jar:20.0:compile
> [INFO] +- com.google.protobuf:protobuf-java:jar:3.6.1:compile
> [INFO] +- io.netty:netty-all:jar:4.1.50.Final:compile
> [INFO] +- io.grpc:grpc-core:jar:1.26.0:compile
> [INFO] | +- io.grpc:grpc-api:jar:1.26.0:compile (version selected from
> constraint [1.26.0,1.26.0])
> [INFO] | | +- io.grpc:grpc-context:jar:1.26.0:compile
> [INFO] | | +-
> com.google.errorprone:error_prone_annotations:jar:2.3.3:compile
> [INFO] | | \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
> [INFO] | +- com.google.code.gson:gson:jar:2.2.4:compile
> [INFO] | +- com.google.android:annotations:jar:4.1.1.4:compile
> [INFO] | +- io.perfmark:perfmark-api:jar:0.19.0:compile
> [INFO] | +- io.opencensus:opencensus-api:jar:0.24.0:compile
> [INFO] | \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.24.0:compile
> [INFO] +- io.grpc:grpc-protobuf:jar:1.26.0:compile
> [INFO] | +- com.google.api.grpc:proto-google-common-protos:jar:1.12.0:compile
> [INFO] | \- io.grpc:grpc-protobuf-lite:jar:1.26.0:compile
> [INFO] +- io.grpc:grpc-stub:jar:1.26.0:compile
> [INFO] +- io.grpc:grpc-netty:jar:1.26.0:compile
> [INFO] | +- io.netty:netty-codec-http2:jar:4.1.42.Final:compile (version
> selected from constraint [4.1.42.Final,4.1.42.Final])
> [INFO] | | +- io.netty:netty-common:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-buffer:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-transport:jar:4.1.42.Final:compile
> [INFO] | | | \- io.netty:netty-resolver:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-codec:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-handler:jar:4.1.42.Final:compile
> [INFO] | | \- io.netty:netty-codec-http:jar:4.1.42.Final:compile
> [INFO] | \- io.netty:netty-handler-proxy:jar:4.1.42.Final:compile
> [INFO] | \- io.netty:netty-codec-socks:jar:4.1.42.Final:compile
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10344) Sync netty versions in hadoop-yarn-csi
[
https://issues.apache.org/jira/browse/YARN-10344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10344:
-
Affects Version/s: 3.3.0
> Sync netty versions in hadoop-yarn-csi
> --
>
> Key: YARN-10344
> URL: https://issues.apache.org/jira/browse/YARN-10344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build
>Affects Versions: 3.3.0
>Reporter: Akira Ajisaka
>Priority: Major
>
> netty-all is now 4.1.50.Final but the other netty libraries are 4.1.42.Final:
> {noformat}
> [INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ hadoop-yarn-csi
> ---
> [INFO] org.apache.hadoop:hadoop-yarn-csi:jar:3.3.0
> [INFO] +- com.google.guava:guava:jar:20.0:compile
> [INFO] +- com.google.protobuf:protobuf-java:jar:3.6.1:compile
> [INFO] +- io.netty:netty-all:jar:4.1.50.Final:compile
> [INFO] +- io.grpc:grpc-core:jar:1.26.0:compile
> [INFO] | +- io.grpc:grpc-api:jar:1.26.0:compile (version selected from
> constraint [1.26.0,1.26.0])
> [INFO] | | +- io.grpc:grpc-context:jar:1.26.0:compile
> [INFO] | | +-
> com.google.errorprone:error_prone_annotations:jar:2.3.3:compile
> [INFO] | | \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
> [INFO] | +- com.google.code.gson:gson:jar:2.2.4:compile
> [INFO] | +- com.google.android:annotations:jar:4.1.1.4:compile
> [INFO] | +- io.perfmark:perfmark-api:jar:0.19.0:compile
> [INFO] | +- io.opencensus:opencensus-api:jar:0.24.0:compile
> [INFO] | \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.24.0:compile
> [INFO] +- io.grpc:grpc-protobuf:jar:1.26.0:compile
> [INFO] | +- com.google.api.grpc:proto-google-common-protos:jar:1.12.0:compile
> [INFO] | \- io.grpc:grpc-protobuf-lite:jar:1.26.0:compile
> [INFO] +- io.grpc:grpc-stub:jar:1.26.0:compile
> [INFO] +- io.grpc:grpc-netty:jar:1.26.0:compile
> [INFO] | +- io.netty:netty-codec-http2:jar:4.1.42.Final:compile (version
> selected from constraint [4.1.42.Final,4.1.42.Final])
> [INFO] | | +- io.netty:netty-common:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-buffer:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-transport:jar:4.1.42.Final:compile
> [INFO] | | | \- io.netty:netty-resolver:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-codec:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-handler:jar:4.1.42.Final:compile
> [INFO] | | \- io.netty:netty-codec-http:jar:4.1.42.Final:compile
> [INFO] | \- io.netty:netty-handler-proxy:jar:4.1.42.Final:compile
> [INFO] | \- io.netty:netty-codec-socks:jar:4.1.42.Final:compile
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10344) Sync netty versions in hadoop-yarn-csi
[
https://issues.apache.org/jira/browse/YARN-10344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated YARN-10344:
-
Summary: Sync netty versions in hadoop-yarn-csi (was: Sync netty-*
versions in hadoop-yarn-csi)
> Sync netty versions in hadoop-yarn-csi
> --
>
> Key: YARN-10344
> URL: https://issues.apache.org/jira/browse/YARN-10344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build
>Reporter: Akira Ajisaka
>Priority: Major
>
> netty-all is now 4.1.50.Final but the other netty libraries are 4.1.42.Final:
> {noformat}
> [INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ hadoop-yarn-csi
> ---
> [INFO] org.apache.hadoop:hadoop-yarn-csi:jar:3.3.0
> [INFO] +- com.google.guava:guava:jar:20.0:compile
> [INFO] +- com.google.protobuf:protobuf-java:jar:3.6.1:compile
> [INFO] +- io.netty:netty-all:jar:4.1.50.Final:compile
> [INFO] +- io.grpc:grpc-core:jar:1.26.0:compile
> [INFO] | +- io.grpc:grpc-api:jar:1.26.0:compile (version selected from
> constraint [1.26.0,1.26.0])
> [INFO] | | +- io.grpc:grpc-context:jar:1.26.0:compile
> [INFO] | | +-
> com.google.errorprone:error_prone_annotations:jar:2.3.3:compile
> [INFO] | | \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
> [INFO] | +- com.google.code.gson:gson:jar:2.2.4:compile
> [INFO] | +- com.google.android:annotations:jar:4.1.1.4:compile
> [INFO] | +- io.perfmark:perfmark-api:jar:0.19.0:compile
> [INFO] | +- io.opencensus:opencensus-api:jar:0.24.0:compile
> [INFO] | \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.24.0:compile
> [INFO] +- io.grpc:grpc-protobuf:jar:1.26.0:compile
> [INFO] | +- com.google.api.grpc:proto-google-common-protos:jar:1.12.0:compile
> [INFO] | \- io.grpc:grpc-protobuf-lite:jar:1.26.0:compile
> [INFO] +- io.grpc:grpc-stub:jar:1.26.0:compile
> [INFO] +- io.grpc:grpc-netty:jar:1.26.0:compile
> [INFO] | +- io.netty:netty-codec-http2:jar:4.1.42.Final:compile (version
> selected from constraint [4.1.42.Final,4.1.42.Final])
> [INFO] | | +- io.netty:netty-common:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-buffer:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-transport:jar:4.1.42.Final:compile
> [INFO] | | | \- io.netty:netty-resolver:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-codec:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-handler:jar:4.1.42.Final:compile
> [INFO] | | \- io.netty:netty-codec-http:jar:4.1.42.Final:compile
> [INFO] | \- io.netty:netty-handler-proxy:jar:4.1.42.Final:compile
> [INFO] | \- io.netty:netty-codec-socks:jar:4.1.42.Final:compile
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10344) Sync netty-* versions in hadoop-yarn-csi
[
https://issues.apache.org/jira/browse/YARN-10344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153178#comment-17153178
]
Akira Ajisaka commented on YARN-10344:
--
Given hadoop-yarn-csi module directly depends on netty-all, we can safely
exclude the other netty dependencies.
> Sync netty-* versions in hadoop-yarn-csi
>
>
> Key: YARN-10344
> URL: https://issues.apache.org/jira/browse/YARN-10344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build
>Reporter: Akira Ajisaka
>Priority: Major
>
> netty-all is now 4.1.50.Final but the other netty libraries are 4.1.42.Final:
> {noformat}
> [INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ hadoop-yarn-csi
> ---
> [INFO] org.apache.hadoop:hadoop-yarn-csi:jar:3.3.0
> [INFO] +- com.google.guava:guava:jar:20.0:compile
> [INFO] +- com.google.protobuf:protobuf-java:jar:3.6.1:compile
> [INFO] +- io.netty:netty-all:jar:4.1.50.Final:compile
> [INFO] +- io.grpc:grpc-core:jar:1.26.0:compile
> [INFO] | +- io.grpc:grpc-api:jar:1.26.0:compile (version selected from
> constraint [1.26.0,1.26.0])
> [INFO] | | +- io.grpc:grpc-context:jar:1.26.0:compile
> [INFO] | | +-
> com.google.errorprone:error_prone_annotations:jar:2.3.3:compile
> [INFO] | | \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
> [INFO] | +- com.google.code.gson:gson:jar:2.2.4:compile
> [INFO] | +- com.google.android:annotations:jar:4.1.1.4:compile
> [INFO] | +- io.perfmark:perfmark-api:jar:0.19.0:compile
> [INFO] | +- io.opencensus:opencensus-api:jar:0.24.0:compile
> [INFO] | \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.24.0:compile
> [INFO] +- io.grpc:grpc-protobuf:jar:1.26.0:compile
> [INFO] | +- com.google.api.grpc:proto-google-common-protos:jar:1.12.0:compile
> [INFO] | \- io.grpc:grpc-protobuf-lite:jar:1.26.0:compile
> [INFO] +- io.grpc:grpc-stub:jar:1.26.0:compile
> [INFO] +- io.grpc:grpc-netty:jar:1.26.0:compile
> [INFO] | +- io.netty:netty-codec-http2:jar:4.1.42.Final:compile (version
> selected from constraint [4.1.42.Final,4.1.42.Final])
> [INFO] | | +- io.netty:netty-common:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-buffer:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-transport:jar:4.1.42.Final:compile
> [INFO] | | | \- io.netty:netty-resolver:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-codec:jar:4.1.42.Final:compile
> [INFO] | | +- io.netty:netty-handler:jar:4.1.42.Final:compile
> [INFO] | | \- io.netty:netty-codec-http:jar:4.1.42.Final:compile
> [INFO] | \- io.netty:netty-handler-proxy:jar:4.1.42.Final:compile
> [INFO] | \- io.netty:netty-codec-socks:jar:4.1.42.Final:compile
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10344) Sync netty-* versions in hadoop-yarn-csi
Akira Ajisaka created YARN-10344:
Summary: Sync netty-* versions in hadoop-yarn-csi
Key: YARN-10344
URL: https://issues.apache.org/jira/browse/YARN-10344
Project: Hadoop YARN
Issue Type: Bug
Components: build
Reporter: Akira Ajisaka
netty-all is now 4.1.50.Final but the other netty libraries are 4.1.42.Final:
{noformat}
[INFO] --- maven-dependency-plugin:3.0.2:tree (default-cli) @ hadoop-yarn-csi
---
[INFO] org.apache.hadoop:hadoop-yarn-csi:jar:3.3.0
[INFO] +- com.google.guava:guava:jar:20.0:compile
[INFO] +- com.google.protobuf:protobuf-java:jar:3.6.1:compile
[INFO] +- io.netty:netty-all:jar:4.1.50.Final:compile
[INFO] +- io.grpc:grpc-core:jar:1.26.0:compile
[INFO] | +- io.grpc:grpc-api:jar:1.26.0:compile (version selected from
constraint [1.26.0,1.26.0])
[INFO] | | +- io.grpc:grpc-context:jar:1.26.0:compile
[INFO] | | +- com.google.errorprone:error_prone_annotations:jar:2.3.3:compile
[INFO] | | \- org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
[INFO] | +- com.google.code.gson:gson:jar:2.2.4:compile
[INFO] | +- com.google.android:annotations:jar:4.1.1.4:compile
[INFO] | +- io.perfmark:perfmark-api:jar:0.19.0:compile
[INFO] | +- io.opencensus:opencensus-api:jar:0.24.0:compile
[INFO] | \- io.opencensus:opencensus-contrib-grpc-metrics:jar:0.24.0:compile
[INFO] +- io.grpc:grpc-protobuf:jar:1.26.0:compile
[INFO] | +- com.google.api.grpc:proto-google-common-protos:jar:1.12.0:compile
[INFO] | \- io.grpc:grpc-protobuf-lite:jar:1.26.0:compile
[INFO] +- io.grpc:grpc-stub:jar:1.26.0:compile
[INFO] +- io.grpc:grpc-netty:jar:1.26.0:compile
[INFO] | +- io.netty:netty-codec-http2:jar:4.1.42.Final:compile (version
selected from constraint [4.1.42.Final,4.1.42.Final])
[INFO] | | +- io.netty:netty-common:jar:4.1.42.Final:compile
[INFO] | | +- io.netty:netty-buffer:jar:4.1.42.Final:compile
[INFO] | | +- io.netty:netty-transport:jar:4.1.42.Final:compile
[INFO] | | | \- io.netty:netty-resolver:jar:4.1.42.Final:compile
[INFO] | | +- io.netty:netty-codec:jar:4.1.42.Final:compile
[INFO] | | +- io.netty:netty-handler:jar:4.1.42.Final:compile
[INFO] | | \- io.netty:netty-codec-http:jar:4.1.42.Final:compile
[INFO] | \- io.netty:netty-handler-proxy:jar:4.1.42.Final:compile
[INFO] | \- io.netty:netty-codec-socks:jar:4.1.42.Final:compile
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10343) Legacy RM UI should include labeled metrics for allocated, total, and reserved resources.
[ https://issues.apache.org/jira/browse/YARN-10343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153098#comment-17153098 ] Eric Payne commented on YARN-10343: --- I see two major problems with this structure: - Resource metrics in the overview section are not accurate. It could even show 0 resources if all nodes are labeled. - Even in the apps section where used resources are show, reserved resources are not show if the app runs in a labeled queue. In the following cluster, there are 4 NodeManagers. Each node has 5GB memory and 10 VCores. Each node in the cluster is labeled. This results in the overview section showing that the cluster has 0 resources for used, total, and reserved metrics, even though the apps section shows memory and vcores being used. However, even in the apps seciton, when an appis reserving memory and vcores, it shows as 0 if the app is running in a labeled queue. !Screen Shot 2020-07-07 at 1.03.26 PM.png! In the nodes page, you can also see that the resources are being used even though the overview section looks like there are no resources in the cluster: !Screen Shot 2020-07-07 at 1.00.22 PM.png! > Legacy RM UI should include labeled metrics for allocated, total, and > reserved resources. > - > > Key: YARN-10343 > URL: https://issues.apache.org/jira/browse/YARN-10343 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.10.0, 3.2.1, 3.1.3 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > Attachments: Screen Shot 2020-07-07 at 1.00.22 PM.png, Screen Shot > 2020-07-07 at 1.03.26 PM.png > > > The current legacy RM UI only includes resources metrics for the default > partition. If a cluster has labeled nodes, those are not included in the > resource metrics for allocated, total, and reserved resources. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10343) Legacy RM UI should include labeled metrics for allocated, total, and reserved resources.
[ https://issues.apache.org/jira/browse/YARN-10343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated YARN-10343: -- Attachment: Screen Shot 2020-07-07 at 1.00.22 PM.png > Legacy RM UI should include labeled metrics for allocated, total, and > reserved resources. > - > > Key: YARN-10343 > URL: https://issues.apache.org/jira/browse/YARN-10343 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.10.0, 3.2.1, 3.1.3 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > Attachments: Screen Shot 2020-07-07 at 1.00.22 PM.png, Screen Shot > 2020-07-07 at 1.03.26 PM.png > > > The current legacy RM UI only includes resources metrics for the default > partition. If a cluster has labeled nodes, those are not included in the > resource metrics for allocated, total, and reserved resources. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10343) Legacy RM UI should include labeled metrics for allocated, total, and reserved resources.
[ https://issues.apache.org/jira/browse/YARN-10343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated YARN-10343: -- Attachment: Screen Shot 2020-07-07 at 1.03.26 PM.png > Legacy RM UI should include labeled metrics for allocated, total, and > reserved resources. > - > > Key: YARN-10343 > URL: https://issues.apache.org/jira/browse/YARN-10343 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.10.0, 3.2.1, 3.1.3 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > Attachments: Screen Shot 2020-07-07 at 1.03.26 PM.png > > > The current legacy RM UI only includes resources metrics for the default > partition. If a cluster has labeled nodes, those are not included in the > resource metrics for allocated, total, and reserved resources. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10343) Legacy RM UI should include labeled metrics for allocated, total, and reserved resources.
[ https://issues.apache.org/jira/browse/YARN-10343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated YARN-10343: -- Description: The current legacy RM UI only includes resources metrics for the default partition. If a cluster has labeled nodes, those are not included in the resource metrics for allocated, total, and reserved resources. > Legacy RM UI should include labeled metrics for allocated, total, and > reserved resources. > - > > Key: YARN-10343 > URL: https://issues.apache.org/jira/browse/YARN-10343 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.10.0, 3.2.1, 3.1.3 >Reporter: Eric Payne >Assignee: Eric Payne >Priority: Major > > The current legacy RM UI only includes resources metrics for the default > partition. If a cluster has labeled nodes, those are not included in the > resource metrics for allocated, total, and reserved resources. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10343) Legacy RM UI should include labeled metrics for allocated, total, and reserved resources.
Eric Payne created YARN-10343: - Summary: Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Key: YARN-10343 URL: https://issues.apache.org/jira/browse/YARN-10343 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 3.1.3, 3.2.1, 2.10.0 Reporter: Eric Payne Assignee: Eric Payne -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10341) Yarn Service Container Completed event doesn't get processed
[
https://issues.apache.org/jira/browse/YARN-10341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152870#comment-17152870
]
Billie Rinaldi commented on YARN-10341:
---
I agree, continue looks better here.
> Yarn Service Container Completed event doesn't get processed
> -
>
> Key: YARN-10341
> URL: https://issues.apache.org/jira/browse/YARN-10341
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Critical
> Attachments: YARN-10341.001.patch
>
>
> If there 10 workers running and if containers get killed , after a while we
> see that there are just 9 workers runnning. This is due to CONTAINER
> COMPLETED Event is not processed on AM side.
> Issue is in below code:
> {code:java}
> public void onContainersCompleted(List statuses) {
> for (ContainerStatus status : statuses) {
> ContainerId containerId = status.getContainerId();
> ComponentInstance instance =
> liveInstances.get(status.getContainerId());
> if (instance == null) {
> LOG.warn(
> "Container {} Completed. No component instance exists.
> exitStatus={}. diagnostics={} ",
> containerId, status.getExitStatus(), status.getDiagnostics());
> return;
> }
> ComponentEvent event =
> new ComponentEvent(instance.getCompName(), CONTAINER_COMPLETED)
> .setStatus(status).setInstance(instance)
> .setContainerId(containerId);
> dispatcher.getEventHandler().handle(event);
> }
> {code}
> If component instance doesnt exist for a container, it doesnt iterate over
> other containers as its returning from method
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8047) RMWebApp make external class pluggable
[
https://issues.apache.org/jira/browse/YARN-8047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152854#comment-17152854
]
Hadoop QA commented on YARN-8047:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
52s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
22m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
11s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m
12s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
2s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
15s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 30s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 211 unchanged - 0 fixed = 212 total (was 211) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 26s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
10s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
56s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
12s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 93m 43s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}208m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed
[jira] [Commented] (YARN-10106) Yarn logs CLI filtering by application attempt
[
https://issues.apache.org/jira/browse/YARN-10106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152777#comment-17152777
]
Benjamin Teke commented on YARN-10106:
--
Hi [~mhudaky],
Thanks for the patch! It generally looks good, some minor nits, mostly about
the logs:
* LogsCLI#259: __ changing ApplicationId to Application ID doesn't seem
necessary, it's safer to leave it as is (otherwise it can cause a conflict
during a backport)
* LogsCLI#271: For debugging purposes I would log the appId and the
appAttemptId.getApplicationId, as the latter doesn't come from a command line
parameter
* LogsCLI#284 and 291: Same here, logging containerId.getApplicationAttemptId
or simply containerId could be useful (and it was logged before)
* For the whole patch: _is belonging to_ should be _belongs to_
> Yarn logs CLI filtering by application attempt
> --
>
> Key: YARN-10106
> URL: https://issues.apache.org/jira/browse/YARN-10106
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Adam Antal
>Assignee: Hudáky Márton Gyula
>Priority: Trivial
> Attachments: YARN-10106.001.patch, YARN-10106.002.patch,
> YARN-10106.003.patch, YARN-10106.004.patch, YARN-10106.005.patch
>
>
> {{ContainerLogsRequest}} got a new parameter in YARN-10101, which is the
> {{applicationAttempt}} - we can use this new parameter in Yarn logs CLI as
> well to filter by application attempt.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10341) Yarn Service Container Completed event doesn't get processed
[
https://issues.apache.org/jira/browse/YARN-10341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152749#comment-17152749
]
Brahma Reddy Battula commented on YARN-10341:
-
[~BilwaST] thanks for reporting.
Looks to be hidden bug here..Patch lgtm.. Try to add one UT for this.
> Yarn Service Container Completed event doesn't get processed
> -
>
> Key: YARN-10341
> URL: https://issues.apache.org/jira/browse/YARN-10341
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Critical
> Attachments: YARN-10341.001.patch
>
>
> If there 10 workers running and if containers get killed , after a while we
> see that there are just 9 workers runnning. This is due to CONTAINER
> COMPLETED Event is not processed on AM side.
> Issue is in below code:
> {code:java}
> public void onContainersCompleted(List statuses) {
> for (ContainerStatus status : statuses) {
> ContainerId containerId = status.getContainerId();
> ComponentInstance instance =
> liveInstances.get(status.getContainerId());
> if (instance == null) {
> LOG.warn(
> "Container {} Completed. No component instance exists.
> exitStatus={}. diagnostics={} ",
> containerId, status.getExitStatus(), status.getDiagnostics());
> return;
> }
> ComponentEvent event =
> new ComponentEvent(instance.getCompName(), CONTAINER_COMPLETED)
> .setStatus(status).setInstance(instance)
> .setContainerId(containerId);
> dispatcher.getEventHandler().handle(event);
> }
> {code}
> If component instance doesnt exist for a container, it doesnt iterate over
> other containers as its returning from method
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10337) TestRMHATimelineCollectors fails on hadoop trunk
[
https://issues.apache.org/jira/browse/YARN-10337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152719#comment-17152719
]
Bilwa S T commented on YARN-10337:
--
Thanks [~prabhujoseph]
> TestRMHATimelineCollectors fails on hadoop trunk
>
>
> Key: YARN-10337
> URL: https://issues.apache.org/jira/browse/YARN-10337
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test, yarn
>Reporter: Ahmed Hussein
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10337.001.patch
>
>
> {{TestRMHATimelineCollectors}} has been failing on trunk. I see it frequently
> in the qbt reports and the yetus reprts
> {code:bash}
> [INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.95
> s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR]
> testRebuildCollectorDataOnFailover(org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors)
> Time elapsed: 5.615 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover(TestRMHATimelineCollectors.java:105)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:80)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover:105
> NullPointer
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [ERROR] There are test failures.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8047) RMWebApp make external class pluggable
[ https://issues.apache.org/jira/browse/YARN-8047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152718#comment-17152718 ] Bilwa S T commented on YARN-8047: - Thanks [~prabhujoseph] for reviewing. I have handled all comments . Is it ok if i add testcase in different jira ? . > RMWebApp make external class pluggable > -- > > Key: YARN-8047 > URL: https://issues.apache.org/jira/browse/YARN-8047 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bibin Chundatt >Assignee: Bilwa S T >Priority: Minor > Attachments: YARN-8047-001.patch, YARN-8047-002.patch, > YARN-8047-003.patch, YARN-8047.004.patch, YARN-8047.005.patch, > YARN-8047.006.patch > > > JIra should make sure we should be able to plugin webservices and web pages > of scheduler in Resourcemanager > * RMWebApp allow to bind external classes > * RMController allow to plugin scheduler classes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8047) RMWebApp make external class pluggable
[ https://issues.apache.org/jira/browse/YARN-8047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilwa S T updated YARN-8047: Attachment: YARN-8047.006.patch > RMWebApp make external class pluggable > -- > > Key: YARN-8047 > URL: https://issues.apache.org/jira/browse/YARN-8047 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bibin Chundatt >Assignee: Bilwa S T >Priority: Minor > Attachments: YARN-8047-001.patch, YARN-8047-002.patch, > YARN-8047-003.patch, YARN-8047.004.patch, YARN-8047.005.patch, > YARN-8047.006.patch > > > JIra should make sure we should be able to plugin webservices and web pages > of scheduler in Resourcemanager > * RMWebApp allow to bind external classes > * RMController allow to plugin scheduler classes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10337) TestRMHATimelineCollectors fails on hadoop trunk
[
https://issues.apache.org/jira/browse/YARN-10337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152693#comment-17152693
]
Hudson commented on YARN-10337:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18416 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18416/])
YARN-10337. Fix failing testcase TestRMHATimelineCollectors. (pjoseph: rev
2bbd00dff498027241a5d84713f4e3f13ac45e65)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMHATimelineCollectors.java
> TestRMHATimelineCollectors fails on hadoop trunk
>
>
> Key: YARN-10337
> URL: https://issues.apache.org/jira/browse/YARN-10337
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test, yarn
>Reporter: Ahmed Hussein
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10337.001.patch
>
>
> {{TestRMHATimelineCollectors}} has been failing on trunk. I see it frequently
> in the qbt reports and the yetus reprts
> {code:bash}
> [INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.95
> s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR]
> testRebuildCollectorDataOnFailover(org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors)
> Time elapsed: 5.615 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover(TestRMHATimelineCollectors.java:105)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:80)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover:105
> NullPointer
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [ERROR] There are test failures.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail:
[jira] [Updated] (YARN-10337) TestRMHATimelineCollectors fails on hadoop trunk
[
https://issues.apache.org/jira/browse/YARN-10337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated YARN-10337:
-
Parent: YARN-9802
Issue Type: Sub-task (was: Bug)
> TestRMHATimelineCollectors fails on hadoop trunk
>
>
> Key: YARN-10337
> URL: https://issues.apache.org/jira/browse/YARN-10337
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test, yarn
>Reporter: Ahmed Hussein
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10337.001.patch
>
>
> {{TestRMHATimelineCollectors}} has been failing on trunk. I see it frequently
> in the qbt reports and the yetus reprts
> {code:bash}
> [INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.95
> s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR]
> testRebuildCollectorDataOnFailover(org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors)
> Time elapsed: 5.615 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover(TestRMHATimelineCollectors.java:105)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:80)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover:105
> NullPointer
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [ERROR] There are test failures.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10337) TestRMHATimelineCollectors fails on hadoop trunk
[
https://issues.apache.org/jira/browse/YARN-10337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152680#comment-17152680
]
Prabhu Joseph commented on YARN-10337:
--
Thanks [~BilwaST] for the patch.
+1, have committed it to trunk. Will resolve the Jira.
> TestRMHATimelineCollectors fails on hadoop trunk
>
>
> Key: YARN-10337
> URL: https://issues.apache.org/jira/browse/YARN-10337
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, yarn
>Reporter: Ahmed Hussein
>Assignee: Bilwa S T
>Priority: Major
> Attachments: YARN-10337.001.patch
>
>
> {{TestRMHATimelineCollectors}} has been failing on trunk. I see it frequently
> in the qbt reports and the yetus reprts
> {code:bash}
> [INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.95
> s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors
> [ERROR]
> testRebuildCollectorDataOnFailover(org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors)
> Time elapsed: 5.615 s <<< ERROR!
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover(TestRMHATimelineCollectors.java:105)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:80)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> [INFO]
> [INFO] Results:
> [INFO]
> [ERROR] Errors:
> [ERROR] TestRMHATimelineCollectors.testRebuildCollectorDataOnFailover:105
> NullPointer
> [INFO]
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
> [INFO]
> [ERROR] There are test failures.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10342) [UI1] Provide a way to hide Tools section in Web UIv1
Andras Gyori created YARN-10342: --- Summary: [UI1] Provide a way to hide Tools section in Web UIv1 Key: YARN-10342 URL: https://issues.apache.org/jira/browse/YARN-10342 Project: Hadoop YARN Issue Type: Improvement Reporter: Andras Gyori Assignee: Andras Gyori The Tools section in web UI1 might contain sensitive information, which should ideally be hidden from end users. We should provide a configurable value to hide it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10339) Timeline Client in Nodemanager gets 403 errors when simple auth is used in kerberos environments
[
https://issues.apache.org/jira/browse/YARN-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152643#comment-17152643
]
Hadoop QA commented on YARN-10339:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
1s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
6s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
29s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
55s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue}
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
no findbugs output file (findbugsXml.xml) {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
44s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 29s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 322 unchanged - 1 fixed = 323 total (was 323) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
28s{color} | {color:blue}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests has
no data from findbugs {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
1s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
9s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
49s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
59s{color} | {color:green} hadoop-yarn-server-applicationhistoryservice in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
13s{color} | {color:green} hadoop-yarn-server-tests in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color}
[jira] [Comment Edited] (YARN-10339) Timeline Client in Nodemanager gets 403 errors when simple auth is used in kerberos environments
[
https://issues.apache.org/jira/browse/YARN-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152548#comment-17152548
]
Tarun Parimi edited comment on YARN-10339 at 7/7/20, 8:17 AM:
--
Thanks [~prabhujoseph] . When atsv1 is enabled, delegation tokens are used even
when auth is simple. I made changes in this patch, to add Timeline Delegation
Token only when auth is kerberos. And fixed unit test failures and checkstyle.
was (Author: tarunparimi):
Thanks [~prabhujoseph] . When atsv1 is enabled, delegation tokens are used even
when auth is simple. I made changes in this patch, to add Timeline Delegation
Token only when auth is simple. And fixed unit test failures and checkstyle.
> Timeline Client in Nodemanager gets 403 errors when simple auth is used in
> kerberos environments
>
>
> Key: YARN-10339
> URL: https://issues.apache.org/jira/browse/YARN-10339
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineclient
>Affects Versions: 3.1.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Attachments: YARN-10339.001.patch, YARN-10339.002.patch
>
>
> We get below errors in NodeManager logs whenever we set
> yarn.timeline-service.http-authentication.type=simple in a cluster which has
> kerberos enabled. There are use cases where simple auth is used only in
> timeline server for convenience although kerberos is enabled.
> {code:java}
> 2020-05-20 20:06:30,181 ERROR impl.TimelineV2ClientImpl
> (TimelineV2ClientImpl.java:putObjects(321)) - Response from the timeline
> server is not successful, HTTP error code: 403, Server response:
> {"exception":"ForbiddenException","message":"java.lang.Exception: The owner
> of the posted timeline entities is not
> set","javaClassName":"org.apache.hadoop.yarn.webapp.ForbiddenException"}
> {code}
> This seems to affect the NM timeline publisher which uses
> TimelineV2ClientImpl. Doing a simple auth directly to timeline service via
> curl works fine. So this issue is in the authenticator configuration in
> timeline client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10339) Timeline Client in Nodemanager gets 403 errors when simple auth is used in kerberos environments
[
https://issues.apache.org/jira/browse/YARN-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152548#comment-17152548
]
Tarun Parimi commented on YARN-10339:
-
Thanks [~prabhujoseph] . When atsv1 is enabled, delegation tokens are used even
when auth is simple. I made changes in this patch, to add Timeline Delegation
Token only when auth is simple. And fixed unit test failures and checkstyle.
> Timeline Client in Nodemanager gets 403 errors when simple auth is used in
> kerberos environments
>
>
> Key: YARN-10339
> URL: https://issues.apache.org/jira/browse/YARN-10339
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineclient
>Affects Versions: 3.1.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Attachments: YARN-10339.001.patch, YARN-10339.002.patch
>
>
> We get below errors in NodeManager logs whenever we set
> yarn.timeline-service.http-authentication.type=simple in a cluster which has
> kerberos enabled. There are use cases where simple auth is used only in
> timeline server for convenience although kerberos is enabled.
> {code:java}
> 2020-05-20 20:06:30,181 ERROR impl.TimelineV2ClientImpl
> (TimelineV2ClientImpl.java:putObjects(321)) - Response from the timeline
> server is not successful, HTTP error code: 403, Server response:
> {"exception":"ForbiddenException","message":"java.lang.Exception: The owner
> of the posted timeline entities is not
> set","javaClassName":"org.apache.hadoop.yarn.webapp.ForbiddenException"}
> {code}
> This seems to affect the NM timeline publisher which uses
> TimelineV2ClientImpl. Doing a simple auth directly to timeline service via
> curl works fine. So this issue is in the authenticator configuration in
> timeline client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10339) Timeline Client in Nodemanager gets 403 errors when simple auth is used in kerberos environments
[
https://issues.apache.org/jira/browse/YARN-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tarun Parimi updated YARN-10339:
Attachment: YARN-10339.002.patch
> Timeline Client in Nodemanager gets 403 errors when simple auth is used in
> kerberos environments
>
>
> Key: YARN-10339
> URL: https://issues.apache.org/jira/browse/YARN-10339
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineclient
>Affects Versions: 3.1.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Attachments: YARN-10339.001.patch, YARN-10339.002.patch
>
>
> We get below errors in NodeManager logs whenever we set
> yarn.timeline-service.http-authentication.type=simple in a cluster which has
> kerberos enabled. There are use cases where simple auth is used only in
> timeline server for convenience although kerberos is enabled.
> {code:java}
> 2020-05-20 20:06:30,181 ERROR impl.TimelineV2ClientImpl
> (TimelineV2ClientImpl.java:putObjects(321)) - Response from the timeline
> server is not successful, HTTP error code: 403, Server response:
> {"exception":"ForbiddenException","message":"java.lang.Exception: The owner
> of the posted timeline entities is not
> set","javaClassName":"org.apache.hadoop.yarn.webapp.ForbiddenException"}
> {code}
> This seems to affect the NM timeline publisher which uses
> TimelineV2ClientImpl. Doing a simple auth directly to timeline service via
> curl works fine. So this issue is in the authenticator configuration in
> timeline client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10339) Timeline Client in Nodemanager gets 403 errors when simple auth is used in kerberos environments
[
https://issues.apache.org/jira/browse/YARN-10339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152527#comment-17152527
]
Prabhu Joseph commented on YARN-10339:
--
[~tarunparimi] Thanks for the patch. The patch looks good. Can you fix the
checkstyle issues and failing testcase.
> Timeline Client in Nodemanager gets 403 errors when simple auth is used in
> kerberos environments
>
>
> Key: YARN-10339
> URL: https://issues.apache.org/jira/browse/YARN-10339
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineclient
>Affects Versions: 3.1.0
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Attachments: YARN-10339.001.patch
>
>
> We get below errors in NodeManager logs whenever we set
> yarn.timeline-service.http-authentication.type=simple in a cluster which has
> kerberos enabled. There are use cases where simple auth is used only in
> timeline server for convenience although kerberos is enabled.
> {code:java}
> 2020-05-20 20:06:30,181 ERROR impl.TimelineV2ClientImpl
> (TimelineV2ClientImpl.java:putObjects(321)) - Response from the timeline
> server is not successful, HTTP error code: 403, Server response:
> {"exception":"ForbiddenException","message":"java.lang.Exception: The owner
> of the posted timeline entities is not
> set","javaClassName":"org.apache.hadoop.yarn.webapp.ForbiddenException"}
> {code}
> This seems to affect the NM timeline publisher which uses
> TimelineV2ClientImpl. Doing a simple auth directly to timeline service via
> curl works fine. So this issue is in the authenticator configuration in
> timeline client.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10340) HsWebServices getContainerReport uses loginUser instead of remoteUser to access ApplicationClientProtocol
[
https://issues.apache.org/jira/browse/YARN-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17152501#comment-17152501
]
Tarun Parimi commented on YARN-10340:
-
[~prabhujoseph],[~brahmareddy] The WebServices#getContainer works properly when
called by RMWebServices or AHSWebServices. This could be because they use their
own ClientRMService and ApplicationHistoryClientService respectively.
But HsWebServices now uses ClientRMService remotely and so doAs doesn't work
here as expected.
> HsWebServices getContainerReport uses loginUser instead of remoteUser to
> access ApplicationClientProtocol
> -
>
> Key: YARN-10340
> URL: https://issues.apache.org/jira/browse/YARN-10340
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Prabhu Joseph
>Assignee: Tarun Parimi
>Priority: Major
>
> HsWebServices getContainerReport uses loginUser instead of remoteUser to
> access ApplicationClientProtocol
>
> [http://:19888/ws/v1/history/containers/container_e03_1594030808801_0002_01_03/logs|http://pjoseph-secure-1.pjoseph-secure.root.hwx.site:19888/ws/v1/history/containers/container_e03_1594030808801_0002_01_03/logs]
> While accessing above link using systest user, the request fails saying
> mapred user does not have access to the job
>
> {code:java}
> 2020-07-06 14:02:59,178 WARN org.apache.hadoop.yarn.server.webapp.LogServlet:
> Could not obtain node HTTP address from provider.
> javax.ws.rs.WebApplicationException:
> org.apache.hadoop.yarn.exceptions.YarnException: User mapred does not have
> privilege to see this application application_1593997842459_0214
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getContainerReport(ClientRMService.java:516)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getContainerReport(ApplicationClientProtocolPBServiceImpl.java:466)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:639)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:528)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:985)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:913)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1876)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2882)
> at
> org.apache.hadoop.yarn.server.webapp.WebServices.rewrapAndThrowThrowable(WebServices.java:544)
> at
> org.apache.hadoop.yarn.server.webapp.WebServices.rewrapAndThrowException(WebServices.java:530)
> at
> org.apache.hadoop.yarn.server.webapp.WebServices.getContainer(WebServices.java:405)
> at
> org.apache.hadoop.yarn.server.webapp.WebServices.getNodeHttpAddress(WebServices.java:373)
> at
> org.apache.hadoop.yarn.server.webapp.LogServlet.getContainerLogsInfo(LogServlet.java:268)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getContainerLogs(HsWebServices.java:461)
>
> {code}
> On Analyzing, found WebServices#getContainer uses doAs using UGI created by
> createRemoteUser(end user) to access RM#ApplicationClientProtocol which does
> not work. Need to use createProxyUser to do the same.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
