[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257587#comment-15257587 ] Rohith Sharma K S commented on YARN-4478: - It got failed yesterday too!! [HadoopQA Report| https://issues.apache.org/jira/secure/EditComment!default.jspa?id=12958048=15256199]. I am not very sure whether it is docker container setting environment problem!! > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Test > Components: test, yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257581#comment-15257581 ] Allen Wittenauer commented on YARN-4478: Have you seen it fail consistently in the past 10 months we've been running precommit under Docker? > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Test > Components: test, yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4555) TestDefaultContainerExecutor#testContainerLaunchError fails on non-englihsh locale environment
[
https://issues.apache.org/jira/browse/YARN-4555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257573#comment-15257573
]
Hadoop QA commented on YARN-4555:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 59s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed with
JDK v1.8.0_92. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 30s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed with
JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 1s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12780967/YARN-4555.1.patch |
| JIRA Issue | YARN-4555 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f0103d68cb50 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8eadd71 |
| Default Java | 1.7.0_95 |
| Multi-JDK versions |
[jira] [Updated] (YARN-4989) TestWorkPreservingRMRestart#testCapacitySchedulerRecovery fails intermittently
[
https://issues.apache.org/jira/browse/YARN-4989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4989:
Issue Type: Test (was: Bug)
> TestWorkPreservingRMRestart#testCapacitySchedulerRecovery fails
> intermittently
> ---
>
> Key: YARN-4989
> URL: https://issues.apache.org/jira/browse/YARN-4989
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Ajith S
>
> Sometimes TestWorkPreservingRMRestart#testCapacitySchedulerRecovery fails
> randomly.
> {noformat}
> java.lang.AssertionError: expected:<> but
> was:<>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart.checkCSLeafQueue(TestWorkPreservingRMRestart.java:289)
> at
> org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart.testCapacitySchedulerRecovery(TestWorkPreservingRMRestart.java:501)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S resolved YARN-4478. - Resolution: Fixed Hadoop Flags: Reviewed I closed the umbrella JIRA as fixed and Hadoop Flags has been set as Reviewed. And I have not put any *Fix version* number since these are gone in several branches like trunk/branch-2/branch-2.8 and some of the JIRA are back ported also. If any changes required to do for checkpoints for closing Umbrella JIRA, let me know so that I will do necessary correction in check points. > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Test > Components: test, yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-4478: Issue Type: Test (was: Bug) > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Test > Components: test, yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-4478: Component/s: test > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: test, yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257556#comment-15257556 ] Rohith Sharma K S commented on YARN-4478: - Curious to know that if precommit runs on docker, don't we face test failure like HADOOP-12687? > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4954) TestYarnClient.testReservationAPIs fails on machines with less than 4 GB available memory
[
https://issues.apache.org/jira/browse/YARN-4954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4954:
Issue Type: Test (was: Sub-task)
Parent: (was: YARN-4478)
> TestYarnClient.testReservationAPIs fails on machines with less than 4 GB
> available memory
> -
>
> Key: YARN-4954
> URL: https://issues.apache.org/jira/browse/YARN-4954
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Affects Versions: 3.0.0
>Reporter: Gergely Novák
>Assignee: Gergely Novák
>Priority: Minor
> Attachments: YARN-4954.001.patch
>
>
> TestYarnClient.testReservationAPIs sometimes fails with this error:
> {noformat}
> java.lang.AssertionError:
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The request cannot be satisfied
> at
> org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1254)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitReservation(ApplicationClientProtocolPBServiceImpl.java:457)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:515)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:637)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2422)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2418)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1742)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2416)
> Caused by:
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The request cannot be satisfied
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.IterativePlanner.computeJobAllocation(IterativePlanner.java:151)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.allocateUser(PlanningAlgorithm.java:64)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.createReservation(PlanningAlgorithm.java:140)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.TryManyReservationAgents.createReservation(TryManyReservationAgents.java:55)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy.createReservation(AlignedPlannerWithGreedy.java:84)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1237)
> ... 10 more
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient.testReservationAPIs(TestYarnClient.java:1227)
> {noformat}
> This is caused by really not having enough available memory to complete the
> reservation (4 * 1024 MB). In my opinion lowering the required memory (either
> by lowering the number of containers to 2, or the memory to 512 MB) would
> make the test more stable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4982) Test timeout :yarn-client testcase timeout and failures
[
https://issues.apache.org/jira/browse/YARN-4982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4982:
Issue Type: Test (was: Sub-task)
Parent: (was: YARN-4478)
> Test timeout :yarn-client testcase timeout and failures
> ---
>
> Key: YARN-4982
> URL: https://issues.apache.org/jira/browse/YARN-4982
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Reporter: Bibin A Chundatt
>Priority: Blocker
> Attachments: 0001-YARN-4982.patch, 0002-YARN-4982.patch,
> 0003-YARN-4982.patch
>
>
> https://builds.apache.org/job/PreCommit-YARN-Build/11088/testReport/junit/org.apache.hadoop.yarn.client.api.impl/TestAMRMProxy/testAMRMProxyE2E/
> In hadoop-yarn-client package test {{TestAMRMProxy}} testcase timeout always
> {noformat}
> java.lang.Exception: test timed out after 6 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:160)
> at com.sun.proxy.$Proxy85.getNewApplication(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getNewApplication(YarnClientImpl.java:227)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.createApplication(YarnClientImpl.java:235)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMProxy.createApp(TestAMRMProxy.java:367)
> at
> org.apache.hadoop.yarn.client.api.impl.TestAMRMProxy.testAMRMProxyE2E(TestAMRMProxy.java:110)
> {noformat}
> Other classes having similar failures
> org.apache.hadoop.yarn.client.cli.TestYarnCLI
> org.apache.hadoop.yarn.client.api.impl.TestYarnClient
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
> org.apache.hadoop.yarn.client.api.impl.TestNMClien
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4974) Random test failure:TestRMApplicationHistoryWriter#testRMWritingMassiveHistoryForCapacitySche
[
https://issues.apache.org/jira/browse/YARN-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4974:
Component/s: test
> Random test
> failure:TestRMApplicationHistoryWriter#testRMWritingMassiveHistoryForCapacitySche
> -
>
> Key: YARN-4974
> URL: https://issues.apache.org/jira/browse/YARN-4974
> Project: Hadoop YARN
> Issue Type: Test
> Components: test, yarn
>Reporter: Bibin A Chundatt
>
> https://builds.apache.org/job/PreCommit-YARN-Build/11128/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt
> {noformat}
> Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 196.959 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
> testRMWritingMassiveHistoryForCapacitySche(org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter)
> Time elapsed: 125.174 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter.testRMWritingMassiveHistory(TestRMApplicationHistoryWriter.java:441)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter.testRMWritingMassiveHistoryForCapacitySche(TestRMApplicationHistoryWriter.java:383)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4947) Test timeout is happening for TestRMWebServicesNodes
[ https://issues.apache.org/jira/browse/YARN-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-4947: Component/s: test > Test timeout is happening for TestRMWebServicesNodes > > > Key: YARN-4947 > URL: https://issues.apache.org/jira/browse/YARN-4947 > Project: Hadoop YARN > Issue Type: Test > Components: test >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-4947.patch, 0002-YARN-4947.patch, > 0003-YARN-4947.patch, 0004-YARN-4947.patch > > > Testcase timeout for TestRMWebServicesNodes is happening after YARN-4893 > [timeout|https://builds.apache.org/job/PreCommit-YARN-Build/11044/testReport/] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4627) TestRMWebServicesAppsModification fails with BindException
[
https://issues.apache.org/jira/browse/YARN-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4627:
Issue Type: Test (was: Sub-task)
Parent: (was: YARN-4478)
> TestRMWebServicesAppsModification fails with BindException
> --
>
> Key: YARN-4627
> URL: https://issues.apache.org/jira/browse/YARN-4627
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Reporter: Rohith Sharma K S
>
> In the build
> [report|https://builds.apache.org/job/PreCommit-YARN-Build/10367/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt]
> , TestRMWebServicesAppsModification fails with BindExceptions.
> {noformat}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
> Tests run: 44, Failures: 0, Errors: 27, Skipped: 0, Time elapsed: 90.028 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
> testSingleAppKillInvalidId[1](org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification)
> Time elapsed: 0.009 sec <<< ERROR!
> com.sun.jersey.test.framework.spi.container.TestContainerException:
> java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at
> org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:413)
> at
> org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:384)
> at
> org.glassfish.grizzly.nio.transport.TCPNIOTransport.bind(TCPNIOTransport.java:375)
> at
> org.glassfish.grizzly.http.server.NetworkListener.start(NetworkListener.java:549)
> at
> org.glassfish.grizzly.http.server.HttpServer.start(HttpServer.java:255)
> at
> com.sun.jersey.api.container.grizzly2.GrizzlyServerFactory.createHttpServer(GrizzlyServerFactory.java:326)
> at
> com.sun.jersey.api.container.grizzly2.GrizzlyServerFactory.createHttpServer(GrizzlyServerFactory.java:343)
> at
> com.sun.jersey.test.framework.spi.container.grizzly2.web.GrizzlyWebTestContainerFactory$GrizzlyWebTestContainer.instantiateGrizzlyWebServer(GrizzlyWebTestContainerFactory.java:219)
> at
> com.sun.jersey.test.framework.spi.container.grizzly2.web.GrizzlyWebTestContainerFactory$GrizzlyWebTestContainer.(GrizzlyWebTestContainerFactory.java:129)
> at
> com.sun.jersey.test.framework.spi.container.grizzly2.web.GrizzlyWebTestContainerFactory$GrizzlyWebTestContainer.(GrizzlyWebTestContainerFactory.java:86)
> at
> com.sun.jersey.test.framework.spi.container.grizzly2.web.GrizzlyWebTestContainerFactory.create(GrizzlyWebTestContainerFactory.java:79)
> at
> com.sun.jersey.test.framework.JerseyTest.getContainer(JerseyTest.java:342)
> at com.sun.jersey.test.framework.JerseyTest.(JerseyTest.java:217)
> at
> org.apache.hadoop.yarn.webapp.JerseyTestBase.(JerseyTestBase.java:30)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification.(TestRMWebServicesAppsModification.java:277)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4974) Random test failure:TestRMApplicationHistoryWriter#testRMWritingMassiveHistoryForCapacitySche
[
https://issues.apache.org/jira/browse/YARN-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4974:
Issue Type: Test (was: Sub-task)
Parent: (was: YARN-4478)
> Random test
> failure:TestRMApplicationHistoryWriter#testRMWritingMassiveHistoryForCapacitySche
> -
>
> Key: YARN-4974
> URL: https://issues.apache.org/jira/browse/YARN-4974
> Project: Hadoop YARN
> Issue Type: Test
> Components: yarn
>Reporter: Bibin A Chundatt
>
> https://builds.apache.org/job/PreCommit-YARN-Build/11128/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_77.txt
> {noformat}
> Tests run: 7, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 196.959 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
> testRMWritingMassiveHistoryForCapacitySche(org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter)
> Time elapsed: 125.174 sec <<< FAILURE!
> java.lang.AssertionError: null
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertTrue(Assert.java:52)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter.testRMWritingMassiveHistory(TestRMApplicationHistoryWriter.java:441)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter.testRMWritingMassiveHistoryForCapacitySche(TestRMApplicationHistoryWriter.java:383)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4947) Test timeout is happening for TestRMWebServicesNodes
[ https://issues.apache.org/jira/browse/YARN-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-4947: Issue Type: Test (was: Sub-task) Parent: (was: YARN-4478) > Test timeout is happening for TestRMWebServicesNodes > > > Key: YARN-4947 > URL: https://issues.apache.org/jira/browse/YARN-4947 > Project: Hadoop YARN > Issue Type: Test > Components: test >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-4947.patch, 0002-YARN-4947.patch, > 0003-YARN-4947.patch, 0004-YARN-4947.patch > > > Testcase timeout for TestRMWebServicesNodes is happening after YARN-4893 > [timeout|https://builds.apache.org/job/PreCommit-YARN-Build/11044/testReport/] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4453) TestMiniYarnClusterNodeUtilization occasionally times out in trunk
[
https://issues.apache.org/jira/browse/YARN-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4453:
Issue Type: Test (was: Sub-task)
Parent: (was: YARN-4478)
> TestMiniYarnClusterNodeUtilization occasionally times out in trunk
> --
>
> Key: YARN-4453
> URL: https://issues.apache.org/jira/browse/YARN-4453
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Reporter: Sunil G
>
> TestMiniYarnClusterNodeUtilization failures are observed in few test runs in
> YARN-4293.
> In local also, same test case is timing out.
> {noformat}
> java.lang.Exception: test timed out after 6 milliseconds
> at java.lang.Thread.sleep(Native Method)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:158)
> at com.sun.proxy.$Proxy85.nodeHeartbeat(Unknown Source)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:113)
> {noformat}
> YARN-3980, where this test are added, reported few timed-out cases. I think
> this is to be investigated because its not looks good to increase timeout for
> tests, if tests fail.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[ https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257550#comment-15257550 ] Allen Wittenauer commented on YARN-4478: You might as well close INFRA-11150 as won't fix given precommit runs in a docker container. > [Umbrella] : Track all the Test failures in YARN > > > Key: YARN-4478 > URL: https://issues.apache.org/jira/browse/YARN-4478 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Rohith Sharma K S > > Recently many test cases are failing either timed out or new bug fix caused > impact. Many test faiures JIRA are raised and are in progress. > This is to track all the test failures JIRA's -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3568) TestAMRMTokens should use some random port
[
https://issues.apache.org/jira/browse/YARN-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-3568:
Issue Type: Test (was: Sub-task)
Parent: (was: YARN-4478)
> TestAMRMTokens should use some random port
> --
>
> Key: YARN-3568
> URL: https://issues.apache.org/jira/browse/YARN-3568
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Affects Versions: 2.8.0
>Reporter: Gera Shegalov
>Assignee: Takashi Ohnishi
> Attachments: YARN-3568.1.patch
>
>
> Since the default port is used for yarn.resourcemanager.scheduler.address, if
> we already run a pseudo-distributed cluster on the same development machine,
> the test fails like this:
> {code}
> testMasterKeyRollOver[0](org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens)
> Time elapsed: 1.511 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.net.BindException: Problem binding to [0.0.0.0:8030]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:413)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:590)
> at org.apache.hadoop.ipc.Server.(Server.java:2340)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:534)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:509)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.createServer(RpcServerFactoryPBImpl.java:169)
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:132)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
> at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.serviceStart(ApplicationMasterService.java:140)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:586)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:996)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1037)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1033)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1033)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1073)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens.testMasterKeyRollOver(TestAMRMTokens.java:235)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3568) TestAMRMTokens should use some random port
[
https://issues.apache.org/jira/browse/YARN-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-3568:
Component/s: test
> TestAMRMTokens should use some random port
> --
>
> Key: YARN-3568
> URL: https://issues.apache.org/jira/browse/YARN-3568
> Project: Hadoop YARN
> Issue Type: Test
> Components: test
>Affects Versions: 2.8.0
>Reporter: Gera Shegalov
>Assignee: Takashi Ohnishi
> Attachments: YARN-3568.1.patch
>
>
> Since the default port is used for yarn.resourcemanager.scheduler.address, if
> we already run a pseudo-distributed cluster on the same development machine,
> the test fails like this:
> {code}
> testMasterKeyRollOver[0](org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens)
> Time elapsed: 1.511 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.net.BindException: Problem binding to [0.0.0.0:8030]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:413)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:590)
> at org.apache.hadoop.ipc.Server.(Server.java:2340)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:534)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:509)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.createServer(RpcServerFactoryPBImpl.java:169)
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:132)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
> at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.serviceStart(ApplicationMasterService.java:140)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:586)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:996)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1037)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1033)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1033)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1073)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens.testMasterKeyRollOver(TestAMRMTokens.java:235)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4572) TestCapacityScheduler#testHeadRoomCalculationWithDRC failing
[
https://issues.apache.org/jira/browse/YARN-4572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4572:
Component/s: test
> TestCapacityScheduler#testHeadRoomCalculationWithDRC failing
>
>
> Key: YARN-4572
> URL: https://issues.apache.org/jira/browse/YARN-4572
> Project: Hadoop YARN
> Issue Type: Test
> Components: test, yarn
>Reporter: Bibin A Chundatt
> Attachments: YARN-4572.1.patch
>
>
> {noformat}
> Tests run: 46, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 127.996 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> testHeadRoomCalculationWithDRC(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler)
> Time elapsed: 0.189 sec <<< FAILURE!
> java.lang.AssertionError: expected:<6144> but was:<16384>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testHeadRoomCalculationWithDRC(TestCapacityScheduler.java:3041)
> {noformat}
> https://builds.apache.org/job/PreCommit-YARN-Build/10204/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
> https://builds.apache.org/job/PreCommit-YARN-Build/10204/testReport/
> Failed in jdk8 locally the same is passing
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4572) TestCapacityScheduler#testHeadRoomCalculationWithDRC failing
[
https://issues.apache.org/jira/browse/YARN-4572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4572:
Issue Type: Test (was: Sub-task)
Parent: (was: YARN-4478)
> TestCapacityScheduler#testHeadRoomCalculationWithDRC failing
>
>
> Key: YARN-4572
> URL: https://issues.apache.org/jira/browse/YARN-4572
> Project: Hadoop YARN
> Issue Type: Test
> Components: test, yarn
>Reporter: Bibin A Chundatt
> Attachments: YARN-4572.1.patch
>
>
> {noformat}
> Tests run: 46, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 127.996 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
> testHeadRoomCalculationWithDRC(org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler)
> Time elapsed: 0.189 sec <<< FAILURE!
> java.lang.AssertionError: expected:<6144> but was:<16384>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler.testHeadRoomCalculationWithDRC(TestCapacityScheduler.java:3041)
> {noformat}
> https://builds.apache.org/job/PreCommit-YARN-Build/10204/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
> https://builds.apache.org/job/PreCommit-YARN-Build/10204/testReport/
> Failed in jdk8 locally the same is passing
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3150) [Documentation] Documenting the timeline service v2
[
https://issues.apache.org/jira/browse/YARN-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257547#comment-15257547
]
Hadoop QA commented on YARN-3150:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m 39s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
2s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 57s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 21s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 57s
{color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
54s {color} | {color:green} YARN-2928 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped branch modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s {color} | {color:green} YARN-2928 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s
{color} | {color:green} YARN-2928 passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 2s
{color} | {color:green} YARN-2928 passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 58s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 58s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patch modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s
{color} | {color:green} hadoop-yarn-api in the patch passed with JDK v1.8.0_92.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 59s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_92. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 38s
[jira] [Updated] (YARN-4555) TestDefaultContainerExecutor#testContainerLaunchError fails on non-englihsh locale environment
[
https://issues.apache.org/jira/browse/YARN-4555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4555:
Component/s: test
> TestDefaultContainerExecutor#testContainerLaunchError fails on non-englihsh
> locale environment
> --
>
> Key: YARN-4555
> URL: https://issues.apache.org/jira/browse/YARN-4555
> Project: Hadoop YARN
> Issue Type: Test
> Components: nodemanager, test
>Affects Versions: 2.7.1
>Reporter: Takashi Ohnishi
>Assignee: Takashi Ohnishi
>Priority: Minor
> Attachments: YARN-4555.1.patch
>
>
> In my env where LANG=ja_JP.UTF-8, the test fails with
> {code}
> ---
> Test set:
> org.apache.hadoop.yarn.server.nodemanager.TestDefaultContainerExecutor
> ---
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.286 sec <<<
> FAILURE! - in
> org.apache.hadoop.yarn.server.nodemanager.TestDefaultContainerExecutor
> testContainerLaunchError(org.apache.hadoop.yarn.server.nodemanager.TestDefaultContainerExecutor)
> Time elapsed: 1.149 sec <<< FAILURE!
> java.lang.AssertionError: Invalid Diagnostics message: Exception from
> container-launch.
> Container id: CONTAINER_ID
> Exit code: 127
> Exception message: bash:
> target/TestDefaultContainerExecutor/localDir/default_container_executor.sh:
> そのようなファイルやディレクトリはありません
> Stack trace: ExitCodeException exitCode=127: bash:
> target/TestDefaultContainerExecutor/localDir/default_container_executor.sh:
> そのようなファイルやディ>レクトリはありません
> {code}
> This is because the test code assertion assumes the English locale as below.
> {code}
> 250 public Object answer(InvocationOnMock invocationOnMock)
> 251 throws Throwable {
> 252 String diagnostics = (String)
> invocationOnMock.getArguments()[0];
> 253 assertTrue("Invalid Diagnostics message: " + diagnostics,
> 254 diagnostics.contains("No such file or directory"));
> 255 return null;
> 256 }
> {code}
> This exists on trunk, too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4556) TestFifoScheduler.testResourceOverCommit fails
[
https://issues.apache.org/jira/browse/YARN-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4556:
Issue Type: Test (was: Sub-task)
Parent: (was: YARN-4478)
> TestFifoScheduler.testResourceOverCommit fails
> ---
>
> Key: YARN-4556
> URL: https://issues.apache.org/jira/browse/YARN-4556
> Project: Hadoop YARN
> Issue Type: Test
> Components: scheduler, test
>Reporter: Akihiro Suda
>Assignee: Akihiro Suda
> Fix For: 2.8.0
>
> Attachments: YARN-4556-1.patch, YARN-4556-branch-2.7.001.patch
>
>
> From YARN-4548 Jenkins log:
> https://builds.apache.org/job/PreCommit-YARN-Build/10181/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
> {code}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
> Tests run: 16, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 31.004 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
> testResourceOverCommit(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler)
> Time elapsed: 4.746 sec <<< FAILURE!
> java.lang.AssertionError: expected:<-2048> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler.testResourceOverCommit(TestFifoScheduler.java:1142)
> {code}
> https://github.com/apache/hadoop/blob/8676a118a12165ae5a8b80a2a4596c133471ebc1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java#L1142
> It seems that Jenkins has been hitting this intermittently since April 2015
> https://www.google.com/search?q=TestFifoScheduler.testResourceOverCommit
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4555) TestDefaultContainerExecutor#testContainerLaunchError fails on non-englihsh locale environment
[
https://issues.apache.org/jira/browse/YARN-4555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4555:
Issue Type: Test (was: Sub-task)
Parent: (was: YARN-4478)
> TestDefaultContainerExecutor#testContainerLaunchError fails on non-englihsh
> locale environment
> --
>
> Key: YARN-4555
> URL: https://issues.apache.org/jira/browse/YARN-4555
> Project: Hadoop YARN
> Issue Type: Test
> Components: nodemanager, test
>Affects Versions: 2.7.1
>Reporter: Takashi Ohnishi
>Assignee: Takashi Ohnishi
>Priority: Minor
> Attachments: YARN-4555.1.patch
>
>
> In my env where LANG=ja_JP.UTF-8, the test fails with
> {code}
> ---
> Test set:
> org.apache.hadoop.yarn.server.nodemanager.TestDefaultContainerExecutor
> ---
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.286 sec <<<
> FAILURE! - in
> org.apache.hadoop.yarn.server.nodemanager.TestDefaultContainerExecutor
> testContainerLaunchError(org.apache.hadoop.yarn.server.nodemanager.TestDefaultContainerExecutor)
> Time elapsed: 1.149 sec <<< FAILURE!
> java.lang.AssertionError: Invalid Diagnostics message: Exception from
> container-launch.
> Container id: CONTAINER_ID
> Exit code: 127
> Exception message: bash:
> target/TestDefaultContainerExecutor/localDir/default_container_executor.sh:
> そのようなファイルやディレクトリはありません
> Stack trace: ExitCodeException exitCode=127: bash:
> target/TestDefaultContainerExecutor/localDir/default_container_executor.sh:
> そのようなファイルやディ>レクトリはありません
> {code}
> This is because the test code assertion assumes the English locale as below.
> {code}
> 250 public Object answer(InvocationOnMock invocationOnMock)
> 251 throws Throwable {
> 252 String diagnostics = (String)
> invocationOnMock.getArguments()[0];
> 253 assertTrue("Invalid Diagnostics message: " + diagnostics,
> 254 diagnostics.contains("No such file or directory"));
> 255 return null;
> 256 }
> {code}
> This exists on trunk, too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4351) Tests in h.y.c.TestGetGroups get failed on trunk
[ https://issues.apache.org/jira/browse/YARN-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-4351: Component/s: test > Tests in h.y.c.TestGetGroups get failed on trunk > > > Key: YARN-4351 > URL: https://issues.apache.org/jira/browse/YARN-4351 > Project: Hadoop YARN > Issue Type: Test > Components: test >Reporter: Junping Du > > From test report: > https://builds.apache.org/job/PreCommit-YARN-Build/9661/testReport/, we can > see there are several test failures for TestGetGroups. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4351) Tests in h.y.c.TestGetGroups get failed on trunk
[ https://issues.apache.org/jira/browse/YARN-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-4351: Issue Type: Test (was: Sub-task) Parent: (was: YARN-4478) > Tests in h.y.c.TestGetGroups get failed on trunk > > > Key: YARN-4351 > URL: https://issues.apache.org/jira/browse/YARN-4351 > Project: Hadoop YARN > Issue Type: Test > Components: test >Reporter: Junping Du > > From test report: > https://builds.apache.org/job/PreCommit-YARN-Build/9661/testReport/, we can > see there are several test failures for TestGetGroups. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4556) TestFifoScheduler.testResourceOverCommit fails
[
https://issues.apache.org/jira/browse/YARN-4556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-4556:
Component/s: test
> TestFifoScheduler.testResourceOverCommit fails
> ---
>
> Key: YARN-4556
> URL: https://issues.apache.org/jira/browse/YARN-4556
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler, test
>Reporter: Akihiro Suda
>Assignee: Akihiro Suda
> Fix For: 2.8.0
>
> Attachments: YARN-4556-1.patch, YARN-4556-branch-2.7.001.patch
>
>
> From YARN-4548 Jenkins log:
> https://builds.apache.org/job/PreCommit-YARN-Build/10181/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.8.0_66.txt
> {code}
> Running
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
> Tests run: 16, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 31.004 sec
> <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler
> testResourceOverCommit(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler)
> Time elapsed: 4.746 sec <<< FAILURE!
> java.lang.AssertionError: expected:<-2048> but was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler.testResourceOverCommit(TestFifoScheduler.java:1142)
> {code}
> https://github.com/apache/hadoop/blob/8676a118a12165ae5a8b80a2a4596c133471ebc1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java#L1142
> It seems that Jenkins has been hitting this intermittently since April 2015
> https://www.google.com/search?q=TestFifoScheduler.testResourceOverCommit
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4478) [Umbrella] : Track all the Test failures in YARN
[
https://issues.apache.org/jira/browse/YARN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257541#comment-15257541
]
Rohith Sharma K S commented on YARN-4478:
-
Thanks [~vinodkv] for your attention on this JIRA. Yes, I will close this
Umbrella Jira since it is going very long and I think compared to where we
started this tracking, test case runs seem better.
Recently I had discussion with [~sunilg] about various test failure root causes
and made observations from this Umbrella JIRA. Some of the test case failure
which are said to be *random*, seems to be fixable. I will share some of the
observations made while fixing/reviewing/committing test cases in this umbrella
JIRA.
Types of failures seen
# Yarn event model - AsyncDispatcher : Most of the random test failures seen in
this category. For example, After registering node to RM, asserting for cluster
resource from scheduler.
{code}
rm.start();
rm.registerNode("h1:1234", 5120);
assertEquals(5120,rm.getResourceScheduler().getClusterResource());
{code}
Many a times, contributors forget while writing test cases that yarn events are
async. Many random failures are because of these events processing delay which
seems running in local eclipse tests.
# System Settings : We have seen few test case fails regularly. Mainly because
of DNS configurations. See HADOOP-12687 and INFRA-11150. This I am not sure
how should it be resolved since neither code check in preferred since it breaks
RFC standards.
# As test cases were made running in parrellel, we ran into "Address bind
exception" issues.
# MockRM APIs : In MockRM many API's are there to submit job and lunch AM which
is added over time. Few such methods are internally waiting for some events to
happen and few others explicitly need to wait for these events from test case
(contributors has to take care of this). For test case writing, contributors
should be aware of MockRM#API what it does internally. And we have seen a mix
of these apis causing random failures.
For handling open issues in this Umbrella, I will detach it and make it as Test
bug. Let us handle these separately.
> [Umbrella] : Track all the Test failures in YARN
>
>
> Key: YARN-4478
> URL: https://issues.apache.org/jira/browse/YARN-4478
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Reporter: Rohith Sharma K S
>
> Recently many test cases are failing either timed out or new bug fix caused
> impact. Many test faiures JIRA are raised and are in progress.
> This is to track all the test failures JIRA's
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4390) Do surgical preemption based on reserved container in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257484#comment-15257484
]
Hadoop QA commented on YARN-4390:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 49s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 12 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 2s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 22s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
patch generated 30 new + 506 unchanged - 16 fixed = 536 total (was 522)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 17s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 30s
{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdk1.7.0_95
with JDK v1.7.0_95 generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2)
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 6s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_92. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 32s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 184m 15s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
[jira] [Commented] (YARN-4390) Do surgical preemption based on reserved container in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257433#comment-15257433
]
Jian He commented on YARN-4390:
---
that sounds ok to me. some other comments
- why does this need to be volatile ? totalKillableResources reference is not
updated anywhere
{code}
private volatile Resource totalKillableResources = Resource.newInstance(0,
0);
{code}
- I think the reason why the getters for these two fields are synchrnozied
before is to make sure they are consistent with each other(i.e. updated
together)
The patch removed the synchrnozed keyword for these two fields. This will lead
to inconsistent view to the callers
{code}
private volatile Resource unallocatedResource = Resource.newInstance(0, 0);
private volatile Resource totalResource;
{code}
- why is this exception changed to be ignored ? (combine this try-catch clause
with the one underneath)
{code}
try {
//invoke the preemption policy at a regular pace
//the policy will generate preemption or kill events
//managed by the dispatcher
invokePolicy();
} catch (YarnRuntimeException e) {
LOG.error("YarnRuntimeException raised while executing preemption"
+ " checker, skip this run..., exception=", e);
}
{code}
- code does not match comment? comment says not
considersReservedResourceWhenCalculateIdeal
{code}
* So only compute scalingFactor when we're not doing reserved container
* preemption. {@link ReservedContainerCandidateSelector} will limit total
* preempted resources less than permitted.
*/
float scalingFactor = 1.0F;
if (considersReservedResourceWhenCalculateIdeal && Resources.greaterThan(rc,
tot_guarant, totPreemptionNeeded, totalPreemptionAllowed)) {
scalingFactor = Resources.divide(rc, tot_guarant, totalPreemptionAllowed,
totPreemptionNeeded);
}
{code}
> Do surgical preemption based on reserved container in CapacityScheduler
> ---
>
> Key: YARN-4390
> URL: https://issues.apache.org/jira/browse/YARN-4390
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.0.0, 2.8.0, 2.7.3
>Reporter: Eric Payne
>Assignee: Wangda Tan
> Attachments: YARN-4390-design.1.pdf, YARN-4390-test-results.pdf,
> YARN-4390.1.patch, YARN-4390.2.patch, YARN-4390.3.branch-2.patch,
> YARN-4390.3.patch, YARN-4390.4.patch, YARN-4390.5.patch, YARN-4390.6.patch
>
>
> There are multiple reasons why preemption could unnecessarily preempt
> containers. One is that an app could be requesting a large container (say
> 8-GB), and the preemption monitor could conceivably preempt multiple
> containers (say 8, 1-GB containers) in order to fill the large container
> request. These smaller containers would then be rejected by the requesting AM
> and potentially given right back to the preempted app.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4905) Improve Yarn log Command line option to show log metadata
[ https://issues.apache.org/jira/browse/YARN-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-4905: Attachment: YARN-4905.4.patch > Improve Yarn log Command line option to show log metadata > - > > Key: YARN-4905 > URL: https://issues.apache.org/jira/browse/YARN-4905 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4905.1.patch, YARN-4905.2.patch, YARN-4905.3.patch, > YARN-4905.4.patch > > > Improve the Yarn log commandline to have "ls" command which can list > containers for which we have logs, list files within each container, along > with file size -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4905) Improve Yarn log Command line option to show log metadata
[ https://issues.apache.org/jira/browse/YARN-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257408#comment-15257408 ] Xuan Gong commented on YARN-4905: - Thanks for the detail review, vinod. Attached a new patch to address all of your comments. > Improve Yarn log Command line option to show log metadata > - > > Key: YARN-4905 > URL: https://issues.apache.org/jira/browse/YARN-4905 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4905.1.patch, YARN-4905.2.patch, YARN-4905.3.patch > > > Improve the Yarn log commandline to have "ls" command which can list > containers for which we have logs, list files within each container, along > with file size -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3150) [Documentation] Documenting the timeline service v2
[ https://issues.apache.org/jira/browse/YARN-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257377#comment-15257377 ] Sangjin Lee commented on YARN-3150: --- Sounds good. I'll do that when I have updated the patch next time. Thanks. > [Documentation] Documenting the timeline service v2 > --- > > Key: YARN-3150 > URL: https://issues.apache.org/jira/browse/YARN-3150 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Sangjin Lee > Labels: yarn-2928-1st-milestone > Attachments: TimelineServiceV2.html, YARN-3150-YARN-2928.01.patch, > YARN-3150-YARN-2928.02.patch, YARN-3150-YARN-2928.03.patch > > > Let's make sure we will have a document to describe what's new in TS v2, the > APIs, the client libs and so on. We should do better around documentation in > v2 than v1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3150) [Documentation] Documenting the timeline service v2
[ https://issues.apache.org/jira/browse/YARN-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257368#comment-15257368 ] Li Lu commented on YARN-3150: - Thanks [~sjlee0] for the prompt update! By saying update YARN-2928, I was actually meaning to post a comment about this JIRA, so that all interested parties in the watch list may come and take a look at the proposed documentation? We can also do this after we actually freeze the code and before the proposed branch merge. I'm fine with either ways. > [Documentation] Documenting the timeline service v2 > --- > > Key: YARN-3150 > URL: https://issues.apache.org/jira/browse/YARN-3150 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Sangjin Lee > Labels: yarn-2928-1st-milestone > Attachments: TimelineServiceV2.html, YARN-3150-YARN-2928.01.patch, > YARN-3150-YARN-2928.02.patch, YARN-3150-YARN-2928.03.patch > > > Let's make sure we will have a document to describe what's new in TS v2, the > APIs, the client libs and so on. We should do better around documentation in > v2 than v1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3150) [Documentation] Documenting the timeline service v2
[
https://issues.apache.org/jira/browse/YARN-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257361#comment-15257361
]
Sangjin Lee commented on YARN-3150:
---
Thanks [~gtCarrera9] for your review. I hope the latest version addresses most
of them. Please note that this is still not final. We're still looking into the
steps for loading the coprocessor. I'll update the doc when we have the
contents ready for that.
bq. 1. What does the white box outside the NM stand for? I'm a little bit
confused by it...
Do you mean the stack of boxes? It is a way to indicate "multiple worker nodes".
bq. 2. In the text I can see we're discussing that currently collectors are
launched as aux services inside NM. Are we planning to represent this
relationship in the figure? If not, maybe we can briefly say something to avoid
confusion on the arrow from NM to the collector.
I added more captions to indicate that the timeline collector is on the node
that's running the AM, and other (remote) NMs can write to that timeline
collector. Hopefully the captions make it clearer. I also added a few sentences
that describe that.
bq. 3. IIUC, the bold arrows in the figure mean write data flow, and the solid
ones mean read request? Maybe we want a simple legend for this, or explain it
in the caption?
Good suggestion. I made that change.
bq. For the configuration table, I think we can use bold or italic to mark new
configs? In this way we don't need a new column?
Another good suggestion. I made that change too.
bq. Because the whole doc is about YARN timeline service, I think it will be
helpful to say mapreduce.job.emit-timeline-data is in mapred-site?
I added that that property should be in {{mapred-site.xml}}.
bq. Shall we update the YARN-2928 JIRA for this?
I wasn't 100% sure what you meant by this. Did you mean we should update the
description, add a comment, or?
> [Documentation] Documenting the timeline service v2
> ---
>
> Key: YARN-3150
> URL: https://issues.apache.org/jira/browse/YARN-3150
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Sangjin Lee
> Labels: yarn-2928-1st-milestone
> Attachments: TimelineServiceV2.html, YARN-3150-YARN-2928.01.patch,
> YARN-3150-YARN-2928.02.patch, YARN-3150-YARN-2928.03.patch
>
>
> Let's make sure we will have a document to describe what's new in TS v2, the
> APIs, the client libs and so on. We should do better around documentation in
> v2 than v1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3150) [Documentation] Documenting the timeline service v2
[ https://issues.apache.org/jira/browse/YARN-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sangjin Lee updated YARN-3150: -- Attachment: YARN-3150-YARN-2928.03.patch Posted patch v.3 addressing most of Li's comments. > [Documentation] Documenting the timeline service v2 > --- > > Key: YARN-3150 > URL: https://issues.apache.org/jira/browse/YARN-3150 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Sangjin Lee > Labels: yarn-2928-1st-milestone > Attachments: TimelineServiceV2.html, YARN-3150-YARN-2928.01.patch, > YARN-3150-YARN-2928.02.patch, YARN-3150-YARN-2928.03.patch > > > Let's make sure we will have a document to describe what's new in TS v2, the > APIs, the client libs and so on. We should do better around documentation in > v2 than v1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4390) Do surgical preemption based on reserved container in CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257344#comment-15257344 ] Wangda Tan commented on YARN-4390: -- [~jianhe], [~sunilg], For your concerns regarding to performance impact of the new approach. Ran SLS test with the latest patch, mocked a 1000 nodes cluster, each node has 128G memory, typically the cluster runs 20K+ containers concurrently. For 98% cases, total time of each PCPP execution is less than 10 ms. Only few runs (out of 200+) use 10-25ms. Since time complexity of the approach is O\(n\), if we run a cluster with 10k nodes, theoretically it can take up to 200ms, which is also acceptable to me. Thoughts? > Do surgical preemption based on reserved container in CapacityScheduler > --- > > Key: YARN-4390 > URL: https://issues.apache.org/jira/browse/YARN-4390 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.0.0, 2.8.0, 2.7.3 >Reporter: Eric Payne >Assignee: Wangda Tan > Attachments: YARN-4390-design.1.pdf, YARN-4390-test-results.pdf, > YARN-4390.1.patch, YARN-4390.2.patch, YARN-4390.3.branch-2.patch, > YARN-4390.3.patch, YARN-4390.4.patch, YARN-4390.5.patch, YARN-4390.6.patch > > > There are multiple reasons why preemption could unnecessarily preempt > containers. One is that an app could be requesting a large container (say > 8-GB), and the preemption monitor could conceivably preempt multiple > containers (say 8, 1-GB containers) in order to fill the large container > request. These smaller containers would then be rejected by the requesting AM > and potentially given right back to the preempted app. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4676) Automatic and Asynchronous Decommissioning Nodes Status Tracking
[ https://issues.apache.org/jira/browse/YARN-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257315#comment-15257315 ] Daniel Zhi commented on YARN-4676: -- 1. For client-side timeout tracking, I assume you are talking about the "private int refreshNodes(long timeout)" method in RMAdminCLI.java, where the code continuously (every second) checks and waits for all decommissioning nodes to become decommissioned. For any remaining decommissioning nodes upon timeout, the client will send FORCEFUL decommission request. The code remains mostly the same except as RM also got the timeout (through RefreshNodesRequest()) and enforces such timeout, normally the client won't need to explicitly invoke FORCEFUL decommission as nodes will become DECOMMISSIONED by then. (Should server for some reason didn't turn the node into DECOMMISSIONED, client will force it). Does the combined behavior appear fine to you? 2. I am not very familiar with YARN internals when yarn.resourcemanager.recovery.enabled is true. My understanding of current (pre YARN-4676) behavior is: when RM restarts, NodesListManager creates a pseudo RMNodeImpl for excluded node and DECOMMISSION the node right away. Further, any invalid node will be rejected and told to SHUTDOWN inside registerNodeManager(). So when recovery is enabled, and RM restart during DECOMMISSIONING, although applications and containers are likely resumed, DECOMMISSIONING nodes will be DECOMMISSIONed right away. RM state store does not appear to serialize and restore RmNode but instead, after RM restart, RESYNC is replied in nodeHeartbeat() and new RmNode is created in following registerNodeManager(). So decommissioning start time get lost. To possibly resume the DECOMMISSIONING nodes, the decommissioning start time, possibly the DECOMMISSIONING state need to be stored and restored. I am not very familiar with RM state store but it appears non-trivial work involved, the cost-benefits justifications would also depend on how essential to resume DECOMMISSIONING nodes after RM restart. And if so whether it's better to create and handle it in a separate task/JIRA. > Automatic and Asynchronous Decommissioning Nodes Status Tracking > > > Key: YARN-4676 > URL: https://issues.apache.org/jira/browse/YARN-4676 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daniel Zhi >Assignee: Daniel Zhi > Labels: features > Attachments: GracefulDecommissionYarnNode.pdf, YARN-4676.004.patch, > YARN-4676.005.patch, YARN-4676.006.patch, YARN-4676.007.patch, > YARN-4676.008.patch, YARN-4676.009.patch, YARN-4676.010.patch, > YARN-4676.011.patch, YARN-4676.012.patch, YARN-4676.013.patch > > > DecommissioningNodeWatcher inside ResourceTrackingService tracks > DECOMMISSIONING nodes status automatically and asynchronously after > client/admin made the graceful decommission request. It tracks > DECOMMISSIONING nodes status to decide when, after all running containers on > the node have completed, will be transitioned into DECOMMISSIONED state. > NodesListManager detect and handle include and exclude list changes to kick > out decommission or recommission as necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4390) Do surgical preemption based on reserved container in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257292#comment-15257292
]
Wangda Tan commented on YARN-4390:
--
[~sunilg],
Thanks for comments:
bq. 1. In
CapacitySchedulerPreemptionUtils#deductPreemptableResourcesBasedSelectedCandidates,
partition is retrieved from SchedulerNode...
Retriving it from SchedulerNode is because scheduler handles node label changes
asynchronizely, and preemption policy should make decision based on scheduler's
state. And it doesn't have bad performance issue as well.
bq. 2. context.getQueueByPartition has a code path flow to return NULL...
Modified patch, now throw YarnRuntimeException when null happens and handled it
outside. (skips the preemption run)
bq. 3 3. From FifoCandidatesSelector,...
This is a good point. However, since we have different ways to calculate ideal
resource allocation, which belongs to selector, such as:
{code}
preemptableAmountCalculator.computeIdealAllocation(clusterResource,
totalPreemptionAllowed);
{code}
Deduct resources should happen after ideal allocation as well.
bq. 4. One doubt about the newly changed PreemptableResourceCalculator ctor...
Since PreemptableResourceCalculator belongs to Selector, PreemptionContext
doesn't know what's the correct value of
considersReservedResourceWhenCalculateIdeal.
Added a simple Java doc to PreemptableResourceCalculator constructor for more
details.
bq. 5. getNodesForPreemption in ReservedContainerCandidatesSelector has to scan
all nodes in cluster
Jian has similar concern, will run benchmark tests for this.
bq. 6. In ReservedContainerCandidatesSelector, containers are sorted against
launch time.
Launch time may not be correct because we should respect FIFO. I think we
should consider more about this, for example, application priority, etc. In
latest patch I updated it to sort based on containerId (reverse order), we can
do more work for it in a separate patch
bq. 7. As I see RMContainer is read-only interface. I feel setQueueName can be
moved out from this interface and place in RMContainer...
Done
bq. Minor comments ..
All addressed.
> Do surgical preemption based on reserved container in CapacityScheduler
> ---
>
> Key: YARN-4390
> URL: https://issues.apache.org/jira/browse/YARN-4390
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.0.0, 2.8.0, 2.7.3
>Reporter: Eric Payne
>Assignee: Wangda Tan
> Attachments: YARN-4390-design.1.pdf, YARN-4390-test-results.pdf,
> YARN-4390.1.patch, YARN-4390.2.patch, YARN-4390.3.branch-2.patch,
> YARN-4390.3.patch, YARN-4390.4.patch, YARN-4390.5.patch, YARN-4390.6.patch
>
>
> There are multiple reasons why preemption could unnecessarily preempt
> containers. One is that an app could be requesting a large container (say
> 8-GB), and the preemption monitor could conceivably preempt multiple
> containers (say 8, 1-GB containers) in order to fill the large container
> request. These smaller containers would then be rejected by the requesting AM
> and potentially given right back to the preempted app.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4390) Do surgical preemption based on reserved container in CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-4390: - Attachment: YARN-4390.6.patch Attached ver.6 patch. > Do surgical preemption based on reserved container in CapacityScheduler > --- > > Key: YARN-4390 > URL: https://issues.apache.org/jira/browse/YARN-4390 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.0.0, 2.8.0, 2.7.3 >Reporter: Eric Payne >Assignee: Wangda Tan > Attachments: YARN-4390-design.1.pdf, YARN-4390-test-results.pdf, > YARN-4390.1.patch, YARN-4390.2.patch, YARN-4390.3.branch-2.patch, > YARN-4390.3.patch, YARN-4390.4.patch, YARN-4390.5.patch, YARN-4390.6.patch > > > There are multiple reasons why preemption could unnecessarily preempt > containers. One is that an app could be requesting a large container (say > 8-GB), and the preemption monitor could conceivably preempt multiple > containers (say 8, 1-GB containers) in order to fill the large container > request. These smaller containers would then be rejected by the requesting AM > and potentially given right back to the preempted app. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4842) yarn logs command should not require the appOwner argument
[
https://issues.apache.org/jira/browse/YARN-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257236#comment-15257236
]
Hadoop QA commented on YARN-4842:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 25m 6s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 45s
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 10s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 15s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 53s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 41s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 2 new +
49 unchanged - 2 fixed = 51 total (was 51) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 48s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_92. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 26s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_92.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 20s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 33s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 204m 37s {color}
|
[jira] [Commented] (YARN-3150) [Documentation] Documenting the timeline service v2
[
https://issues.apache.org/jira/browse/YARN-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257177#comment-15257177
]
Li Lu commented on YARN-3150:
-
Thanks [~sjlee0] for the update! Quite close, some comments on the newly added
figure and some nits:
- For the figure (timeline_v2.jpg), some questions and comments:
1. What does the white box outside the NM stand for? I'm a little bit confused
by it...
2. In the text I can see we're discussing that currently collectors are
launched as aux services inside NM. Are we planning to represent this
relationship in the figure? If not, maybe we can briefly say something to avoid
confusion on the arrow from NM to the collector.
3. IIUC, the bold arrows in the figure mean write data flow, and the solid ones
mean read request? Maybe we want a simple legend for this, or explain it in the
caption?
- For the configuration table, I think we can use bold or italic to mark new
configs? In this way we don't need a new column?
- Because the whole doc is about YARN timeline service, I think it will be
helpful to say {{mapreduce.job.emit-timeline-data}} is in mapred-site?
Other parts LGTM. I'd encourage everyone interested to take a look at it. Shall
we update the YARN-2928 JIRA for this?
> [Documentation] Documenting the timeline service v2
> ---
>
> Key: YARN-3150
> URL: https://issues.apache.org/jira/browse/YARN-3150
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Sangjin Lee
> Labels: yarn-2928-1st-milestone
> Attachments: TimelineServiceV2.html, YARN-3150-YARN-2928.01.patch,
> YARN-3150-YARN-2928.02.patch
>
>
> Let's make sure we will have a document to describe what's new in TS v2, the
> APIs, the client libs and so on. We should do better around documentation in
> v2 than v1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4390) Do surgical preemption based on reserved container in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257178#comment-15257178
]
Wangda Tan commented on YARN-4390:
--
[~jianhe].
Thanks for review:
bq. Looks like the approach is to loop over almost all containers several times
on the cluster for every preemption cycle (3 secs by default), to see whether
some containers can be preempted to make room for the reserved container on the
same node. Will this cause too much overhead in a large cluster where we have a
large amount of containers ? A unit test may test the time cost for this mega
loop.
Will run benchmark test for this.
bq. Does this equal to node.getUnallocatedResource?
Yes it is, I cannot remember why I chose not to use it, will update.
bq. Insn't FifoCandidatesSelector the first selector and the selectedCandidates
is empty ?
No, it could be second selector as well.
{code}
if (selectCandidatesForResevedContainers) {
candidatesSelectionPolicies.add(
new ReservedContainerCandidatesSelector(this));
}
// initialize candidates preemption selection policies
candidatesSelectionPolicies.add(
new FifoCandidatesSelector(this));
{code}
> Do surgical preemption based on reserved container in CapacityScheduler
> ---
>
> Key: YARN-4390
> URL: https://issues.apache.org/jira/browse/YARN-4390
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.0.0, 2.8.0, 2.7.3
>Reporter: Eric Payne
>Assignee: Wangda Tan
> Attachments: YARN-4390-design.1.pdf, YARN-4390-test-results.pdf,
> YARN-4390.1.patch, YARN-4390.2.patch, YARN-4390.3.branch-2.patch,
> YARN-4390.3.patch, YARN-4390.4.patch, YARN-4390.5.patch
>
>
> There are multiple reasons why preemption could unnecessarily preempt
> containers. One is that an app could be requesting a large container (say
> 8-GB), and the preemption monitor could conceivably preempt multiple
> containers (say 8, 1-GB containers) in order to fill the large container
> request. These smaller containers would then be rejected by the requesting AM
> and potentially given right back to the preempted app.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4983) JVM and UGI metrics disappear after RM is once transitioned to standby mode
[ https://issues.apache.org/jira/browse/YARN-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257168#comment-15257168 ] Li Lu commented on YARN-4983: - Could not reproduce the UT failures locally. At least two UT failures (TestClientRMTokens and TestAMAuthorization) also appears in other unrelated JIRAs, such as YARN-3150 (a documentation patch). > JVM and UGI metrics disappear after RM is once transitioned to standby mode > --- > > Key: YARN-4983 > URL: https://issues.apache.org/jira/browse/YARN-4983 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Li Lu >Assignee: Li Lu > Attachments: YARN-4983-trunk.000.patch, YARN-4983-trunk.001.patch, > YARN-4983-trunk.002.patch > > > When get transitioned to standby, the RM will shutdown the existing metric > system and relaunch a new one. This will cause the jvm metrics and ugi > metrics to miss in the new metric system. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4958) The file localization process should allow for wildcards to reduce the application footprint in the state store
[
https://issues.apache.org/jira/browse/YARN-4958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257160#comment-15257160
]
Sangjin Lee commented on YARN-4958:
---
Thanks for your proposal [~templedf]! This is an important improvement. I took
a look at the patch, and have some early feedback. I haven't had a chance to
run the patch against a cluster yet, however.
A quick note on the shared cache: I don't think this patch will break the
shared cache. This patch goes to work mostly after the shared cache has done
its part. The only thing is that jobs with a set of resources that are heavily
drawn from the shared cache would not benefit from this patch as there would be
few classpath files that will be coming from the staging directory. But that's
for later...
Also, I think HADOOP-12747 is largely orthogonal. It merely gives users a
shorthand to address a set of jars.
More questions and comments:
- We'll need unit tests for this.
- I suppose a test will quickly confirm this, but does this work correctly if
we're dealing with a non-jar entry in the staging libjars directory? I just
wanted to confirm that it gets added to the container-side classpath
explicitly. The java classpath of "\*" does not include non-jar resources in
the directory.
- Would there be any cross-platform issues? Have you had a chance to test it on
Windows specifically? At first glance, there was nothing obvious that might be
a platform-specific issue, but it would be good to double check.
- This is just noting that the size of a wildcard entry (as in
{{mapreduce.job.cache.files.filesizes}}) would be reported as 0. This is an
existing behavior/issue with a directory entry.
(ClientDistributedCacheManager.java)
- l.242-243: Would it be simpler to reset {{current}} to the parent directory
and simply invoke {{ancestorsHaveExecutePermissions()}} on it instead? Then,
{{getFileStatus}} doesn't need to change, and the stat cache would also have
only real paths (i.e. no "*" paths). Thoughts?
(MRApps.java)
- l.323-329: Would there be a case where there can be multiple attempts for the
same directory? Is it for the case both "dir" and "dir/*" are included in
cache.files? I'm not sure if you're addressing a new concern or an existing one.
- l.338-341: Why would there be a wildcard for the paths (which come from
{{mapreduce.job.classpath.files}})?
(ContainerExecutor.java)
- This would apply to *any* entries that have the wildcard, and would effect
things like {{PWD/*}} too?
> The file localization process should allow for wildcards to reduce the
> application footprint in the state store
> ---
>
> Key: YARN-4958
> URL: https://issues.apache.org/jira/browse/YARN-4958
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Critical
> Attachments: YARN-4958.001.patch
>
>
> When using the -libjars option to add classes to the classpath, every library
> so added is explicitly listed in the {{ContainerLaunchContext}}'s local
> resources even though they're all uploaded to the same directory in HDFS.
> When using tools like Crunch without an uber JAR or when trying to take
> advantage of the shared cache, the number of libraries can be quite large.
> We've seen many cases where we had to turn down the max number of
> applications to prevent ZK from running out of heap because of the size of
> the state store entries.
> Rather than listing all files independently, this JIRA proposes to have the
> NM allow wildcards in the resource localization paths. Specifically, we
> propose to allow a path to have a final component (name) set to "*", which is
> interpreted by the NM as "download the full directory and link to every file
> in it from the job's working directory." This behavior is the same as the
> current behavior when using -libjars, but avoids explicitly listing every
> file.
> This JIRA does not attempt to provide more general purpose wildcards, such as
> "\*.jar" or "file\*", as having multiple entries for a single directory
> presents numerous logistical issues.
> This JIRA also does not attempt to integrate with the shared cache. That
> work will be left to a future JIRA. Specifically, this JIRA only applies
> when a full directory is uploaded. Currently the shared cache does not
> handle directory uploads.
> This JIRA proposes to allow for wildcards both in the internal processing of
> the -libjars switch and in paths added through the {{Job}} and
> {{DistributedCache}} classes.
> The proposed approach is to treat a path, "dir/\*", as "dir" for purposes of
> all file verification and localization. In the
[jira] [Commented] (YARN-4844) Upgrade fields of o.a.h.y.api.records.Resource from int32 to int64
[ https://issues.apache.org/jira/browse/YARN-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257158#comment-15257158 ] Wangda Tan commented on YARN-4844: -- bq. ...Given the debate about the extent of the changes we want to make, can we put a patch that changes the int32 to int64, adds getMemoryLong with a Private annotation(so that we can make changes later if we wish) and only fixes the pending memory check that was added in 2.8?... I agree size of the patch looks scary :-p, however, if you look into the patch, they're all very simple fixes, I don't think it will cause a lot of issues. You may feel better once I fixed all Jenkins issues. I have considered fix the pending resource calculation only, it looks hard to me. Because calculation of pending resource uses ResourceCalculator/ResourceUsage. And ResourceCalculator and related static methods of Resources used everywhere in RM. It's a good idea to me to mark get___Long to @Private, currently pending resource hasn't been exposed to application via Java API yet. Now it is only exposed in REST API which is fixed by the patch already. Thoughts? > Upgrade fields of o.a.h.y.api.records.Resource from int32 to int64 > -- > > Key: YARN-4844 > URL: https://issues.apache.org/jira/browse/YARN-4844 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-4844.1.patch, YARN-4844.2.patch > > > We use int32 for memory now, if a cluster has 10k nodes, each node has 210G > memory, we will get a negative total cluster memory. > And another case that easier overflows int32 is: we added all pending > resources of running apps to cluster's total pending resources. If a > problematic app requires too much resources (let's say 1M+ containers, each > of them has 3G containers), int32 will be not enough. > Even if we can cap each app's pending request, we cannot handle the case that > there're many running apps, each of them has capped but still significant > numbers of pending resources. > So we may possibly need to upgrade int32 memory field (could include v-cores > as well) to int64 to avoid integer overflow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4844) Upgrade fields of o.a.h.y.api.records.Resource from int32 to int64
[ https://issues.apache.org/jira/browse/YARN-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257142#comment-15257142 ] Wangda Tan commented on YARN-4844: -- bq. So the plan is to force users to change their usage of these APIs in some version of 3.x but not in 3.0.0 ? Regardless of debates about the first release of 3.x, let's assume it happens soon. The plan in my mind is to make sure incompatible API changes are get in when 3.x enters beta releases. We have a couple of other API changes on the way, in YARN for example, ATSv2, new web UI, etc. bq. Additionally we are not talking about use in production but rather making upstream apps change as needed to work with 3.x and over time stabilize 3.x. Per my understanding, changing from int to long won't affect downstream project a lot, it's an error which can be captured by compiler directly. And getMemory/getVCores should not be used intensively by downstream project. For example, MR uses only ~20 times of getMemory()/VCores for non-testing code. Which can be easily fixed. bq. Making an API change earlier rather than later is actually better as the API changes in this case have no relevance to production stability. I agree that it won't affect production stability. However, it adds additional overhead to development works which I don't want. > Upgrade fields of o.a.h.y.api.records.Resource from int32 to int64 > -- > > Key: YARN-4844 > URL: https://issues.apache.org/jira/browse/YARN-4844 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-4844.1.patch, YARN-4844.2.patch > > > We use int32 for memory now, if a cluster has 10k nodes, each node has 210G > memory, we will get a negative total cluster memory. > And another case that easier overflows int32 is: we added all pending > resources of running apps to cluster's total pending resources. If a > problematic app requires too much resources (let's say 1M+ containers, each > of them has 3G containers), int32 will be not enough. > Even if we can cap each app's pending request, we cannot handle the case that > there're many running apps, each of them has capped but still significant > numbers of pending resources. > So we may possibly need to upgrade int32 memory field (could include v-cores > as well) to int64 to avoid integer overflow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4983) JVM and UGI metrics disappear after RM is once transitioned to standby mode
[
https://issues.apache.org/jira/browse/YARN-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257107#comment-15257107
]
Hadoop QA commented on YARN-4983:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 6m 52s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 49s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 18s
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 59s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 30s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
43s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 32s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 5s
{color} | {color:red} root: patch generated 1 new + 172 unchanged - 1 fixed =
173 total (was 173) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 4s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 21s
{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed
with JDK v1.8.0_92. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 27s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 48s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_92.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 0s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_92. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 3s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_95.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 10s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}
[jira] [Commented] (YARN-4795) ContainerMetrics drops records
[
https://issues.apache.org/jira/browse/YARN-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257104#comment-15257104
]
Ray Chiang commented on YARN-4795:
--
Looks like re-launches only work when the patch status is "Patch Available":
https://builds.apache.org/job/PreCommit-YARN-Build/11211/consoleText
> ContainerMetrics drops records
> --
>
> Key: YARN-4795
> URL: https://issues.apache.org/jira/browse/YARN-4795
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-4795.001.patch, YARN-4795.002.patch
>
>
> The metrics2 system was implemented to deal with persistent sources.
> {{ContainerMetrics}} is an ephemeral source, and so it causes problems.
> Specifically, the {{ContainerMetrics}} only reports metrics once after the
> container has been stopped. This behavior is a problem because the metrics2
> system can ask sources for reports that will be quietly dropped by the sinks
> that care. (It's a metrics2 feature, not a bug.) If that final report is
> silently dropped, it's lost, because the {{ContainerMetrics}} won't report
> anything else ever anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4966) More improvement to get Container logs without specify nodeId
[
https://issues.apache.org/jira/browse/YARN-4966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15257022#comment-15257022
]
Hadoop QA commented on YARN-4966:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 54s
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 18s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 31s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 1 new +
43 unchanged - 8 fixed = 44 total (was 51) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 0s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.8.0_92. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 12s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_92.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 10s
{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 22s {color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 175m 27s {color}
|
[jira] [Commented] (YARN-4595) Add support for configurable read-only mounts
[
https://issues.apache.org/jira/browse/YARN-4595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256958#comment-15256958
]
Hadoop QA commented on YARN-4595:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 13m 36s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
50s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
patch generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.8.0_92 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 4s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed with
JDK v1.8.0_92. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m 35s
{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed with
JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 10s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12800591/YARN-4595.4.patch |
| JIRA Issue | YARN-4595 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 7e982a33867d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-4842) yarn logs command should not require the appOwner argument
[ https://issues.apache.org/jira/browse/YARN-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256815#comment-15256815 ] Xuan Gong commented on YARN-4842: - Thanks for the review. Vinod. The patch addressed all the latest comments. I have added more testcases: 1. The previous testcases can verify that we can get app logs if the appowner is not specified. 2. Added a new testcase to verify that we can not get app logs if we specify an invalid appOwner 3. Added a new testcase to verify that we can get app logs if we can not get appReport > yarn logs command should not require the appOwner argument > -- > > Key: YARN-4842 > URL: https://issues.apache.org/jira/browse/YARN-4842 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Ram Venkatesh >Assignee: Ram Venkatesh > Attachments: YARN-4842.1.patch, YARN-4842.2.patch, YARN-4842.3.patch > > > The yarn logs command is among the most common ways to troubleshoot yarn app > failures, especially by an admin. > Currently if you run the command as a user different from the job owner, the > command will fail with a subtle message that it could not find the app under > the running user's name. This can be confusing especially to new admins. > We can figure out the job owner from the app report returned by the RM or the > AHS, or, by looking for the app directory using a glob pattern, so in most > cases this error can be avoided. > Question - are there scenarios where users will still need to specify the > -appOwner option? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-4842) yarn logs command should not require the appOwner argument
[ https://issues.apache.org/jira/browse/YARN-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong reassigned YARN-4842: --- Assignee: Xuan Gong (was: Ram Venkatesh) > yarn logs command should not require the appOwner argument > -- > > Key: YARN-4842 > URL: https://issues.apache.org/jira/browse/YARN-4842 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Ram Venkatesh >Assignee: Xuan Gong > Attachments: YARN-4842.1.patch, YARN-4842.2.patch, YARN-4842.3.patch > > > The yarn logs command is among the most common ways to troubleshoot yarn app > failures, especially by an admin. > Currently if you run the command as a user different from the job owner, the > command will fail with a subtle message that it could not find the app under > the running user's name. This can be confusing especially to new admins. > We can figure out the job owner from the app report returned by the RM or the > AHS, or, by looking for the app directory using a glob pattern, so in most > cases this error can be avoided. > Question - are there scenarios where users will still need to specify the > -appOwner option? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4842) yarn logs command should not require the appOwner argument
[ https://issues.apache.org/jira/browse/YARN-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-4842: Attachment: YARN-4842.3.patch > yarn logs command should not require the appOwner argument > -- > > Key: YARN-4842 > URL: https://issues.apache.org/jira/browse/YARN-4842 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Ram Venkatesh >Assignee: Ram Venkatesh > Attachments: YARN-4842.1.patch, YARN-4842.2.patch, YARN-4842.3.patch > > > The yarn logs command is among the most common ways to troubleshoot yarn app > failures, especially by an admin. > Currently if you run the command as a user different from the job owner, the > command will fail with a subtle message that it could not find the app under > the running user's name. This can be confusing especially to new admins. > We can figure out the job owner from the app report returned by the RM or the > AHS, or, by looking for the app directory using a glob pattern, so in most > cases this error can be avoided. > Question - are there scenarios where users will still need to specify the > -appOwner option? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1134) Add support for zipping/unzipping logs while in transit for the NM logs web-service
[
https://issues.apache.org/jira/browse/YARN-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256774#comment-15256774
]
Vinod Kumar Vavilapalli commented on YARN-1134:
---
We should also incorporate YARN-4919 in this - to have corresponding options in
"yarn logs" CLI.
> Add support for zipping/unzipping logs while in transit for the NM logs
> web-service
> ---
>
> Key: YARN-1134
> URL: https://issues.apache.org/jira/browse/YARN-1134
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Xuan Gong
>
> As [~zjshen] pointed out at
> [YARN-649|https://issues.apache.org/jira/browse/YARN-649?focusedCommentId=13698415=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13698415],
> {quote}
> For the long running applications, they may have a big log file, such that it
> will take a long time to download the log file via the RESTful API.
> Consequently, HTTP connection may timeout before downloading before
> downloading a complete log file. Maybe it is good to zip the log file before
> sending it, and unzip it after receiving it.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-1134) Add support for zipping/unzipping logs while in transit for the NM logs web-service
[
https://issues.apache.org/jira/browse/YARN-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated YARN-1134:
--
Issue Type: Sub-task (was: Bug)
Parent: YARN-4904
> Add support for zipping/unzipping logs while in transit for the NM logs
> web-service
> ---
>
> Key: YARN-1134
> URL: https://issues.apache.org/jira/browse/YARN-1134
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Xuan Gong
>
> As [~zjshen] pointed out at
> [YARN-649|https://issues.apache.org/jira/browse/YARN-649?focusedCommentId=13698415=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13698415],
> {quote}
> For the long running applications, they may have a big log file, such that it
> will take a long time to download the log file via the RESTful API.
> Consequently, HTTP connection may timeout before downloading before
> downloading a complete log file. Maybe it is good to zip the log file before
> sending it, and unzip it after receiving it.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-1134) Add support for zipping/unzipping logs while in transit for the NM logs web-service
[
https://issues.apache.org/jira/browse/YARN-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated YARN-1134:
--
Issue Type: Bug (was: Sub-task)
Parent: (was: YARN-431)
> Add support for zipping/unzipping logs while in transit for the NM logs
> web-service
> ---
>
> Key: YARN-1134
> URL: https://issues.apache.org/jira/browse/YARN-1134
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Xuan Gong
>
> As [~zjshen] pointed out at
> [YARN-649|https://issues.apache.org/jira/browse/YARN-649?focusedCommentId=13698415=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13698415],
> {quote}
> For the long running applications, they may have a big log file, such that it
> will take a long time to download the log file via the RESTful API.
> Consequently, HTTP connection may timeout before downloading before
> downloading a complete log file. Maybe it is good to zip the log file before
> sending it, and unzip it after receiving it.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4595) Add support for configurable read-only mounts
[ https://issues.apache.org/jira/browse/YARN-4595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-4595: - Attachment: YARN-4595.4.patch Sorry, forgot to rebase. > Add support for configurable read-only mounts > - > > Key: YARN-4595 > URL: https://issues.apache.org/jira/browse/YARN-4595 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Attachments: YARN-4595.1.patch, YARN-4595.2.patch, YARN-4595.3.patch, > YARN-4595.4.patch > > > Mounting files or directories from the host is one way of passing > configuration and other information into a docker container. We could allow > the user to set a list of mounts in the environment of ContainerLaunchContext > (e.g. /dir1:/targetdir1,/dir2:/targetdir2). These would be mounted read-only > to the specified target locations. > Due to permissions and user concerns, for this ticket we will require the > mounts to be resources that are in the distributed cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4983) JVM and UGI metrics disappear after RM is once transitioned to standby mode
[
https://issues.apache.org/jira/browse/YARN-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256754#comment-15256754
]
Li Lu commented on YARN-4983:
-
Thanks for the review [~djp]! I totally agree that we need to refactor the code
of RM HA so that the code is more maintainable. For the boolean flag here,
{{initialize}} and {{fromActive}} are currently decoupled.
{{createAndInitActiveServices}} has two call sites, one in RM#serviceInit,
another in reinitialize. When reinitializing, we're using the flag
{{initialize}} to distinguish if we need to reinitialize the active service.
This method is called when we do a state transition to standby and active.
Therefore, we only need to set fromActive to true when we're reinitializing the
active services and is transitioning from active to standby. Due to the amount
of work to refactor RM HA, maybe we want to resolve that in a separate JIRA? We
can focus on a quick fix for the current symptom in this JIRA? Thanks!
BTW, I kicked another Jenkins run to see if there are any other UT problems.
> JVM and UGI metrics disappear after RM is once transitioned to standby mode
> ---
>
> Key: YARN-4983
> URL: https://issues.apache.org/jira/browse/YARN-4983
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Li Lu
>Assignee: Li Lu
> Attachments: YARN-4983-trunk.000.patch, YARN-4983-trunk.001.patch,
> YARN-4983-trunk.002.patch
>
>
> When get transitioned to standby, the RM will shutdown the existing metric
> system and relaunch a new one. This will cause the jvm metrics and ugi
> metrics to miss in the new metric system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4913) Yarn logs should take a -out option to write to a directory
[ https://issues.apache.org/jira/browse/YARN-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256741#comment-15256741 ] Vinod Kumar Vavilapalli commented on YARN-4913: --- What is the purpose of this option? Why is "yarn logs -out $outdir" simpler / better than "yarn logs > $targetFile" ? > Yarn logs should take a -out option to write to a directory > --- > > Key: YARN-4913 > URL: https://issues.apache.org/jira/browse/YARN-4913 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4913.1.patch, YARN-4913.2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4992) Fix miscellaneous testcase errors due to YARN-2885 and YARN-4335
[ https://issues.apache.org/jira/browse/YARN-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256735#comment-15256735 ] Ray Chiang commented on YARN-4992: -- - No need to skip checking yarn.nodemanager.container-queuing-enabled, since it exists in both YarnConfiguration and yarn-default.xml. Either add it using xmlPropertyToSkipCompare() or remove the call configurationPrefixToSkipCompare(). - I see a change to AMRMClient.java. I assume that's not needed. > Fix miscellaneous testcase errors due to YARN-2885 and YARN-4335 > > > Key: YARN-4992 > URL: https://issues.apache.org/jira/browse/YARN-4992 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-4992.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4595) Add support for configurable read-only mounts
[ https://issues.apache.org/jira/browse/YARN-4595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-4595: - Description: Mounting files or directories from the host is one way of passing configuration and other information into a docker container. We could allow the user to set a list of mounts in the environment of ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2). These would be mounted read-only to the specified target locations. Due to permissions and user concerns, for this ticket we will require the mounts to be resources that are in the distributed cache. was:Mounting files or directories from the host is one way of passing configuration and other information into a docker container. We could allow the user to set a list of mounts in the environment of ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2). These would be mounted read-only to the specified target locations. > Add support for configurable read-only mounts > - > > Key: YARN-4595 > URL: https://issues.apache.org/jira/browse/YARN-4595 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Attachments: YARN-4595.1.patch, YARN-4595.2.patch, YARN-4595.3.patch > > > Mounting files or directories from the host is one way of passing > configuration and other information into a docker container. We could allow > the user to set a list of mounts in the environment of ContainerLaunchContext > (e.g. /dir1:/targetdir1,/dir2:/targetdir2). These would be mounted read-only > to the specified target locations. > Due to permissions and user concerns, for this ticket we will require the > mounts to be resources that are in the distributed cache. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4595) Add support for configurable read-only mounts
[
https://issues.apache.org/jira/browse/YARN-4595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256706#comment-15256706
]
Hadoop QA commented on YARN-4595:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} YARN-4595 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12800580/YARN-4595.3.patch |
| JIRA Issue | YARN-4595 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/11208/console |
| Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
> Add support for configurable read-only mounts
> -
>
> Key: YARN-4595
> URL: https://issues.apache.org/jira/browse/YARN-4595
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Billie Rinaldi
>Assignee: Billie Rinaldi
> Attachments: YARN-4595.1.patch, YARN-4595.2.patch, YARN-4595.3.patch
>
>
> Mounting files or directories from the host is one way of passing
> configuration and other information into a docker container. We could allow
> the user to set a list of mounts in the environment of ContainerLaunchContext
> (e.g. /dir1:/targetdir1,/dir2:/targetdir2). These would be mounted read-only
> to the specified target locations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4595) Add support for configurable read-only mounts
[ https://issues.apache.org/jira/browse/YARN-4595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billie Rinaldi updated YARN-4595: - Attachment: YARN-4595.3.patch Attaching a new patch that ensures the mounted directories are from the distributed cache. It uses the localized resources to determine the correct Path to be mounted, and also double checks that the Path is absolute and is not a symlink. > Add support for configurable read-only mounts > - > > Key: YARN-4595 > URL: https://issues.apache.org/jira/browse/YARN-4595 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Billie Rinaldi >Assignee: Billie Rinaldi > Attachments: YARN-4595.1.patch, YARN-4595.2.patch, YARN-4595.3.patch > > > Mounting files or directories from the host is one way of passing > configuration and other information into a docker container. We could allow > the user to set a list of mounts in the environment of ContainerLaunchContext > (e.g. /dir1:/targetdir1,/dir2:/targetdir2). These would be mounted read-only > to the specified target locations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4807) MockAM#waitForState sleep duration is too long
[ https://issues.apache.org/jira/browse/YARN-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256689#comment-15256689 ] Yufei Gu commented on YARN-4807: All test failures are not related to the patch after my investigation. [~kasha], could you please take a look? Thanks. > MockAM#waitForState sleep duration is too long > -- > > Key: YARN-4807 > URL: https://issues.apache.org/jira/browse/YARN-4807 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla >Assignee: Yufei Gu > Labels: newbie > Attachments: YARN-4807.001.patch, YARN-4807.002.patch, > YARN-4807.003.patch, YARN-4807.004.patch, YARN-4807.005.patch, > YARN-4807.006.patch, YARN-4807.007.patch, YARN-4807.008.patch, > YARN-4807.009.patch, YARN-4807.010.patch, YARN-4807.011.patch, > YARN-4807.012.patch, YARN-4807.013.patch, YARN-4807.014.patch, > YARN-4807.015.patch > > > MockAM#waitForState sleep duration (500 ms) is too long. Also, there is > significant duplication with MockRM#waitForState. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4966) More improvement to get Container logs without specify nodeId
[ https://issues.apache.org/jira/browse/YARN-4966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256662#comment-15256662 ] Xuan Gong commented on YARN-4966: - [~vvasudev] [~djp] Have created https://issues.apache.org/jira/browse/YARN-4993 to track the refactory work. > More improvement to get Container logs without specify nodeId > - > > Key: YARN-4966 > URL: https://issues.apache.org/jira/browse/YARN-4966 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4966.1.patch, YARN-4966.2.patch, YARN-4966.3.patch, > YARN-4966.4.patch > > > Currently, for the finished application, we can get the container logs > without specify node id, but we need to enable > yarn.timeline-service.generic-application-history.enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4966) More improvement to get Container logs without specify nodeId
[
https://issues.apache.org/jira/browse/YARN-4966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256661#comment-15256661
]
Xuan Gong commented on YARN-4966:
-
Thanks, junping for the review.
bq. Why we must need node address if app is not finished? Can we ask RM to get
the nodeAddress instead?
So, currently, we could only get the running container report from RM, and get
the finished container report from ATS. It would go to this else-if clause when:
1. the application is running
2. the container is finished
3. this container is not the AM Container, and we only send AM Container
metrics to ATS.
This is a good catch. This makes people confuse.
{code}
+ System.out.println("Please specify the nodeAddress, and use "
+ + "yarn logs -applicationId -containerId "
+ + "--nodeAddress to get the container logs");
{code}
Have modified the print out message, and make it more clear to users.
> More improvement to get Container logs without specify nodeId
> -
>
> Key: YARN-4966
> URL: https://issues.apache.org/jira/browse/YARN-4966
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-4966.1.patch, YARN-4966.2.patch, YARN-4966.3.patch,
> YARN-4966.4.patch
>
>
> Currently, for the finished application, we can get the container logs
> without specify node id, but we need to enable
> yarn.timeline-service.generic-application-history.enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2919) Potential race between renew and cancel in DelegationTokenRenwer
[ https://issues.apache.org/jira/browse/YARN-2919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256650#comment-15256650 ] Ray Chiang commented on YARN-2919: -- The patch still needs updating. > Potential race between renew and cancel in DelegationTokenRenwer > - > > Key: YARN-2919 > URL: https://issues.apache.org/jira/browse/YARN-2919 > Project: Hadoop YARN > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Karthik Kambatla >Assignee: Naganarasimha G R >Priority: Critical > Attachments: YARN-2919.20141209-1.patch > > > YARN-2874 fixes a deadlock in DelegationTokenRenewer, but there is still a > race because of which a renewal in flight isn't interrupted by a cancel. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4966) More improvement to get Container logs without specify nodeId
[ https://issues.apache.org/jira/browse/YARN-4966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256647#comment-15256647 ] Xuan Gong commented on YARN-4966: - Thanks , varun for the review. Attached a new patch to address your comment. > More improvement to get Container logs without specify nodeId > - > > Key: YARN-4966 > URL: https://issues.apache.org/jira/browse/YARN-4966 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4966.1.patch, YARN-4966.2.patch, YARN-4966.3.patch, > YARN-4966.4.patch > > > Currently, for the finished application, we can get the container logs > without specify node id, but we need to enable > yarn.timeline-service.generic-application-history.enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4966) More improvement to get Container logs without specify nodeId
[ https://issues.apache.org/jira/browse/YARN-4966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-4966: Attachment: YARN-4966.4.patch > More improvement to get Container logs without specify nodeId > - > > Key: YARN-4966 > URL: https://issues.apache.org/jira/browse/YARN-4966 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4966.1.patch, YARN-4966.2.patch, YARN-4966.3.patch, > YARN-4966.4.patch > > > Currently, for the finished application, we can get the container logs > without specify node id, but we need to enable > yarn.timeline-service.generic-application-history.enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4844) Upgrade fields of o.a.h.y.api.records.Resource from int32 to int64
[ https://issues.apache.org/jira/browse/YARN-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256606#comment-15256606 ] Varun Vasudev commented on YARN-4844: - I'm a little concerned with the size of the changes required, coming in so close to the 2.8 release. Given the debate about the extent of the changes we want to make, can we put a patch that changes the int32 to int64, adds getMemoryLong with a Private annotation(so that we can make changes later if we wish) and only fixes the pending memory check that was added in 2.8? We can do a follow up on how to fix this in the wider community. What do you think [~leftnoteasy]? > Upgrade fields of o.a.h.y.api.records.Resource from int32 to int64 > -- > > Key: YARN-4844 > URL: https://issues.apache.org/jira/browse/YARN-4844 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-4844.1.patch, YARN-4844.2.patch > > > We use int32 for memory now, if a cluster has 10k nodes, each node has 210G > memory, we will get a negative total cluster memory. > And another case that easier overflows int32 is: we added all pending > resources of running apps to cluster's total pending resources. If a > problematic app requires too much resources (let's say 1M+ containers, each > of them has 3G containers), int32 will be not enough. > Even if we can cap each app's pending request, we cannot handle the case that > there're many running apps, each of them has capped but still significant > numbers of pending resources. > So we may possibly need to upgrade int32 memory field (could include v-cores > as well) to int64 to avoid integer overflow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3568) TestAMRMTokens should use some random port
[
https://issues.apache.org/jira/browse/YARN-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256588#comment-15256588
]
Hadoop QA commented on YARN-3568:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
52s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 37s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 6s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 48s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 180m 50s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| JDK v1.8.0_77 Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes |
| JDK v1.7.0_95 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| JDK v1.7.0_95 Timed out junit tests |
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
|
[jira] [Resolved] (YARN-4959) MiniYARNCluster should implement AutoCloseable
[
https://issues.apache.org/jira/browse/YARN-4959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Zhuge resolved YARN-4959.
--
Resolution: Not A Problem
Fix Version/s: 2.7.0
Already supported because:
* {{MiniYARNCluster}} inherits from {{CompositeService}} => {{AbstractService}}
=> {{Service}} => {{Closeable}} => {{AutoCloseable}}.
* {{AbstractService.close}} calls {{stop}} which calls {{serviceStop}}.
* {{MiniYARNCluster}} performs cleanup in its overridden {{serviceStop}}
Thanks, [~boky01], for point it out in a comment for HDFS-10287.
> MiniYARNCluster should implement AutoCloseable
> --
>
> Key: YARN-4959
> URL: https://issues.apache.org/jira/browse/YARN-4959
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: test
>Affects Versions: 2.7.0
>Reporter: John Zhuge
>Priority: Trivial
> Fix For: 2.7.0
>
>
> {{MiniYARNCluster}} should implement {{AutoCloseable}} in order to support
> [try-with-resources|https://docs.oracle.com/javase/tutorial/essential/exceptions/tryResourceClose.html].
> It will make test code a little cleaner and more reliable.
> Since {{AutoCloseable}} is only in Java 1.7 or later, this can not be
> backported to Hadoop version prior to 2.7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4993) Refactory ContainersLogsBlock, AggregatedLogsBlock and container log webservice introduced in AHS to minimize the duplication.
[ https://issues.apache.org/jira/browse/YARN-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256549#comment-15256549 ] Xuan Gong commented on YARN-4993: - The refactory list also includes LogsCLI and LogCLIHelpers > Refactory ContainersLogsBlock, AggregatedLogsBlock and container log > webservice introduced in AHS to minimize the duplication. > -- > > Key: YARN-4993 > URL: https://issues.apache.org/jira/browse/YARN-4993 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > > There are many duplicate code in ContainersLogsBlock, AggregatedLogsBlock and > container log webservice introduced by YARN-4920. We should move the > duplications to a common web utility class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4390) Do surgical preemption based on reserved container in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256541#comment-15256541
]
Sunil G commented on YARN-4390:
---
HI [~leftnoteasy]
Thanks for sharing patch here, Few comments/doubts from my side.
1. In
{{CapacitySchedulerPreemptionUtils#deductPreemptableResourcesBasedSelectedCandidates}},
partition is retrieved from {{SchedulerNode}}. I think we can get this from
{{RMContainer#getNodeLabelExpression}} itself considering we will update
RMContainer labelExpression (via a set call) while we invoke
{{replaceLabelsOnNode}}.
2. {{context.getQueueByPartition}} has a code path flow to return NULL
eventhough its not possible to happen. With this assumption, null check is not
handled in caller side. So is it better to raise an exception and skip that
round of preemption to add more clarify?
3. From {{FifoCandidatesSelector}},
{code}
CapacitySchedulerPreemptionUtils
72 .deductPreemptableResourcesBasedSelectedCandidates(
{code}
Ideally this code snippet helps to deduct already selected containers resource
from its corresponding {{TempQueuePartition}}. On that note, I think this code
need not have to be inside one such selector. Rather it may be better suited
from the caller end and ensure that after each round of selection, possible
deduction is done.
4. One doubt about the newly changed {{PreemptableResourceCalculator}} ctor.
Could we get the information about
{{considersReservedResourceWhenCalculateIdeal}} from {{preemptionContext}}
itself. We could add this as a getter api so that code may be more clean.
5. {{getNodesForPreemption}} in {{ReservedContainerCandidatesSelector}} has to
scan all nodes in cluster to get whether any node has a reserved container or
not. For recent UI work, I thought having a metric to know the count of
reserved nodes in cluster. Is it costlier to keep a reserved nodes list in
scheduler? If so, could that be used here. Its just a thought, I havent really
checked the cost of keeping such list. So its fine to scrap this idea also :)
6. In {{ReservedContainerCandidatesSelector}}, containers are sorted against
launch time. Is it possible that we may select a high priority latest container
against some low priority slightly old one.
7. As I see {{RMContainer}} is read-only interface. I feel {{setQueueName}}
can be moved out from this interface and place in {{RMContainerImpl}}. I think
we could downcast and use the same.
Minor nits:
1. {{ReservedContainerCandidatesSelector#getPreemptionCandidatesOnNode}} has
some commented code. It can be removed.
2. in {{TempQueuePerPartition}}, could we add {{reserved}} to {{toString}} for
logging.
I think test class looks very clean and readable after new framework change. It
is very good and really useful :).
I thought of trying this framework sometime tomorrow,and will share if I have
comments.
> Do surgical preemption based on reserved container in CapacityScheduler
> ---
>
> Key: YARN-4390
> URL: https://issues.apache.org/jira/browse/YARN-4390
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.0.0, 2.8.0, 2.7.3
>Reporter: Eric Payne
>Assignee: Wangda Tan
> Attachments: YARN-4390-design.1.pdf, YARN-4390-test-results.pdf,
> YARN-4390.1.patch, YARN-4390.2.patch, YARN-4390.3.branch-2.patch,
> YARN-4390.3.patch, YARN-4390.4.patch, YARN-4390.5.patch
>
>
> There are multiple reasons why preemption could unnecessarily preempt
> containers. One is that an app could be requesting a large container (say
> 8-GB), and the preemption monitor could conceivably preempt multiple
> containers (say 8, 1-GB containers) in order to fill the large container
> request. These smaller containers would then be rejected by the requesting AM
> and potentially given right back to the preempted app.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4676) Automatic and Asynchronous Decommissioning Nodes Status Tracking
[ https://issues.apache.org/jira/browse/YARN-4676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256516#comment-15256516 ] Junping Du commented on YARN-4676: -- Hi [~danzhi], thanks for the patch update. Sorry for coming late in review, I just quickly go through the patch and have some high level comments so far: 1. In discussion of YARN-914 (umbrella for graceful decommission), we are proposing two ways to track node's decommission timeout. One is in command line side (YARN-3225) and the other in RM side (patch here). I think both way have pros and cons, and can useful in different condition. However, in your patch, it looks like you remove the client side track of timeout value. Can we instead to have a configuration to configure which way we are using to track - CLI side or RM side and keep both logic here? 2. If we need to track the node decommissioning progress in RM side, then we need to make sure we don't lose tracking work during RM restart with work preserving. I didn't see related code in your patch and attached proposal - that's something we need to propose a solution. (CLI side has RMProxy for handling this case) > Automatic and Asynchronous Decommissioning Nodes Status Tracking > > > Key: YARN-4676 > URL: https://issues.apache.org/jira/browse/YARN-4676 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.8.0 >Reporter: Daniel Zhi >Assignee: Daniel Zhi > Labels: features > Attachments: GracefulDecommissionYarnNode.pdf, YARN-4676.004.patch, > YARN-4676.005.patch, YARN-4676.006.patch, YARN-4676.007.patch, > YARN-4676.008.patch, YARN-4676.009.patch, YARN-4676.010.patch, > YARN-4676.011.patch, YARN-4676.012.patch, YARN-4676.013.patch > > > DecommissioningNodeWatcher inside ResourceTrackingService tracks > DECOMMISSIONING nodes status automatically and asynchronously after > client/admin made the graceful decommission request. It tracks > DECOMMISSIONING nodes status to decide when, after all running containers on > the node have completed, will be transitioned into DECOMMISSIONED state. > NodesListManager detect and handle include and exclude list changes to kick > out decommission or recommission as necessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4920) ATS/NM should support a link to dowload/get the logs in text format
[ https://issues.apache.org/jira/browse/YARN-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256512#comment-15256512 ] Xuan Gong commented on YARN-4920: - There will be many duplicate codes. Created a separate ticket https://issues.apache.org/jira/browse/YARN-4993 to track this. > ATS/NM should support a link to dowload/get the logs in text format > --- > > Key: YARN-4920 > URL: https://issues.apache.org/jira/browse/YARN-4920 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4920.20160424.branch-2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4993) Refactory ContainersLogsBlock, AggregatedLogsBlock and container log webservice introduced in AHS to minimize the duplication.
[ https://issues.apache.org/jira/browse/YARN-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-4993: Description: There are many duplicate code in ContainersLogsBlock, AggregatedLogsBlock and container log webservice introduced by YARN-4920. We should move the duplications to a common web utility class. (was: There are many duplicate code in ) > Refactory ContainersLogsBlock, AggregatedLogsBlock and container log > webservice introduced in AHS to minimize the duplication. > -- > > Key: YARN-4993 > URL: https://issues.apache.org/jira/browse/YARN-4993 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > > There are many duplicate code in ContainersLogsBlock, AggregatedLogsBlock and > container log webservice introduced by YARN-4920. We should move the > duplications to a common web utility class. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (YARN-4993) Refactory ContainersLogsBlock, AggregatedLogsBlock and container log webservice introduced in AHS to minimize the duplication.
[ https://issues.apache.org/jira/browse/YARN-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong reassigned YARN-4993: --- Assignee: Xuan Gong > Refactory ContainersLogsBlock, AggregatedLogsBlock and container log > webservice introduced in AHS to minimize the duplication. > -- > > Key: YARN-4993 > URL: https://issues.apache.org/jira/browse/YARN-4993 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (YARN-4993) Refactory ContainersLogsBlock, AggregatedLogsBlock and container log webservice introduced in AHS to minimize the duplication.
Xuan Gong created YARN-4993: --- Summary: Refactory ContainersLogsBlock, AggregatedLogsBlock and container log webservice introduced in AHS to minimize the duplication. Key: YARN-4993 URL: https://issues.apache.org/jira/browse/YARN-4993 Project: Hadoop YARN Issue Type: Sub-task Reporter: Xuan Gong -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4993) Refactory ContainersLogsBlock, AggregatedLogsBlock and container log webservice introduced in AHS to minimize the duplication.
[ https://issues.apache.org/jira/browse/YARN-4993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-4993: Description: There are many duplicate code in > Refactory ContainersLogsBlock, AggregatedLogsBlock and container log > webservice introduced in AHS to minimize the duplication. > -- > > Key: YARN-4993 > URL: https://issues.apache.org/jira/browse/YARN-4993 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > > There are many duplicate code in -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4920) ATS/NM should support a link to dowload/get the logs in text format
[
https://issues.apache.org/jira/browse/YARN-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256506#comment-15256506
]
Xuan Gong commented on YARN-4920:
-
Create a new web service in AHS web service.
{code}
use
ats_host:atsport/ws/v1/applicationhistory/containerlogs/{containerid}/{filename}
to show the container logs in ui
use
ats_host:atsport/ws/v1/applicationhistory/containerlogs/{containerid}/{filename}?download=true
to download the container logs
{code}
For the running application, it will redirect the request to the related NM.
For the finished apps, it will parse and download logs from hdfs
> ATS/NM should support a link to dowload/get the logs in text format
> ---
>
> Key: YARN-4920
> URL: https://issues.apache.org/jira/browse/YARN-4920
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-4920.20160424.branch-2.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4795) ContainerMetrics drops records
[
https://issues.apache.org/jira/browse/YARN-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-4795:
---
Attachment: YARN-4795.002.patch
> ContainerMetrics drops records
> --
>
> Key: YARN-4795
> URL: https://issues.apache.org/jira/browse/YARN-4795
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-4795.001.patch, YARN-4795.002.patch
>
>
> The metrics2 system was implemented to deal with persistent sources.
> {{ContainerMetrics}} is an ephemeral source, and so it causes problems.
> Specifically, the {{ContainerMetrics}} only reports metrics once after the
> container has been stopped. This behavior is a problem because the metrics2
> system can ask sources for reports that will be quietly dropped by the sinks
> that care. (It's a metrics2 feature, not a bug.) If that final report is
> silently dropped, it's lost, because the {{ContainerMetrics}} won't report
> anything else ever anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4920) ATS/NM should support a link to dowload/get the logs in text format
[ https://issues.apache.org/jira/browse/YARN-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-4920: Attachment: YARN-4920.20160424.branch-2.patch > ATS/NM should support a link to dowload/get the logs in text format > --- > > Key: YARN-4920 > URL: https://issues.apache.org/jira/browse/YARN-4920 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4920.20160424.branch-2.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-4795) ContainerMetrics drops records
[
https://issues.apache.org/jira/browse/YARN-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated YARN-4795:
---
Attachment: (was: YARN-4795.002.patch)
> ContainerMetrics drops records
> --
>
> Key: YARN-4795
> URL: https://issues.apache.org/jira/browse/YARN-4795
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.9.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-4795.001.patch, YARN-4795.002.patch
>
>
> The metrics2 system was implemented to deal with persistent sources.
> {{ContainerMetrics}} is an ephemeral source, and so it causes problems.
> Specifically, the {{ContainerMetrics}} only reports metrics once after the
> container has been stopped. This behavior is a problem because the metrics2
> system can ask sources for reports that will be quietly dropped by the sinks
> that care. (It's a metrics2 feature, not a bug.) If that final report is
> silently dropped, it's lost, because the {{ContainerMetrics}} won't report
> anything else ever anymore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4983) JVM and UGI metrics disappear after RM is once transitioned to standby mode
[
https://issues.apache.org/jira/browse/YARN-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256481#comment-15256481
]
Junping Du commented on YARN-4983:
--
Thanks [~gtCarrera9] for reporting the issue and delivering the fix.
The patch looks good in overall. There are some questions and comments below.
I think our work here is trying to differentiate the metric
initialization/re-initialization with RM first time start with RM transition
from standby. Isn't it? If so, below code:
{noformat}
@@ -1114,7 +1127,7 @@ void reinitialize(boolean initialize) {
QueueMetrics.clearQueueMetrics();
if (initialize) {
resetDispatcher();
- createAndInitActiveServices();
+ createAndInitActiveServices(true);
}
}
{noformat}
Shall we reuse {{boolean initialize}} instead of set true directly? I saw there
are two caller path: one is from RM.serviceStart() and the other is from
transition to standby from active, marked by boolean value of initialize. I
think we should also differentiate the situation here?
In addition, I think our RM start logic is a bit redundant when RM HA is
enabled: we always do serviceInit() first, then serviceStart(). However, within
serviceStart(), we always put RM to standby mode first and do reinitialize
again. Any special reason that we cannot initialize once to be in standby mode?
CC [~jianhe], [~ka...@cloudera.com].
> JVM and UGI metrics disappear after RM is once transitioned to standby mode
> ---
>
> Key: YARN-4983
> URL: https://issues.apache.org/jira/browse/YARN-4983
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Li Lu
>Assignee: Li Lu
> Attachments: YARN-4983-trunk.000.patch, YARN-4983-trunk.001.patch,
> YARN-4983-trunk.002.patch
>
>
> When get transitioned to standby, the RM will shutdown the existing metric
> system and relaunch a new one. This will cause the jvm metrics and ugi
> metrics to miss in the new metric system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4412) Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers
[
https://issues.apache.org/jira/browse/YARN-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256465#comment-15256465
]
Hadoop QA commented on YARN-4412:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 17s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 45s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 40s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 16s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 56s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 35s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 9s
{color} | {color:red} root: patch generated 77 new + 421 unchanged - 12 fixed =
498 total (was 433) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 22s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 55s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 54s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 21s {color}
| {color:red} hadoop-yarn-api in the patch failed with JDK v1.8.0_77. {color} |
| {color:green}+1{color} | {color:green} unit {color} |
[jira] [Commented] (YARN-4966) More improvement to get Container logs without specify nodeId
[
https://issues.apache.org/jira/browse/YARN-4966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256423#comment-15256423
]
Junping Du commented on YARN-4966:
--
Agree with [~vvasudev] that the run() method is too long now and we should do
some refactor work (may be in another JIRA?). Within the same method call, we
are construct/destruct yarnclient for several times, like:
getApplicationState(), getContainerReport(), etc. - that should be improved.
Also, there is question here:
{noformat}
+} else if (!isApplicationFinished(appState)) {
+ System.err.println("Unable to get logs for this container:"
+ + containerIdStr + "for the application:" + appId);
+ System.out.println("Please specify the nodeAddress, and use "
+ + "yarn logs -applicationId -containerId "
+ + "--nodeAddress to get the container logs");
+ return -1;
}
{noformat}
Why we must need node address if app is not finished? Can we ask RM to get the
nodeAddress instead?
> More improvement to get Container logs without specify nodeId
> -
>
> Key: YARN-4966
> URL: https://issues.apache.org/jira/browse/YARN-4966
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-4966.1.patch, YARN-4966.2.patch, YARN-4966.3.patch
>
>
> Currently, for the finished application, we can get the container logs
> without specify node id, but we need to enable
> yarn.timeline-service.generic-application-history.enabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-3568) TestAMRMTokens should use some random port
[
https://issues.apache.org/jira/browse/YARN-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takashi Ohnishi updated YARN-3568:
--
Attachment: YARN-3568.1.patch
> TestAMRMTokens should use some random port
> --
>
> Key: YARN-3568
> URL: https://issues.apache.org/jira/browse/YARN-3568
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.8.0
>Reporter: Gera Shegalov
>Assignee: Takashi Ohnishi
> Attachments: YARN-3568.1.patch
>
>
> Since the default port is used for yarn.resourcemanager.scheduler.address, if
> we already run a pseudo-distributed cluster on the same development machine,
> the test fails like this:
> {code}
> testMasterKeyRollOver[0](org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens)
> Time elapsed: 1.511 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.net.BindException: Problem binding to [0.0.0.0:8030]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:413)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:590)
> at org.apache.hadoop.ipc.Server.(Server.java:2340)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:534)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:509)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.createServer(RpcServerFactoryPBImpl.java:169)
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:132)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
> at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.serviceStart(ApplicationMasterService.java:140)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:586)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:996)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1037)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1033)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1033)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1073)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens.testMasterKeyRollOver(TestAMRMTokens.java:235)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3568) TestAMRMTokens should use some random port
[
https://issues.apache.org/jira/browse/YARN-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256373#comment-15256373
]
Takashi Ohnishi commented on YARN-3568:
---
Hi.
I have created a patch for this.
> TestAMRMTokens should use some random port
> --
>
> Key: YARN-3568
> URL: https://issues.apache.org/jira/browse/YARN-3568
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.8.0
>Reporter: Gera Shegalov
> Attachments: YARN-3568.1.patch
>
>
> Since the default port is used for yarn.resourcemanager.scheduler.address, if
> we already run a pseudo-distributed cluster on the same development machine,
> the test fails like this:
> {code}
> testMasterKeyRollOver[0](org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens)
> Time elapsed: 1.511 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.net.BindException: Problem binding to [0.0.0.0:8030]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:413)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:590)
> at org.apache.hadoop.ipc.Server.(Server.java:2340)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:534)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:509)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.createServer(RpcServerFactoryPBImpl.java:169)
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:132)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
> at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.serviceStart(ApplicationMasterService.java:140)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:586)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:996)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1037)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1033)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1033)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1073)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens.testMasterKeyRollOver(TestAMRMTokens.java:235)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (YARN-3568) TestAMRMTokens should use some random port
[
https://issues.apache.org/jira/browse/YARN-3568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takashi Ohnishi reassigned YARN-3568:
-
Assignee: Takashi Ohnishi
> TestAMRMTokens should use some random port
> --
>
> Key: YARN-3568
> URL: https://issues.apache.org/jira/browse/YARN-3568
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.8.0
>Reporter: Gera Shegalov
>Assignee: Takashi Ohnishi
> Attachments: YARN-3568.1.patch
>
>
> Since the default port is used for yarn.resourcemanager.scheduler.address, if
> we already run a pseudo-distributed cluster on the same development machine,
> the test fails like this:
> {code}
> testMasterKeyRollOver[0](org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens)
> Time elapsed: 1.511 sec <<< ERROR!
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.net.BindException: Problem binding to [0.0.0.0:8030]
> java.net.BindException: Address already in use; For more details see:
> http://wiki.apache.org/hadoop/BindException
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:444)
> at sun.nio.ch.Net.bind(Net.java:436)
> at
> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
> at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
> at org.apache.hadoop.ipc.Server.bind(Server.java:413)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:590)
> at org.apache.hadoop.ipc.Server.(Server.java:2340)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:945)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.(ProtobufRpcEngine.java:534)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:509)
> at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:787)
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.createServer(RpcServerFactoryPBImpl.java:169)
> at
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:132)
> at
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:65)
> at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:54)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.serviceStart(ApplicationMasterService.java:140)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:586)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:996)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1037)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1033)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1033)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1073)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens.testMasterKeyRollOver(TestAMRMTokens.java:235)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3573) MiniMRYarnCluster constructor that starts the timeline server using a boolean should be marked deprecated
[
https://issues.apache.org/jira/browse/YARN-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256295#comment-15256295
]
Andras Bokor commented on YARN-3573:
Hi All,
I run into this JIRA. I do not really understand the purpose of this fix. If
the constructor is deprecated what is the preferred way to create a
MiniYARNCluster object?
> MiniMRYarnCluster constructor that starts the timeline server using a boolean
> should be marked deprecated
> -
>
> Key: YARN-3573
> URL: https://issues.apache.org/jira/browse/YARN-3573
> Project: Hadoop YARN
> Issue Type: Test
> Components: timelineserver
>Affects Versions: 2.6.0
>Reporter: Mit Desai
>Assignee: Brahma Reddy Battula
> Fix For: 2.8.0
>
> Attachments: YARN-3573-002.patch, YARN-3573.patch
>
>
> {code}MiniMRYarnCluster(String testName, int noOfNMs, boolean enableAHS){code}
> starts the timeline server using *boolean enableAHS*. It is better to have
> the timelineserver started based on the config value.
> We should mark this constructor as deprecated to avoid its future use.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-4412) Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers
[ https://issues.apache.org/jira/browse/YARN-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-4412: -- Attachment: YARN-4412.007.patch Uploading patch after rebase with trunk.. > Create ClusterMonitor to compute ordered list of preferred NMs for > OPPORTUNITIC containers > -- > > Key: YARN-4412 > URL: https://issues.apache.org/jira/browse/YARN-4412 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-4412-yarn-2877.v1.patch, > YARN-4412-yarn-2877.v2.patch, YARN-4412-yarn-2877.v3.patch, > YARN-4412-yarn-2877.v4.patch, YARN-4412-yarn-2877.v5.patch, > YARN-4412-yarn-2877.v6.patch, YARN-4412.007.patch > > > Introduce a Cluster Monitor that aggregates load information from individual > Node Managers and computes an ordered list of preferred Node managers to be > used as target Nodes for OPPORTUNISTIC container allocations. > This list can be pushed out to the Node Manager (specifically the AMRMProxy > running on the Node) via the Allocate Response. This will be used to make > local Scheduling decisions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (YARN-4412) Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers
[ https://issues.apache.org/jira/browse/YARN-4412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh reopened YARN-4412: --- Re-opening to run Jenkins before merging with trunk > Create ClusterMonitor to compute ordered list of preferred NMs for > OPPORTUNITIC containers > -- > > Key: YARN-4412 > URL: https://issues.apache.org/jira/browse/YARN-4412 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-4412-yarn-2877.v1.patch, > YARN-4412-yarn-2877.v2.patch, YARN-4412-yarn-2877.v3.patch, > YARN-4412-yarn-2877.v4.patch, YARN-4412-yarn-2877.v5.patch, > YARN-4412-yarn-2877.v6.patch > > > Introduce a Cluster Monitor that aggregates load information from individual > Node Managers and computes an ordered list of preferred Node managers to be > used as target Nodes for OPPORTUNISTIC container allocations. > This list can be pushed out to the Node Manager (specifically the AMRMProxy > running on the Node) via the Allocate Response. This will be used to make > local Scheduling decisions -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4947) Test timeout is happening for TestRMWebServicesNodes
[
https://issues.apache.org/jira/browse/YARN-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256199#comment-15256199
]
Hadoop QA commented on YARN-4947:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
8s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed with JDK v1.8.0_77 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 46s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.8.0_77. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 42s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK
v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 141m 41s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_77 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
| JDK v1.7.0_95 Failed junit tests |
hadoop.yarn.server.resourcemanager.TestAMAuthorization |
| | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:fbe3e86 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12800483/0004-YARN-4947.patch |
| JIRA Issue | YARN-4947 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
|
[jira] [Commented] (YARN-4947) Test timeout is happening for TestRMWebServicesNodes
[
https://issues.apache.org/jira/browse/YARN-4947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256194#comment-15256194
]
Rohith Sharma K S commented on YARN-4947:
-
Thanks [~bibinchundatt] for the updating patch.
some comments
# I think {{rm.NMwaitForState(nm2.getNodeId(), NodeState.NEW);}} code can be
removed. I don't see any value add from this. For the test case
{{testNodesQueryNew}} you can hack it by creating new RMNodeImpl and add to
context explicitly. This will ensure, RMNode is in new state.
# If we handle 1st point, then MockRM changes do not required.
# {{rm.sendNodeStarted(nm1);}} is not required since RM is started which makes
RMNode to transition to running.
# At last, to avoid problems wrong usage an API, I think it is better to add
MockRM service started check in {{MockRM#registerNode}}
> Test timeout is happening for TestRMWebServicesNodes
>
>
> Key: YARN-4947
> URL: https://issues.apache.org/jira/browse/YARN-4947
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4947.patch, 0002-YARN-4947.patch,
> 0003-YARN-4947.patch, 0004-YARN-4947.patch
>
>
> Testcase timeout for TestRMWebServicesNodes is happening after YARN-4893
> [timeout|https://builds.apache.org/job/PreCommit-YARN-Build/11044/testReport/]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4734) Merge branch:YARN-3368 to trunk
[
https://issues.apache.org/jira/browse/YARN-4734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256184#comment-15256184
]
Sunil G commented on YARN-4734:
---
HI [~leftnoteasy]
Patch looks fine for me. One minor nit in documentation. {{yarnui2.md}}. We are
using {{configs.env}} now, but its mentioned as config.js. Or could we handle
this in a documentation ticket, pls suggest.
> Merge branch:YARN-3368 to trunk
> ---
>
> Key: YARN-4734
> URL: https://issues.apache.org/jira/browse/YARN-4734
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Wangda Tan
>Assignee: Wangda Tan
> Attachments: YARN-4734.1.patch, YARN-4734.2.patch, YARN-4734.3.patch,
> YARN-4734.4.patch, YARN-4734.5.patch, YARN-4734.6.patch, YARN-4734.7.patch
>
>
> YARN-2928 branch is planned to merge back to trunk shortly, it depends on
> changes of YARN-3368. This JIRA is to track the merging task.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4966) More improvement to get Container logs without specify nodeId
[ https://issues.apache.org/jira/browse/YARN-4966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256170#comment-15256170 ] Varun Vasudev commented on YARN-4966: - Thanks for the patch [~xgong]! I think the patch itself is ok - just a minor change - change getAppState to appStateKnown. However, it looks we need a re-factoring of the code in LogsCLI; the run function in particular is becoming too big and too unmanageable. What do you think? Should we file another JIRA about cleaning up the code? > More improvement to get Container logs without specify nodeId > - > > Key: YARN-4966 > URL: https://issues.apache.org/jira/browse/YARN-4966 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-4966.1.patch, YARN-4966.2.patch, YARN-4966.3.patch > > > Currently, for the finished application, we can get the container logs > without specify node id, but we need to enable > yarn.timeline-service.generic-application-history.enabled. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4991) ContainerRequest not setting nodelabelExpression
[
https://issues.apache.org/jira/browse/YARN-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256132#comment-15256132
]
Arun Suresh commented on YARN-4991:
---
Thanks for catching this [~bibinchundatt]
I have included your fix in YARN-4992 though..
Apologize for the inconvenience..
> ContainerRequest not setting nodelabelExpression
>
>
> Key: YARN-4991
> URL: https://issues.apache.org/jira/browse/YARN-4991
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4991.patch
>
>
> TestAMRMClient#testAskWithInvalidNodeLabels
> TestAMRMClient#testAskWithNodeLabels are failing
> {{ContainerRequest}} labels are always set as {{null}}
> {code}
> public ContainerRequest(Resource capability, String[] nodes, String[]
> racks,
> Priority priority, boolean relaxLocality, String
> nodeLabelsExpression) {
> this(capability, nodes, racks, priority, relaxLocality, null,
> ExecutionType.GUARANTEED);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
