[jira] [Commented] (YARN-4464) default value of yarn.resourcemanager.state-store.max-completed-applications should lower.
[
https://issues.apache.org/jira/browse/YARN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15267592#comment-15267592
]
Andrew Wang commented on YARN-4464:
---
If it's just changing the config value, this one sounds pretty easy.
[~templedf] do you think we can get it in soon-ish?
> default value of yarn.resourcemanager.state-store.max-completed-applications
> should lower.
> --
>
> Key: YARN-4464
> URL: https://issues.apache.org/jira/browse/YARN-4464
> Project: Hadoop YARN
> Issue Type: Wish
> Components: resourcemanager
>Reporter: KWON BYUNGCHANG
>Assignee: Daniel Templeton
>Priority: Blocker
>
> my cluster has 120 nodes.
> I configured RM Restart feature.
> {code}
> yarn.resourcemanager.recovery.enabled=true
> yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
> yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
> {code}
> unfortunately I did not configure
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> so that property configured default value 10,000.
> I have restarted RM due to changing another configuartion.
> I expected that RM restart immediately.
> recovery process was very slow. I have waited about 20min.
> realize missing
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> its default value is very huge.
> need to change lower value or document notice on [RM Restart
> page|http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5270) Solve miscellaneous issues caused by YARN-4844
[ https://issues.apache.org/jira/browse/YARN-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15364865#comment-15364865 ] Andrew Wang commented on YARN-5270: --- I think this is the last JIRA we're waiting on for 3.0.0-alpha1. If we can't get it done by end-of-week, can I unblock the release by reverting YARN-4844 from the 3.0.0-alpha1 release branch and picking it up in a later alpha? > Solve miscellaneous issues caused by YARN-4844 > -- > > Key: YARN-5270 > URL: https://issues.apache.org/jira/browse/YARN-5270 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-5270-branch-2.001.patch, > YARN-5270-branch-2.002.patch, YARN-5270-branch-2.003.patch, > YARN-5270-branch-2.8.001.patch, YARN-5270-branch-2.8.002.patch, > YARN-5270-branch-2.8.003.patch > > > Such as javac warnings reported by YARN-5077 and type converting issues in > Resources class. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5270) Solve miscellaneous issues caused by YARN-4844
[ https://issues.apache.org/jira/browse/YARN-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365218#comment-15365218 ] Andrew Wang commented on YARN-5270: --- Thanks for the reply Wangda. To clarify, I'm going to cut a 3.0.0-alpha1 branch off of trunk, and was thinking about reverting YARN-4844 just from the release branch (not trunk). Seems like we're close on the patch here though anyway, which is great news. Thanks for the ping about HADOOP-12893, I missed that there was an addendum patch pending for trunk. I'm tracking it more closely now. > Solve miscellaneous issues caused by YARN-4844 > -- > > Key: YARN-5270 > URL: https://issues.apache.org/jira/browse/YARN-5270 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-5270-branch-2.001.patch, > YARN-5270-branch-2.002.patch, YARN-5270-branch-2.003.patch, > YARN-5270-branch-2.8.001.patch, YARN-5270-branch-2.8.002.patch, > YARN-5270-branch-2.8.003.patch, YARN-5270.003.patch > > > Such as javac warnings reported by YARN-5077 and type converting issues in > Resources class. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4202) TestYarnClient#testReservationAPIs fails intermittently
[
https://issues.apache.org/jira/browse/YARN-4202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-4202:
--
Fix Version/s: (was: 3.0.0-alpha1)
> TestYarnClient#testReservationAPIs fails intermittently
> ---
>
> Key: YARN-4202
> URL: https://issues.apache.org/jira/browse/YARN-4202
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Mit Desai
>Assignee: nijel
>
> Found this failure while looking at the Pre-run on one of my Jiras.
> {noformat}
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The planning algorithm could not find a valid allocation for your request
> at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1149)
> at
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitReservation(ApplicationClientProtocolPBServiceImpl.java:428)
> at
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:465)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2230)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2224)
> Caused by:
> org.apache.hadoop.yarn.server.resourcemanager.reservation.exceptions.PlanningException:
> The planning algorithm could not find a valid allocation for your request
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.allocateUser(PlanningAlgorithm.java:69)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.PlanningAlgorithm.createReservation(PlanningAlgorithm.java:140)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.TryManyReservationAgents.createReservation(TryManyReservationAgents.java:55)
> at
> org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy.createReservation(AlignedPlannerWithGreedy.java:84)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitReservation(ClientRMService.java:1132)
> ... 10 more
> {noformat}
> TestReport Link:
> https://builds.apache.org/job/PreCommit-YARN-Build/9243/testReport/
> When I ran this on my local box branch-2, it succeeds.
> {noformat}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.yarn.client.api.impl.TestYarnClient
> Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 22.999 sec -
> in org.apache.hadoop.yarn.client.api.impl.TestYarnClient
> Results :
> Tests run: 21, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 52.029 s
> [INFO] Finished at: 2015-09-23T11:25:04-06:00
> [INFO] Final Memory: 31M/391M
> [INFO]
>
> {noformat}
> Haven't tried if it is a problem in branch-2.7 or not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3876) get_executable() assumes everything is Linux
[ https://issues.apache.org/jira/browse/YARN-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-3876: -- Fix Version/s: (was: 3.0.0-alpha1) > get_executable() assumes everything is Linux > > > Key: YARN-3876 > URL: https://issues.apache.org/jira/browse/YARN-3876 > Project: Hadoop YARN > Issue Type: Sub-task > Components: build >Affects Versions: 2.7.0 >Reporter: Alan Burlison >Assignee: Alan Burlison > Attachments: YARN-3876.001.patch, YARN-3876.002.patch, > YARN-3876.003.patch > > > get_executable() in container-executor.c is non-portable and is hard-coded to > assume Linux's /proc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5121) fix some container-executor portability issues
[ https://issues.apache.org/jira/browse/YARN-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5121: -- Target Version/s: 3.0.0-alpha2 (was: 3.0.0-alpha1) > fix some container-executor portability issues > -- > > Key: YARN-5121 > URL: https://issues.apache.org/jira/browse/YARN-5121 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Attachments: YARN-5121.00.patch, YARN-5121.01.patch, > YARN-5121.02.patch, YARN-5121.03.patch > > > container-executor has some issues that are preventing it from even compiling > on the OS X jenkins instance. Let's fix those. While we're there, let's > also try to take care of some of the other portability problems that have > crept in over the years, since it used to work great on Solaris but now > doesn't. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4464) default value of yarn.resourcemanager.state-store.max-completed-applications should lower.
[
https://issues.apache.org/jira/browse/YARN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-4464:
--
Target Version/s: 3.0.0-alpha2 (was: 3.0.0-alpha1)
> default value of yarn.resourcemanager.state-store.max-completed-applications
> should lower.
> --
>
> Key: YARN-4464
> URL: https://issues.apache.org/jira/browse/YARN-4464
> Project: Hadoop YARN
> Issue Type: Wish
> Components: resourcemanager
>Reporter: KWON BYUNGCHANG
>Assignee: Daniel Templeton
>Priority: Blocker
> Attachments: YARN-4464.001.patch, YARN-4464.002.patch,
> YARN-4464.003.patch, YARN-4464.004.patch
>
>
> my cluster has 120 nodes.
> I configured RM Restart feature.
> {code}
> yarn.resourcemanager.recovery.enabled=true
> yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
> yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
> {code}
> unfortunately I did not configure
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> so that property configured default value 10,000.
> I have restarted RM due to changing another configuartion.
> I expected that RM restart immediately.
> recovery process was very slow. I have waited about 20min.
> realize missing
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> its default value is very huge.
> need to change lower value or document notice on [RM Restart
> page|http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5275) Timeline application page cannot be loaded when no application submitted/running on the cluster after HADOOP-9613
[
https://issues.apache.org/jira/browse/YARN-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-5275:
--
Target Version/s: 3.0.0-alpha2 (was: 3.0.0-alpha1)
> Timeline application page cannot be loaded when no application
> submitted/running on the cluster after HADOOP-9613
> -
>
> Key: YARN-5275
> URL: https://issues.apache.org/jira/browse/YARN-5275
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha1
>Reporter: Tsuyoshi Ozawa
>Priority: Critical
>
> After HADOOP-9613, Timeline Web UI has a problem reported by [~leftnoteasy]
> and [~sunilg]
> {quote}
> when no application submitted/running on the cluster, applications page
> cannot be loaded.
> {quote}
> We should investigate the reason and fix it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5194) Avoid adding yarn-site to all Configuration instances created by the JVM
[
https://issues.apache.org/jira/browse/YARN-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-5194:
--
Target Version/s: 3.0.0-alpha2 (was: 3.0.0-alpha1)
> Avoid adding yarn-site to all Configuration instances created by the JVM
>
>
> Key: YARN-5194
> URL: https://issues.apache.org/jira/browse/YARN-5194
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Siddharth Seth
>
> {code}
> static {
> addDeprecatedKeys();
> Configuration.addDefaultResource(YARN_DEFAULT_CONFIGURATION_FILE);
> Configuration.addDefaultResource(YARN_SITE_CONFIGURATION_FILE);
> }
> {code}
> This puts the contents of yarn-default and yarn-site into every configuration
> instance created in the VM after YarnConfiguration has been initialized.
> This should be changed to a local addResource for the specific
> YarnConfiguration instance, instead of polluting every Configuration instance.
> Incompatible change. Have set the target version to 3.x.
> The same applies to HdfsConfiguration (hdfs-site.xml), and Configuration
> (core-site.xml etc).
> core-site may be worth including everywhere, however it would be better to
> expect users to explicitly add the relevant resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2962) ZKRMStateStore: Limit the number of znodes under a znode
[ https://issues.apache.org/jira/browse/YARN-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-2962: -- Target Version/s: 3.0.0-alpha2 (was: 3.0.0-alpha1) > ZKRMStateStore: Limit the number of znodes under a znode > > > Key: YARN-2962 > URL: https://issues.apache.org/jira/browse/YARN-2962 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.6.0 >Reporter: Karthik Kambatla >Assignee: Varun Saxena >Priority: Critical > Attachments: YARN-2962.01.patch, YARN-2962.04.patch, > YARN-2962.05.patch, YARN-2962.2.patch, YARN-2962.3.patch > > > We ran into this issue where we were hitting the default ZK server message > size configs, primarily because the message had too many znodes even though > they individually they were all small. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2928) YARN Timeline Service v.2: alpha 1
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382765#comment-15382765 ] Andrew Wang commented on YARN-2928: --- Hi, could someone add release notes for this feature? Thanks! > YARN Timeline Service v.2: alpha 1 > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Fix For: 3.0.0-alpha1 > > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, > ATSv2BackendHBaseSchemaproposal.pdf, Data model proposal v1.pdf, The YARN > Timeline Service v.2 Documentation.pdf, Timeline Service Next Gen - Planning > - ppt.pptx, TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf, > YARN-2928.01.patch, YARN-2928.02.patch, YARN-2928.03.patch, > timeline_service_v2_next_milestones.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2928) YARN Timeline Service v.2: alpha 1
[ https://issues.apache.org/jira/browse/YARN-2928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382798#comment-15382798 ] Andrew Wang commented on YARN-2928: --- I think the content can be the same for the release notes here and at HADOOP-13383. I can handle modifying HADOOP-13383 if you provide the content. Thanks Sangjin! > YARN Timeline Service v.2: alpha 1 > -- > > Key: YARN-2928 > URL: https://issues.apache.org/jira/browse/YARN-2928 > Project: Hadoop YARN > Issue Type: New Feature > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Sangjin Lee >Priority: Critical > Fix For: 3.0.0-alpha1 > > Attachments: ATSv2.rev1.pdf, ATSv2.rev2.pdf, > ATSv2BackendHBaseSchemaproposal.pdf, Data model proposal v1.pdf, The YARN > Timeline Service v.2 Documentation.pdf, Timeline Service Next Gen - Planning > - ppt.pptx, TimelineServiceStoragePerformanceTestSummaryYARN-2928.pdf, > YARN-2928.01.patch, YARN-2928.02.patch, YARN-2928.03.patch, > timeline_service_v2_next_milestones.pdf > > > We have the application timeline server implemented in yarn per YARN-1530 and > YARN-321. Although it is a great feature, we have recognized several critical > issues and features that need to be addressed. > This JIRA proposes the design and implementation changes to address those. > This is phase 1 of this effort. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5882) Reconsider YARN-4126 RM should not issue delegation tokens in unsecure mode
Andrew Wang created YARN-5882: - Summary: Reconsider YARN-4126 RM should not issue delegation tokens in unsecure mode Key: YARN-5882 URL: https://issues.apache.org/jira/browse/YARN-5882 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0-alpha1 Reporter: Andrew Wang Priority: Blocker Filed this for tracking. We might want to revert YARN-4126 based on discussion in that JIRA. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4126) RM should not issue delegation tokens in unsecure mode
[ https://issues.apache.org/jira/browse/YARN-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665696#comment-15665696 ] Andrew Wang commented on YARN-4126: --- FYI, I've filed YARN-5882 as an alpha2 blocker so we have an open JIRA for tracking. We can use it to log the revert if we decide to do that. > RM should not issue delegation tokens in unsecure mode > -- > > Key: YARN-4126 > URL: https://issues.apache.org/jira/browse/YARN-4126 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Bibin A Chundatt > Fix For: 3.0.0-alpha1 > > Attachments: 0001-YARN-4126.patch, 0002-YARN-4126.patch, > 0003-YARN-4126.patch, 0004-YARN-4126.patch, 0005-YARN-4126.patch, > 0006-YARN-4126.patch > > > ClientRMService#getDelegationToken is currently returning a delegation token > in insecure mode. We should not return the token if it's in insecure mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5287) LinuxContainerExecutor fails to set proper permission
[ https://issues.apache.org/jira/browse/YARN-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15668690#comment-15668690 ] Andrew Wang commented on YARN-5287: --- Thanks for the ping Brahma, seems like we're all on the same page here. As long as the revert JIRA summary mentions the JIRA being reverted, it should be clear to end-users. > LinuxContainerExecutor fails to set proper permission > - > > Key: YARN-5287 > URL: https://issues.apache.org/jira/browse/YARN-5287 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.2 >Reporter: Ying Zhang >Assignee: Naganarasimha G R >Priority: Minor > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: YARN-5287-tmp.patch, YARN-5287.003.patch, > YARN-5287.004.patch, YARN-5287.005.patch, YARN-5287.branch-2.001.patch > > Original Estimate: 48h > Remaining Estimate: 48h > > LinuxContainerExecutor fails to set the proper permissions on the local > directories(i.e., /hadoop/yarn/local/usercache/... by default) if the cluster > has been configured with a restrictive umask, e.g.: umask 077. Job failed due > to the following reason: > Path /hadoop/yarn/local/usercache/ambari-qa/appcache/application_ has > permission 700 but needs permission 750 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5559) Analyse 2.8.0/3.0.0 jdiff reports and fix any issues
[ https://issues.apache.org/jira/browse/YARN-5559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15671798#comment-15671798 ] Andrew Wang commented on YARN-5559: --- Ping, seems like this patch is pretty close? Can we get it committed? > Analyse 2.8.0/3.0.0 jdiff reports and fix any issues > > > Key: YARN-5559 > URL: https://issues.apache.org/jira/browse/YARN-5559 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Labels: oct16-easy > Attachments: YARN-5559.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[
https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-5667:
--
Affects Version/s: 3.0.0-alpha1
Target Version/s: 3.0.0-alpha2
Priority: Blocker (was: Major)
I'm upgrading this to a 3.0.0-alpha2 blocker, since we need it to compile HBase.
> Move HBase backend code in ATS v2 into its separate module
> ---
>
> Key: YARN-5667
> URL: https://issues.apache.org/jira/browse/YARN-5667
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Blocker
> Attachments: New module structure.png, part1.yarn5667.prelim.patch,
> part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch,
> part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch,
> pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch,
> pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch,
> pt9.yarn5667.001.patch, yarn5667-001.tar.gz
>
>
> The HBase backend code currently lives along with the core ATS v2 code in
> hadoop-yarn-server-timelineservice module. Because Resource Manager depends
> on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM
> module on HBase modules is introduced (HBase backend is pluggable, so we do
> not need to directly pull in HBase jars).
> In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop
> 3, we encountered a circular dependency during our builds between HBase2.0
> and Hadoop3 artifacts.
> {code}
> hadoop-mapreduce-client-common, hadoop-yarn-client,
> hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice,
> hbase-server, hbase-prefix-tree, hbase-hadoop2-compat,
> hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common]
> {code}
> This jira proposes we move all HBase-backend-related code from
> hadoop-yarn-server-timelineservice into its own module (possible name is
> yarn-server-timelineservice-storage) so that core RM modules do not depend on
> HBase modules any more.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5550) TestYarnCLI#testGetContainers should format according to CONTAINER_PATTERN
[ https://issues.apache.org/jira/browse/YARN-5550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5550: -- Fix Version/s: (was: 3.0.0-alpha2) 3.0.0-alpha1 Looks like this was actually included in alpha1. Updating fixversion. > TestYarnCLI#testGetContainers should format according to CONTAINER_PATTERN > -- > > Key: YARN-5550 > URL: https://issues.apache.org/jira/browse/YARN-5550 > Project: Hadoop YARN > Issue Type: Improvement > Components: client, test >Affects Versions: 2.6.4 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Minor > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: YARN-5550.001.patch, YARN-5550.002.patch, > YARN-5550.003.patch > > > TestYarnCLI#testGetContainers hard codes expected output of getting list of > containers via Yarn CLI. If the timestamp is shorter than the number of > expected characters in ApplicationCLI#CONTAINER_PATTERN (which is 20), the > assert will fail due to whitespace. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5495) Remove import wildcard in CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-5495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5495: -- Fix Version/s: (was: 3.0.0-alpha2) 3.0.0-alpha1 > Remove import wildcard in CapacityScheduler > --- > > Key: YARN-5495 > URL: https://issues.apache.org/jira/browse/YARN-5495 > Project: Hadoop YARN > Issue Type: Task > Components: capacityscheduler >Affects Versions: 3.0.0-alpha2 >Reporter: Ray Chiang >Assignee: Ray Chiang >Priority: Trivial > Fix For: 3.0.0-alpha1 > > Attachments: YARN-5495.001.patch > > > YARN-4091 swapped a bunch of > org.apache.hadoop.yarn.server.resourcemanager.scheduler with the wildcard > version. Assuming things haven't changed in the Style Guide, we disallow > wildcards in the import. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5457) Refactor DistributedScheduling framework to pull out common functionality
[ https://issues.apache.org/jira/browse/YARN-5457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5457: -- Fix Version/s: (was: 3.0.0-alpha2) 3.0.0-alpha1 > Refactor DistributedScheduling framework to pull out common functionality > - > > Key: YARN-5457 > URL: https://issues.apache.org/jira/browse/YARN-5457 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.0.0-alpha1 > > Attachments: YARN-5457.001.patch, YARN-5457.002.patch, > YARN-5457.003.patch, YARN-5457.004.patch > > > Opening this JIRA to track the some refactoring missed in YARN-5113: -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5456) container-executor support for FreeBSD, NetBSD, and others if conf path is absolute
[ https://issues.apache.org/jira/browse/YARN-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5456: -- Fix Version/s: (was: 3.0.0-alpha2) 3.0.0-alpha1 > container-executor support for FreeBSD, NetBSD, and others if conf path is > absolute > --- > > Key: YARN-5456 > URL: https://issues.apache.org/jira/browse/YARN-5456 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, security >Affects Versions: 3.0.0-alpha2 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer > Labels: security > Fix For: 3.0.0-alpha1 > > Attachments: YARN-5456.00.patch, YARN-5456.01.patch > > > YARN-5121 fixed quite a few portability issues, but it also changed how it > determines it's location to be very operating specific for security reasons. > We should add support for FreeBSD to unbreak it's ports entry, NetBSD (the > sysctl options are just in a different order), and for operating systems that > do not have a defined method, an escape hatch. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5431) TimeLineReader daemon start should allow to pass its own reader opts
[
https://issues.apache.org/jira/browse/YARN-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-5431:
--
Fix Version/s: (was: 3.0.0-alpha2)
3.0.0-alpha1
> TimeLineReader daemon start should allow to pass its own reader opts
>
>
> Key: YARN-5431
> URL: https://issues.apache.org/jira/browse/YARN-5431
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scripts, timelinereader
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
> Fix For: 3.0.0-alpha1
>
> Attachments: YARN-5431.0.patch, YARN-5431.1.patch, YARN-5431.2.patch
>
>
> In yarn script , timelinereader doesn't allow to pass reader_opts.
> {code}
> timelinereader)
> HADOOP_SUBCMD_SUPPORTDAEMONIZATION="true"
> HADOOP_CLASSNAME='org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer'
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5121) fix some container-executor portability issues
[ https://issues.apache.org/jira/browse/YARN-5121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5121: -- Fix Version/s: (was: 3.0.0-alpha2) 3.0.0-alpha1 > fix some container-executor portability issues > -- > > Key: YARN-5121 > URL: https://issues.apache.org/jira/browse/YARN-5121 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, security >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Assignee: Allen Wittenauer >Priority: Blocker > Labels: security > Fix For: 3.0.0-alpha1 > > Attachments: YARN-5121.00.patch, YARN-5121.01.patch, > YARN-5121.02.patch, YARN-5121.03.patch, YARN-5121.04.patch, > YARN-5121.06.patch, YARN-5121.07.patch, YARN-5121.08.patch > > > container-executor has some issues that are preventing it from even compiling > on the OS X jenkins instance. Let's fix those. While we're there, let's > also try to take care of some of the other portability problems that have > crept in over the years, since it used to work great on Solaris but now > doesn't. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5049) Extend NMStateStore to save queued container information
[ https://issues.apache.org/jira/browse/YARN-5049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5049: -- Fix Version/s: (was: 3.0.0-alpha2) 3.0.0-alpha1 > Extend NMStateStore to save queued container information > > > Key: YARN-5049 > URL: https://issues.apache.org/jira/browse/YARN-5049 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos > Fix For: 3.0.0-alpha1 > > Attachments: YARN-5049.001.patch, YARN-5049.002.patch, > YARN-5049.003.patch > > > This JIRA is about extending the NMStateStore to save queued container > information whenever a new container is added to the NM queue. > It also removes the information from the state store when the queued > container starts its execution. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4091) Add REST API to retrieve scheduler activity
[ https://issues.apache.org/jira/browse/YARN-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-4091: -- Fix Version/s: (was: 3.0.0-alpha2) 3.0.0-alpha1 > Add REST API to retrieve scheduler activity > --- > > Key: YARN-4091 > URL: https://issues.apache.org/jira/browse/YARN-4091 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, resourcemanager >Affects Versions: 2.7.0 >Reporter: Sunil G >Assignee: Chen Ge > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: Improvement on debugdiagnostic information - YARN.pdf, > SchedulerActivityManager-TestReport v2.pdf, > SchedulerActivityManager-TestReport.pdf, YARN-4091-branch-2.001.patch, > YARN-4091-design-doc-v1.pdf, YARN-4091.1.patch, YARN-4091.2.patch, > YARN-4091.3.patch, YARN-4091.4.patch, YARN-4091.5.patch, YARN-4091.5.patch, > YARN-4091.6.patch, YARN-4091.7.patch, YARN-4091.8.patch, > YARN-4091.preliminary.1.patch, app_activities v2.json, app_activities.json, > node_activities v2.json, node_activities.json > > > As schedulers are improved with various new capabilities, more configurations > which tunes the schedulers starts to take actions such as limit assigning > containers to an application, or introduce delay to allocate container etc. > There are no clear information passed down from scheduler to outerworld under > these various scenarios. This makes debugging very tougher. > This ticket is an effort to introduce more defined states on various parts in > scheduler where it skips/rejects container assignment, activate application > etc. Such information will help user to know whats happening in scheduler. > Attaching a short proposal for initial discussion. We would like to improve > on this as we discuss. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5320) [YARN-3368] Add resource usage by applications and queues to cluster overview page.
[ https://issues.apache.org/jira/browse/YARN-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5320: -- Fix Version/s: 3.0.0-alpha1 > [YARN-3368] Add resource usage by applications and queues to cluster overview > page. > --- > > Key: YARN-5320 > URL: https://issues.apache.org/jira/browse/YARN-5320 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 3.0.0-alpha1 > > > With this, we can get understanding about which application / queue is > consuming most resource in the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5321) [YARN-3368] Add resource usage for application by node managers
[ https://issues.apache.org/jira/browse/YARN-5321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5321: -- Fix Version/s: 3.0.0-alpha2 > [YARN-3368] Add resource usage for application by node managers > --- > > Key: YARN-5321 > URL: https://issues.apache.org/jira/browse/YARN-5321 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5321-YARN-3368-0001.patch, > YARN-5321-YARN-3368.0002.patch, YARN-5321-YARN-3368.003.patch, > YARN-5321-YARN-3368.004.patch, YARN-5321-YARN-3368.005.patch, sample-1.png > > > With this, user can understand distribution of resources allocated to this > application. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5348) [YARN-3368] Node details page improvements
[ https://issues.apache.org/jira/browse/YARN-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5348: -- Fix Version/s: 3.0.0-alpha2 > [YARN-3368] Node details page improvements > -- > > Key: YARN-5348 > URL: https://issues.apache.org/jira/browse/YARN-5348 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Fix For: 3.0.0-alpha2 > > > - Improve the component styling > - Correct padding in Node Information table -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5347) [YARN-3368] Applications page improvements
[ https://issues.apache.org/jira/browse/YARN-5347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5347: -- Fix Version/s: 3.0.0-alpha2 > [YARN-3368] Applications page improvements > -- > > Key: YARN-5347 > URL: https://issues.apache.org/jira/browse/YARN-5347 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Fix For: 3.0.0-alpha2 > > > Applications page: > - Add a "Long running service" sub-page > Application details page: > - Improve the layout > -- Correct the component borders - Remove double border & the extra space > -- Layout "Application Basic Information" vertically > - List attempts under the application as a subpage > - Hide the diagnostics panel when, diagnostics data is not available -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5322) [YARN-3368] Add a node heat chart map
[ https://issues.apache.org/jira/browse/YARN-5322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5322: -- Fix Version/s: 3.0.0-alpha2 > [YARN-3368] Add a node heat chart map > - > > Key: YARN-5322 > URL: https://issues.apache.org/jira/browse/YARN-5322 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 3.0.0-alpha2 > > Attachments: sample-1.png > > > With this we can easier figure out hotspot in the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5345) [YARN-3368] Cluster overview page improvements
[ https://issues.apache.org/jira/browse/YARN-5345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5345: -- Fix Version/s: 3.0.0-alpha2 > [YARN-3368] Cluster overview page improvements > -- > > Key: YARN-5345 > URL: https://issues.apache.org/jira/browse/YARN-5345 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Fix For: 3.0.0-alpha2 > > > - Improve the border/font/color etc in existing donut charts > -- Solid lines and colors might give a better looks > -- Ensure the text is confined to the empty space in the donut > -- Use color codes that convey the meaning of statuses -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5346) [YARN-3368] Queues page improvements
[ https://issues.apache.org/jira/browse/YARN-5346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5346: -- Fix Version/s: 3.0.0-alpha2 > [YARN-3368] Queues page improvements > > > Key: YARN-5346 > URL: https://issues.apache.org/jira/browse/YARN-5346 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Fix For: 3.0.0-alpha2 > > > Queues page: > - Reorder contents in existing Queues page, and Improve UI components > - On clicking a queue, the user must be taken to the respective queue's > details page. > - Display queue details on mouseover > - The bar and doughnut charts doesn't update on queue change, that needs to > be fixed > Queue details page: > - Add a sub-page for all applications running under the queue -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5504) [YARN-3368] Fix YARN UI build pom.xml
[ https://issues.apache.org/jira/browse/YARN-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5504: -- Fix Version/s: 3.0.0-alpha2 > [YARN-3368] Fix YARN UI build pom.xml > - > > Key: YARN-5504 > URL: https://issues.apache.org/jira/browse/YARN-5504 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5504-YARN-3368-0001.patch, > YARN-5504-YARN-3368-0002.patch > > > - Disable tests as we don't have UTs. > - Disable lint & hint as they are not followed by the current codebase, and > are throwing build errors. > - Disable clearing of UI package on building, so that n/w is required only in > the first build. > - Remove duplicate bower installs. > -Change the default packaging.type to 'war' as our UI is a Web application- - > Will keep it in the profile > -Final war should just contain the end result of the build and not all files- > [~wangda] [~vinodkv] [~sunilg] please share your thoughts. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5754) Null check missing for earliest in FifoPolicy
[ https://issues.apache.org/jira/browse/YARN-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5754: -- Fix Version/s: 3.0.0-alpha2 > Null check missing for earliest in FifoPolicy > - > > Key: YARN-5754 > URL: https://issues.apache.org/jira/browse/YARN-5754 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.0.0-alpha1 >Reporter: Yufei Gu >Assignee: Yufei Gu > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5754.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5583) [YARN-3368] Fix wrong paths in .gitignore
[ https://issues.apache.org/jira/browse/YARN-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5583: -- Fix Version/s: 3.0.0-alpha2 > [YARN-3368] Fix wrong paths in .gitignore > - > > Key: YARN-5583 > URL: https://issues.apache.org/jira/browse/YARN-5583 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5583-YARN-3368-0001.patch > > > npm-debug.log & testem.log paths are mentioned wrong. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2995) Enhance UI to show cluster resource utilization of various container Execution types
[ https://issues.apache.org/jira/browse/YARN-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-2995: -- Fix Version/s: 3.0.0-alpha2 > Enhance UI to show cluster resource utilization of various container > Execution types > > > Key: YARN-2995 > URL: https://issues.apache.org/jira/browse/YARN-2995 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Sriram Rao >Assignee: Konstantinos Karanasos >Priority: Blocker > Fix For: 3.0.0-alpha2 > > Attachments: YARN-2995.001.patch, YARN-2995.002.patch, > YARN-2995.003.patch, YARN-2995.004.patch, all-nodes.png, all-nodes.png, > opp-container.png > > > This JIRA proposes to extend the Resource manager UI to show how cluster > resources are being used to run *guaranteed start* and *queueable* > containers. For example, a graph that shows over time, the fraction of > running containers that are *guaranteed start* and the fraction of running > containers that are *queueable*. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2009) CapacityScheduler: Add intra-queue preemption for app priority support
[ https://issues.apache.org/jira/browse/YARN-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-2009: -- Fix Version/s: 3.0.0-alpha2 > CapacityScheduler: Add intra-queue preemption for app priority support > -- > > Key: YARN-2009 > URL: https://issues.apache.org/jira/browse/YARN-2009 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Devaraj K >Assignee: Sunil G > Labels: oct16-medium > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-2009.0001.patch, YARN-2009.0002.patch, > YARN-2009.0003.patch, YARN-2009.0004.patch, YARN-2009.0005.patch, > YARN-2009.0006.patch, YARN-2009.0007.patch, YARN-2009.0008.patch, > YARN-2009.0009.patch, YARN-2009.0010.patch, YARN-2009.0011.patch, > YARN-2009.0012.patch, YARN-2009.0013.patch, YARN-2009.0014.patch, > YARN-2009.0015.patch, YARN-2009.0016.patch > > > While preempting containers based on the queue ideal assignment, we may need > to consider preempting the low priority application containers first. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4765) Split TestHBaseTimelineStorage into multiple test classes
[ https://issues.apache.org/jira/browse/YARN-4765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-4765: -- Fix Version/s: 3.0.0-alpha2 > Split TestHBaseTimelineStorage into multiple test classes > - > > Key: YARN-4765 > URL: https://issues.apache.org/jira/browse/YARN-4765 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: YARN-5355, oct16-medium > Fix For: 3.0.0-alpha2, YARN-5355 > > Attachments: YARN-4765-YARN-5355.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4907) Make all MockRM#waitForState consistent.
[
https://issues.apache.org/jira/browse/YARN-4907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-4907:
--
Fix Version/s: 3.0.0-alpha2
> Make all MockRM#waitForState consistent.
> -
>
> Key: YARN-4907
> URL: https://issues.apache.org/jira/browse/YARN-4907
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Labels: oct16-medium
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-4907.001.patch, YARN-4907.002.patch
>
>
> There are some inconsistencies among these {{waitForState}} in {{MockRM}}:
> 1. Some {{waitForState}} return a boolean while others don't.
> 2. Some {{waitForState}} don't have a timeout, they can wait for ever.
> 3. Some {{waitForState}} use LOG.info and others use {{System.out.println}}
> to print messages.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4998) Minor cleanup to UGI use in AdminService
[
https://issues.apache.org/jira/browse/YARN-4998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-4998:
--
Fix Version/s: 3.0.0-alpha2
2.9.0
> Minor cleanup to UGI use in AdminService
>
>
> Key: YARN-4998
> URL: https://issues.apache.org/jira/browse/YARN-4998
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
>Priority: Trivial
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-4998.001.patch, YARN-4998.002.patch
>
>
> Instead of calling {{UserGroupInformation.getCurrentUser()}} over and over,
> we should just use the stored {{daemonUser}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4396) Log the trace information on FSAppAttempt#assignContainer
[ https://issues.apache.org/jira/browse/YARN-4396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-4396: -- Fix Version/s: 3.0.0-alpha2 > Log the trace information on FSAppAttempt#assignContainer > - > > Key: YARN-4396 > URL: https://issues.apache.org/jira/browse/YARN-4396 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications, fairscheduler >Affects Versions: 2.7.1 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Labels: oct16-easy > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-4396.001.patch, YARN-4396.002.patch, > YARN-4396.003.patch, YARN-4396.004.patch, YARN-4396.005.patch > > > When I configure the yarn.scheduler.fair.locality.threshold.node and > yarn.scheduler.fair.locality.threshold.rack to open this function, I have no > detail info of assigning container's locality. And it's important because it > will lead some delay scheduling and will have an influence on my cluster. If > I know these info, I can adjust param in cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4456) Clean up Lint warnings in nodemanager
[ https://issues.apache.org/jira/browse/YARN-4456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-4456: -- Fix Version/s: 3.0.0-alpha2 > Clean up Lint warnings in nodemanager > - > > Key: YARN-4456 > URL: https://issues.apache.org/jira/browse/YARN-4456 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.7.1 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Labels: oct16-easy > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-4456.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5772) Replace old Hadoop logo with new one
[ https://issues.apache.org/jira/browse/YARN-5772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5772: -- Fix Version/s: 3.0.0-alpha2 > Replace old Hadoop logo with new one > > > Key: YARN-5772 > URL: https://issues.apache.org/jira/browse/YARN-5772 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn-ui-v2 >Affects Versions: YARN-3368 >Reporter: Akira Ajisaka >Assignee: Akhil PB > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5772-YARN-3368.0001.patch, ui2-with-newlogo.png > > > YARN-5161 added Apache Hadoop logo in the UI but the logo is old. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5552) Add Builder methods for common yarn API records
[ https://issues.apache.org/jira/browse/YARN-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5552: -- Fix Version/s: 3.0.0-alpha2 > Add Builder methods for common yarn API records > --- > > Key: YARN-5552 > URL: https://issues.apache.org/jira/browse/YARN-5552 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Arun Suresh >Assignee: Tao Jie > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5552.000.patch, YARN-5552.001.patch, > YARN-5552.002.patch, YARN-5552.003.patch, YARN-5552.004.patch, > YARN-5552.005.patch, YARN-5552.006.patch, YARN-5552.007.patch, > YARN-5552.008.patch, YARN-5552.009.patch > > > Currently yarn API records such as ResourceRequest, AllocateRequest/Respone > as well as AMRMClient.ContainerRequest have multiple constructors / > newInstance methods. This makes it very difficult to add new fields to these > records. > It would probably be better if we had Builder classes for many of these > records, which would make evolution of these records a bit easier. > (suggested by [~kasha]) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5793) Trim configuration values in DockerLinuxContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-5793:
--
Fix Version/s: 3.0.0-alpha2
> Trim configuration values in DockerLinuxContainerRuntime
>
>
> Key: YARN-5793
> URL: https://issues.apache.org/jira/browse/YARN-5793
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 3.0.0-alpha1
>Reporter: Tianyin Xu
>Assignee: Tianyin Xu
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5793..patch, YARN-5793.0001.patch
>
>
> The current implementation of {{DockerLinuxContainerRuntime}} does not follow
> the practice of trimming configuration values. This leads to errors if users
> set values containing space or newline.
> see the following YARN commits as reference:
> YARN-3395. FairScheduler: Trim whitespaces when using username for queuename.
> YARN-2869. CapacityScheduler should trim sub queue names when parse
> configuration.
> YARN-2843. Fixed NodeLabelsManager to trim inputs for hosts and labels so as
> to make them work correctly.
> and many other Hadoop/HDFS commits (just list a few):
> HDFS-9708. FSNamesystem.initAuditLoggers() doesn't trim classnames
> HDFS-2799. Trim fs.checkpoint.dir values.
> HADOOP-6578. Configuration should trim whitespace around a lot of value types
> HADOOP-6534. Trim whitespace from directory lists initializing
> Patch is available against trunk
> {code:title=DockerLinuxContainerRuntime.java|borderStyle=solid}
> @@ -219,9 +219,9 @@ public void initialize(Configuration conf)
> dockerClient = new DockerClient(conf);
> allowedNetworks.clear();
> allowedNetworks.addAll(Arrays.asList(
> -
> conf.getStrings(YarnConfiguration.NM_DOCKER_ALLOWED_CONTAINER_NETWORKS,
> +
> conf.getTrimmedStrings(YarnConfiguration.NM_DOCKER_ALLOWED_CONTAINER_NETWORKS,
>
> YarnConfiguration.DEFAULT_NM_DOCKER_ALLOWED_CONTAINER_NETWORKS)));
> -defaultNetwork = conf.get(
> +defaultNetwork = conf.getTrimmed(
> YarnConfiguration.NM_DOCKER_DEFAULT_CONTAINER_NETWORK,
> YarnConfiguration.DEFAULT_NM_DOCKER_DEFAULT_CONTAINER_NETWORK);
>
> @@ -237,7 +237,7 @@ public void initialize(Configuration conf)
>throw new ContainerExecutionException(message);
> }
>
> -privilegedContainersAcl = new AccessControlList(conf.get(
> +privilegedContainersAcl = new AccessControlList(conf.getTrimmed(
> YarnConfiguration.NM_DOCKER_PRIVILEGED_CONTAINERS_ACL,
> YarnConfiguration.DEFAULT_NM_DOCKER_PRIVILEGED_CONTAINERS_ACL));
>}
> @@ -439,7 +439,7 @@ public void launchContainer(ContainerRuntimeContext ctx)
> LOCALIZED_RESOURCES);
> @SuppressWarnings("unchecked")
> List userLocalDirs = ctx.getExecutionAttribute(USER_LOCAL_DIRS);
> -Set capabilities = new HashSet<>(Arrays.asList(conf.getStrings(
> +Set capabilities = new
> HashSet<>(Arrays.asList(conf.getTrimmedStrings(
> YarnConfiguration.NM_DOCKER_CONTAINER_CAPABILITIES,
> YarnConfiguration.DEFAULT_NM_DOCKER_CONTAINER_CAPABILITIES)));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4911) Bad placement policy in FairScheduler causes the RM to crash
[ https://issues.apache.org/jira/browse/YARN-4911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-4911: -- Fix Version/s: 3.0.0-alpha2 > Bad placement policy in FairScheduler causes the RM to crash > > > Key: YARN-4911 > URL: https://issues.apache.org/jira/browse/YARN-4911 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Reporter: Ray Chiang >Assignee: Ray Chiang > Labels: supportability > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-4911.001.patch, YARN-4911.002.patch, > YARN-4911.003.patch, YARN-4911.004.patch > > > When you have a fair-scheduler.xml with the rule: > > > > and the queue okay1 doesn't exist, the following exception occurs in the RM: > 2016-04-01 16:56:33,383 FATAL > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error in > handling event type APP_ADDED to the scheduler > java.lang.IllegalStateException: Should have applied a rule before reaching > here > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.QueuePlacementPolicy.assignAppToQueue(QueuePlacementPolicy.java:173) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.assignToQueue(FairScheduler.java:728) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.addApplication(FairScheduler.java:634) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:1224) > at > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.handle(FairScheduler.java:112) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:691) > at java.lang.Thread.run(Thread.java:745) > which causes the RM to crash. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3432) Cluster metrics have wrong Total Memory when there is reserved memory on CS
[ https://issues.apache.org/jira/browse/YARN-3432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-3432: -- Fix Version/s: 3.0.0-alpha2 > Cluster metrics have wrong Total Memory when there is reserved memory on CS > --- > > Key: YARN-3432 > URL: https://issues.apache.org/jira/browse/YARN-3432 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler, resourcemanager >Affects Versions: 2.6.0 >Reporter: Thomas Graves >Assignee: Brahma Reddy Battula > Labels: oct16-easy > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2 > > Attachments: YARN-3432-002.patch, YARN-3432-003.patch, YARN-3432.patch > > > I noticed that when reservations happen when using the Capacity Scheduler, > the UI and web services report the wrong total memory. > For example. I have a 300GB of total memory in my cluster. I allocate 50 > and I reserve 10. The cluster metrics for total memory get reported as 290GB. > This was broken by https://issues.apache.org/jira/browse/YARN-656 so perhaps > there is a difference between fair scheduler and capacity scheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5047) Refactor nodeUpdate across schedulers
[ https://issues.apache.org/jira/browse/YARN-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5047: -- Fix Version/s: 3.0.0-alpha2 > Refactor nodeUpdate across schedulers > - > > Key: YARN-5047 > URL: https://issues.apache.org/jira/browse/YARN-5047 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler, fairscheduler, scheduler >Affects Versions: 3.0.0-alpha1 >Reporter: Ray Chiang >Assignee: Ray Chiang > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5047.001.patch, YARN-5047.002.patch, > YARN-5047.003.patch, YARN-5047.004.patch, YARN-5047.005.patch, > YARN-5047.006.patch, YARN-5047.007.patch, YARN-5047.008.patch, > YARN-5047.009.patch, YARN-5047.010.patch, YARN-5047.011.patch, > YARN-5047.012.patch > > > FairScheduler#nodeUpdate() and CapacityScheduler#nodeUpdate() have a lot of > commonality in their code. See about refactoring the common parts into > AbstractYARNScheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4218) Metric for resource*time that was preempted
[ https://issues.apache.org/jira/browse/YARN-4218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-4218: -- Fix Version/s: 3.0.0-alpha2 > Metric for resource*time that was preempted > --- > > Key: YARN-4218 > URL: https://issues.apache.org/jira/browse/YARN-4218 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Chang Li >Assignee: Chang Li > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: YARN-4218-branch-2.003.patch, YARN-4218.006.patch, > YARN-4218.2.patch, YARN-4218.2.patch, YARN-4218.2.patch, YARN-4218.2.patch, > YARN-4218.3.patch, YARN-4218.4.patch, YARN-4218.5.patch, > YARN-4218.branch-2.2.patch, YARN-4218.branch-2.patch, YARN-4218.patch, > YARN-4218.trunk.2.patch, YARN-4218.trunk.3.patch, YARN-4218.trunk.patch, > YARN-4218.wip.patch, screenshot-1.png, screenshot-2.png, screenshot-3.png > > > After YARN-415 we have the ability to track the resource*time footprint of a > job and preemption metrics shows how many containers were preempted on a job. > However we don't have a metric showing the resource*time footprint cost of > preemption. In other words, we know how many containers were preempted but we > don't have a good measure of how much work was lost as a result of preemption. > We should add this metric so we can analyze how much work preemption is > costing on a grid and better track which jobs were heavily impacted by it. A > job that has 100 containers preempted that only lasted a minute each and were > very small is going to be less impacted than a job that only lost a single > container but that container was huge and had been running for 3 days. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4033) In FairScheduler, parent queues should also display queue status
[ https://issues.apache.org/jira/browse/YARN-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-4033: -- Fix Version/s: 3.0.0-alpha2 > In FairScheduler, parent queues should also display queue status > - > > Key: YARN-4033 > URL: https://issues.apache.org/jira/browse/YARN-4033 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Reporter: Siqi Li >Assignee: Siqi Li > Labels: oct16-easy > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: Screen Shot 2015-08-07 at 2.04.04 PM.png, > YARN-4033.v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5503) [YARN-3368] Add missing hidden files in webapp folder for deployment
[ https://issues.apache.org/jira/browse/YARN-5503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5503: -- Fix Version/s: 3.0.0-alpha2 > [YARN-3368] Add missing hidden files in webapp folder for deployment > > > Key: YARN-5503 > URL: https://issues.apache.org/jira/browse/YARN-5503 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Sreenath Somarajapuram >Assignee: Sreenath Somarajapuram > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5503-YARN-3368-0001.patch, > YARN-5503-YARN-3368-0001.patch, YARN-5503-YARN-3368-0002.patch, > YARN-5503-YARN-3368-0003.patch, YARN-5503-YARN-3368-0004.patch, > YARN-5503-YARN-3368.0005.patch, YARN-5503-YARN-3368.0006.patch > > > - Feel it might be good to have a readme file with the basic instructions. > - Change package type to war, as ours is a web application > - Just noticed that the hidden files that must be present in the base > directory of ember app, are missing. Most of them are used for configuration, > and when missing the default vakues would be used by ember. > -- They include - .bowerrc, .editorconfig, .ember-cli, .gitignore, .jshintrc, > .travis.yml, .watchmanconfig -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4668) Reuse objectMapper instance in Yarn
[ https://issues.apache.org/jira/browse/YARN-4668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-4668: -- Fix Version/s: 3.0.0-alpha2 > Reuse objectMapper instance in Yarn > --- > > Key: YARN-4668 > URL: https://issues.apache.org/jira/browse/YARN-4668 > Project: Hadoop YARN > Issue Type: Improvement > Components: timelineclient >Affects Versions: 2.7.1 >Reporter: Yiqun Lin >Assignee: Yiqun Lin > Labels: oct16-easy > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-4668.001.patch, YARN-4668.002.patch, > YARN-4668.002_1.patch > > > This jira is similar to MAPREDUCE-6626, we can see detail info about this > problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5161) [YARN-3368] Add Apache Hadoop logo in YarnUI home page
[ https://issues.apache.org/jira/browse/YARN-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5161: -- Fix Version/s: 3.0.0-alpha2 > [YARN-3368] Add Apache Hadoop logo in YarnUI home page > -- > > Key: YARN-5161 > URL: https://issues.apache.org/jira/browse/YARN-5161 > Project: Hadoop YARN > Issue Type: Sub-task > Components: webapp >Reporter: Sunil G >Assignee: Kai Sasaki > Fix For: 3.0.0-alpha2 > > Attachments: Screen Shot 2016-05-31 at 21.22.30.png, Screen Shot > 2016-06-11 at 12.33.39.png, Screen Shot 2016-06-20 at 23.15.05.png, > YARN-5161-YARN-3368.03.patch, YARN-5161-YARN-3368.04.patch, > YARN-5161-YARN-3368.05.patch, YARN-5161.01.patch, YARN-5161.02.patch, > apache_logo.png, hadoop_logo.png > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5915) ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write
[ https://issues.apache.org/jira/browse/YARN-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5915: -- Target Version/s: 3.0.0-alpha2 (was: 3.0.0-alpha1) > ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every > event write > > > Key: YARN-5915 > URL: https://issues.apache.org/jira/browse/YARN-5915 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.0.0-alpha1 >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Attachments: YARN-5915.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5899) A small fix for printing debug info inside function canAssignToThisQueue()
[ https://issues.apache.org/jira/browse/YARN-5899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5899: -- Target Version/s: 3.0.0-alpha2 (was: 3.0.0-alpha1) > A small fix for printing debug info inside function canAssignToThisQueue() > -- > > Key: YARN-5899 > URL: https://issues.apache.org/jira/browse/YARN-5899 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.0.0-alpha1 >Reporter: Ying Zhang >Assignee: Ying Zhang >Priority: Trivial > Attachments: YARN-5899.001.patch > > > A small fix inside function canAssignToThisQueue() for printing DEBUG info. > Please see patch attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5536) Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout
[ https://issues.apache.org/jira/browse/YARN-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703227#comment-15703227 ] Andrew Wang commented on YARN-5536: --- This is marked as a blocker, is there an update? > Multiple format support (JSON, etc.) for exclude node file in NM graceful > decommission with timeout > --- > > Key: YARN-5536 > URL: https://issues.apache.org/jira/browse/YARN-5536 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Junping Du >Priority: Blocker > > Per discussion in YARN-4676, we agree that multiple format (other than xml) > should be supported to decommission nodes with timeout values. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4126) RM should not issue delegation tokens in unsecure mode
[ https://issues.apache.org/jira/browse/YARN-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703265#comment-15703265 ] Andrew Wang commented on YARN-4126: --- Ping on this JIRA. Should we revert while discussion continues? I'd prefer to stick with the 2.x behavior if we're not in agreement, and Daryn's previous comment seemed like a strong -0 if not a -1. > RM should not issue delegation tokens in unsecure mode > -- > > Key: YARN-4126 > URL: https://issues.apache.org/jira/browse/YARN-4126 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Bibin A Chundatt > Fix For: 3.0.0-alpha1 > > Attachments: 0001-YARN-4126.patch, 0002-YARN-4126.patch, > 0003-YARN-4126.patch, 0004-YARN-4126.patch, 0005-YARN-4126.patch, > 0006-YARN-4126.patch > > > ClientRMService#getDelegationToken is currently returning a delegation token > in insecure mode. We should not return the token if it's in insecure mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4752) [Umbrella] FairScheduler: Improve preemption
[ https://issues.apache.org/jira/browse/YARN-4752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703863#comment-15703863 ] Andrew Wang commented on YARN-4752: --- [~templedf] Looks like this umbrella was committed to trunk, should we resolve / set the fix version? > [Umbrella] FairScheduler: Improve preemption > > > Key: YARN-4752 > URL: https://issues.apache.org/jira/browse/YARN-4752 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 2.8.0 >Reporter: Karthik Kambatla > Attachments: YARN-4752.FairSchedulerPreemptionOverhaul.pdf, > yarn-4752-1.patch, yarn-4752.2.patch, yarn-4752.3.patch, yarn-4752.4.patch, > yarn-4752.4.patch > > > A number of issues have been reported with respect to preemption in > FairScheduler along the lines of: > # FairScheduler preempts resources from nodes even if the resultant free > resources cannot fit the incoming request. > # Preemption doesn't preempt from sibling queues > # Preemption doesn't preempt from sibling apps under the same queue that is > over its fairshare > # ... > Filing this umbrella JIRA to group all the issues together and think of a > comprehensive solution. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4330) MiniYARNCluster is showing multiple Failed to instantiate default resource calculator warning messages.
[
https://issues.apache.org/jira/browse/YARN-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-4330:
--
Fix Version/s: 3.0.0-alpha2
2.9.0
> MiniYARNCluster is showing multiple Failed to instantiate default resource
> calculator warning messages.
>
>
> Key: YARN-4330
> URL: https://issues.apache.org/jira/browse/YARN-4330
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, yarn
>Affects Versions: 2.8.0
> Environment: OSX, JUnit
>Reporter: Steve Loughran
>Assignee: Varun Saxena
>Priority: Blocker
> Labels: oct16-hard
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-4330.002.patch, YARN-4330.003.patch,
> YARN-4330.004.patch, YARN-4330.01.patch
>
>
> Whenever I try to start a MiniYARNCluster on Branch-2 (commit #0b61cca), I
> see multiple stack traces warning me that a resource calculator plugin could
> not be created
> {code}
> (ResourceCalculatorPlugin.java:getResourceCalculatorPlugin(184)) -
> java.lang.UnsupportedOperationException: Could not determine OS: Failed to
> instantiate default resource calculator.
> java.lang.UnsupportedOperationException: Could not determine OS
> {code}
> This is a minicluster. It doesn't need resource calculation. It certainly
> doesn't need test logs being cluttered with even more stack traces which will
> only generate false alarms about tests failing.
> There needs to be a way to turn this off, and the minicluster should have it
> that way by default.
> Being ruthless and marking as a blocker, because its a fairly major
> regression for anyone testing with the minicluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5725) Test uncaught exception in TestContainersMonitorResourceChange.testContainersResourceChange when setting IP and host
[ https://issues.apache.org/jira/browse/YARN-5725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5725: -- Fix Version/s: 3.0.0-alpha2 > Test uncaught exception in > TestContainersMonitorResourceChange.testContainersResourceChange when setting > IP and host > > > Key: YARN-5725 > URL: https://issues.apache.org/jira/browse/YARN-5725 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Labels: oct16-easy > Fix For: 3.0.0-alpha2 > > Attachments: YARN-5725.000.patch, YARN-5725.001.patch, > YARN-5725.002.patch, YARN-5725.003.patch, YARN-5725.004.patch, > YARN-5725.005.patch, YARN-5725.006.patch, YARN-5725.007.patch, > YARN-5725.008.patch, YARN-5725.009.patch, YARN-5725.010.patch, > YARN-5725.011.patch > > Original Estimate: 2h > Remaining Estimate: 2h > > The issue is a warning but it prevents container monitor to continue > 2016-10-12 14:38:23,280 WARN [Container Monitor] > monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(594)) - > Uncaught exception in ContainersMonitorImpl while monitoring resource of > container_123456_0001_01_01 > java.lang.NullPointerException > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:455) > 2016-10-12 14:38:23,281 WARN [Container Monitor] > monitor.ContainersMonitorImpl (ContainersMonitorImpl.java:run(613)) - > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl > is interrupted. Exiting. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4330) MiniYARNCluster is showing multiple Failed to instantiate default resource calculator warning messages.
[
https://issues.apache.org/jira/browse/YARN-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15703871#comment-15703871
]
Andrew Wang commented on YARN-4330:
---
I set the fix versions based on current git state. Please update the 2.x fix
version if this goes back to 2.8.
> MiniYARNCluster is showing multiple Failed to instantiate default resource
> calculator warning messages.
>
>
> Key: YARN-4330
> URL: https://issues.apache.org/jira/browse/YARN-4330
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test, yarn
>Affects Versions: 2.8.0
> Environment: OSX, JUnit
>Reporter: Steve Loughran
>Assignee: Varun Saxena
>Priority: Blocker
> Labels: oct16-hard
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-4330.002.patch, YARN-4330.003.patch,
> YARN-4330.004.patch, YARN-4330.01.patch
>
>
> Whenever I try to start a MiniYARNCluster on Branch-2 (commit #0b61cca), I
> see multiple stack traces warning me that a resource calculator plugin could
> not be created
> {code}
> (ResourceCalculatorPlugin.java:getResourceCalculatorPlugin(184)) -
> java.lang.UnsupportedOperationException: Could not determine OS: Failed to
> instantiate default resource calculator.
> java.lang.UnsupportedOperationException: Could not determine OS
> {code}
> This is a minicluster. It doesn't need resource calculation. It certainly
> doesn't need test logs being cluttered with even more stack traces which will
> only generate false alarms about tests failing.
> There needs to be a way to turn this off, and the minicluster should have it
> that way by default.
> Being ruthless and marking as a blocker, because its a fairly major
> regression for anyone testing with the minicluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4395) Typo in comment in ClientServiceDelegate
[
https://issues.apache.org/jira/browse/YARN-4395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15706932#comment-15706932
]
Andrew Wang commented on YARN-4395:
---
In the future let's remember to wait for the precommit bot to run. This typo
fix looks quite safe, but having contributors press "Submit Patch" and look at
the precommit output helps them get used to our workflow.
> Typo in comment in ClientServiceDelegate
>
>
> Key: YARN-4395
> URL: https://issues.apache.org/jira/browse/YARN-4395
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Daniel Templeton
>Assignee: Alison Yu
>Priority: Trivial
> Labels: newbie
> Fix For: 3.0.0-alpha2
>
> Attachments: YARN-4395.001.patch
>
>
> Line 337 in {{invoke()}} has the following comment:
> {code}
> // if it's AM shut down, do not decrement maxClientRetry as we wait
> for
> // AM to be restarted.
> {code}
> Ideally it should be:
> {code}
> // If its AM shut down, do not decrement maxClientRetry while we wait
> // for its AM to be restarted.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5704) Provide config knobs to control enabling/disabling new/work in progress features in container-executor
[ https://issues.apache.org/jira/browse/YARN-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5704: -- Affects Version/s: (was: 3.0.0-alpha2) > Provide config knobs to control enabling/disabling new/work in progress > features in container-executor > -- > > Key: YARN-5704 > URL: https://issues.apache.org/jira/browse/YARN-5704 > Project: Hadoop YARN > Issue Type: Task > Components: yarn >Affects Versions: 2.8.0, 3.0.0-alpha1 >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: YARN-5704-branch-2.8.001.patch, YARN-5704.001.patch > > > Provide a mechanism to enable/disable Docker and TC (Traffic Control) > functionality at the container-executor level. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4843) [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to int64
[ https://issues.apache.org/jira/browse/YARN-4843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-4843: -- Hadoop Flags: (was: Incompatible change) I'm also going to remove the "Incompatible" flag since we're trying to preserve wire compatibility between Hadoop 2 and Hadoop 3, and it seems like there are proposals to implement the other APIs compatibly. > [Umbrella] Revisit YARN ProtocolBuffer int32 usages that need to upgrade to > int64 > - > > Key: YARN-4843 > URL: https://issues.apache.org/jira/browse/YARN-4843 > Project: Hadoop YARN > Issue Type: Bug > Components: api >Affects Versions: 3.0.0-alpha1 >Reporter: Wangda Tan > > This JIRA is to track all int32 usages in YARN's ProtocolBuffer APIs that we > possibly need to update to int64. > One example is resource API. We use int32 for memory now, if a cluster has > 10k nodes, each node has 210G memory, we will get a negative total cluster > memory. > We may have other fields may need to upgrade from int32 to int64. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5667) Move HBase backend code in ATS v2 into its separate module
[
https://issues.apache.org/jira/browse/YARN-5667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15736883#comment-15736883
]
Andrew Wang commented on YARN-5667:
---
Ping [~haibochen], any update on this JIRA?
> Move HBase backend code in ATS v2 into its separate module
> ---
>
> Key: YARN-5667
> URL: https://issues.apache.org/jira/browse/YARN-5667
> Project: Hadoop YARN
> Issue Type: Task
> Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Blocker
> Attachments: New module structure.png, part1.yarn5667.prelim.patch,
> part2.yarn5667.prelim.patch, part3.yarn5667.prelim.patch,
> part4.yarn5667.prelim.patch, part5.yarn5667.prelim.patch,
> pt1.yarn5667.001.patch, pt2.yarn5667.001.patch, pt3.yarn5667.001.patch,
> pt4.yarn5667.001.patch, pt5.yarn5667.001.patch, pt6.yarn5667.001.patch,
> pt9.yarn5667.001.patch, yarn5667-001.tar.gz
>
>
> The HBase backend code currently lives along with the core ATS v2 code in
> hadoop-yarn-server-timelineservice module. Because Resource Manager depends
> on hadoop-yarn-server-timelineservice, an unnecessary dependency of the RM
> module on HBase modules is introduced (HBase backend is pluggable, so we do
> not need to directly pull in HBase jars).
> In our internal effort to try ATS v2 with HBase 2.0 which depends on Hadoop
> 3, we encountered a circular dependency during our builds between HBase2.0
> and Hadoop3 artifacts.
> {code}
> hadoop-mapreduce-client-common, hadoop-yarn-client,
> hadoop-yarn-server-resourcemanager, hadoop-yarn-server-timelineservice,
> hbase-server, hbase-prefix-tree, hbase-hadoop2-compat,
> hadoop-mapreduce-client-jobclient, hadoop-mapreduce-client-common]
> {code}
> This jira proposes we move all HBase-backend-related code from
> hadoop-yarn-server-timelineservice into its own module (possible name is
> yarn-server-timelineservice-storage) so that core RM modules do not depend on
> HBase modules any more.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4126) RM should not issue delegation tokens in unsecure mode
[ https://issues.apache.org/jira/browse/YARN-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15736904#comment-15736904 ] Andrew Wang commented on YARN-4126: --- I'd like to revert this change while discussion continues (using YARN-5882 as a release notes vehicle). Assuming that Oozie wants to keep supporting 2.x releases there must be an Oozie side fix, and keeping the 3.x behavior inline with 2.x is a conservative choice. > RM should not issue delegation tokens in unsecure mode > -- > > Key: YARN-4126 > URL: https://issues.apache.org/jira/browse/YARN-4126 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Bibin A Chundatt > Fix For: 3.0.0-alpha1 > > Attachments: 0001-YARN-4126.patch, 0002-YARN-4126.patch, > 0003-YARN-4126.patch, 0004-YARN-4126.patch, 0005-YARN-4126.patch, > 0006-YARN-4126.patch > > > ClientRMService#getDelegationToken is currently returning a delegation token > in insecure mode. We should not return the token if it's in insecure mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5709) Cleanup leader election configs and pluggability
[
https://issues.apache.org/jira/browse/YARN-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-5709:
--
Target Version/s: 2.8.0, 3.0.0-alpha2 (was: 2.8.0)
> Cleanup leader election configs and pluggability
>
>
> Key: YARN-5709
> URL: https://issues.apache.org/jira/browse/YARN-5709
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>Priority: Blocker
> Attachments: yarn-5709-wip.2.patch, yarn-5709.1.patch,
> yarn-5709.2.patch, yarn-5709.3.patch, yarn-5709.4.patch
>
>
> While reviewing YARN-5677 and YARN-5694, I noticed we could make the
> curator-based election code cleaner. It is nicer to get this fixed in 2.8
> before we ship it, but this can be done at a later time as well.
> # By EmbeddedElector, we meant it was running as part of the RM daemon. Since
> the Curator-based elector is also running embedded, I feel the code should be
> checking for {{!curatorBased}} instead of {{isEmbeddedElector}}
> # {{LeaderElectorService}} should probably be named
> {{CuratorBasedEmbeddedElectorService}} or some such.
> # The code that initializes the elector should be at the same place
> irrespective of whether it is curator-based or not.
> # We seem to be caching the CuratorFramework instance in RM. It makes more
> sense for it to be in RMContext. If others are okay with it, we might even be
> better of having {{RMContext#getCurator()}} method to lazily create the
> curator framework and then cache it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5536) Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout
[ https://issues.apache.org/jira/browse/YARN-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5536: -- Priority: Major (was: Blocker) I'm going to downgrade this to Major, since it sounds like this additional config support can be added compatibly. > Multiple format support (JSON, etc.) for exclude node file in NM graceful > decommission with timeout > --- > > Key: YARN-5536 > URL: https://issues.apache.org/jira/browse/YARN-5536 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Junping Du > > Per discussion in YARN-4676, we agree that multiple format (other than xml) > should be supported to decommission nodes with timeout values. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5963) Spelling errors in logging and exceptions for node manager, client, web-proxy, common, and app history code
[ https://issues.apache.org/jira/browse/YARN-5963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5963: -- Fix Version/s: 3.0.0-alpha2 2.9.0 > Spelling errors in logging and exceptions for node manager, client, > web-proxy, common, and app history code > --- > > Key: YARN-5963 > URL: https://issues.apache.org/jira/browse/YARN-5963 > Project: Hadoop YARN > Issue Type: Bug > Components: client, nodemanager >Reporter: Grant Sohn >Assignee: Grant Sohn >Priority: Trivial > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5963.1.patch > > > A set of spelling errors in the exceptions and logging messages. > Examples: > accessable -> accessible > occured -> occurred > autorized -> authorized -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5761) Separate QueueManager from Scheduler
[ https://issues.apache.org/jira/browse/YARN-5761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5761: -- Fix Version/s: 3.0.0-alpha2 2.9.0 > Separate QueueManager from Scheduler > > > Key: YARN-5761 > URL: https://issues.apache.org/jira/browse/YARN-5761 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Xuan Gong >Assignee: Xuan Gong > Labels: oct16-medium > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5761.1.patch, YARN-5761.1.rebase.patch, > YARN-5761.2.patch, YARN-5761.3.patch, YARN-5761.4.patch, YARN-5761.5.patch, > YARN-5761.6.patch, YARN-5761.7.patch, YARN-5761.7.patch, YARN-5761.8.patch > > > Currently, in scheduler code, we are doing queue manager and scheduling work. > We'd better separate the queue manager out of scheduler logic. In that case, > it would be much easier and safer to extend. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4126) RM should not issue delegation tokens in unsecure mode
[ https://issues.apache.org/jira/browse/YARN-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752344#comment-15752344 ] Andrew Wang commented on YARN-4126: --- Thanks for reverting [~jianhe], though in the future anything that's not a full revert should go through code review, and not be committed with a message that says "Revert ...". I'll use YARN-5882 to log the fact that essentially test-only changes that were made, to trunk/branch-2/branch-2.8. Otherwise the test changes aren't logged by any JIRA. > RM should not issue delegation tokens in unsecure mode > -- > > Key: YARN-4126 > URL: https://issues.apache.org/jira/browse/YARN-4126 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jian He >Assignee: Bibin A Chundatt > Fix For: 3.0.0-alpha1 > > Attachments: 0001-YARN-4126.patch, 0002-YARN-4126.patch, > 0003-YARN-4126.patch, 0004-YARN-4126.patch, 0005-YARN-4126.patch, > 0006-YARN-4126.patch > > > ClientRMService#getDelegationToken is currently returning a delegation token > in insecure mode. We should not return the token if it's in insecure mode. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5882) Test only changes from YARN-4126
[ https://issues.apache.org/jira/browse/YARN-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5882: -- Assignee: Jian He Priority: Major (was: Blocker) Issue Type: Improvement (was: Bug) Summary: Test only changes from YARN-4126 (was: Reconsider YARN-4126 RM should not issue delegation tokens in unsecure mode) > Test only changes from YARN-4126 > > > Key: YARN-5882 > URL: https://issues.apache.org/jira/browse/YARN-5882 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.0.0-alpha1 >Reporter: Andrew Wang >Assignee: Jian He > > Filed this for tracking. We might want to revert YARN-4126 based on > discussion in that JIRA. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-5882) Test only changes from YARN-4126
[
https://issues.apache.org/jira/browse/YARN-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang resolved YARN-5882.
---
Resolution: Fixed
Fix Version/s: 3.0.0-alpha2
2.8.0
Resolving this JIRA to log that the initial commit for YARN-4126 and subsequent
addendum commit left the test-only changes in branch-2, branch-2.8, and trunk.
We like these changes, but it needs a resolved JIRA to show up in the release
notes.
For future reference, the two commits are as follows:
{noformat}
commit ada876cd1d22b61f237603cf339bbed65285dab8
Author: Jian He
Date: Tue Dec 13 20:49:54 2016 -0800
Revert YARN-4126. RM should not issue delegation tokens in unsecure mode.
commit e1b1d7e4aebfed0dec4d7df21561ab37f73ef1d7
Author: Jian He
Date: Mon Sep 14 14:09:19 2015 +0800
YARN-4126. RM should not issue delegation tokens in unsecure mode.
Contributed by Bibin A Chundatt
{noformat}
> Test only changes from YARN-4126
>
>
> Key: YARN-5882
> URL: https://issues.apache.org/jira/browse/YARN-5882
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha1
>Reporter: Andrew Wang
>Assignee: Jian He
> Fix For: 2.8.0, 3.0.0-alpha2
>
>
> Filed this for tracking. We might want to revert YARN-4126 based on
> discussion in that JIRA.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5464) Server-Side NM Graceful Decommissioning with RM HA
[ https://issues.apache.org/jira/browse/YARN-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5464: -- Target Version/s: 2.9.0, 3.0.0-beta1 (was: 2.9.0, 3.0.0-alpha2) > Server-Side NM Graceful Decommissioning with RM HA > -- > > Key: YARN-5464 > URL: https://issues.apache.org/jira/browse/YARN-5464 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Robert Kanter >Assignee: Robert Kanter >Priority: Blocker > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5646) Documentation for scheduling of OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/YARN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5646: -- Target Version/s: 3.0.0-beta1 (was: 3.0.0-alpha2) > Documentation for scheduling of OPPORTUNISTIC containers > > > Key: YARN-5646 > URL: https://issues.apache.org/jira/browse/YARN-5646 > Project: Hadoop YARN > Issue Type: Task >Reporter: Konstantinos Karanasos >Assignee: Konstantinos Karanasos >Priority: Blocker > Attachments: YARN-5646.001.patch > > > This is for adding documentation regarding the scheduling of OPPORTUNISTIC > containers. > It includes both the centralized (YARN-5220) and the distributed (YARN-2877) > scheduling. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5709) Cleanup leader election configs and pluggability
[
https://issues.apache.org/jira/browse/YARN-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-5709:
--
Fix Version/s: 3.0.0-alpha2
> Cleanup leader election configs and pluggability
>
>
> Key: YARN-5709
> URL: https://issues.apache.org/jira/browse/YARN-5709
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>Priority: Blocker
> Fix For: 3.0.0-alpha2
>
> Attachments: yarn-5709-wip.2.patch, yarn-5709.1.patch,
> yarn-5709.2.patch, yarn-5709.3.patch, yarn-5709.4.patch
>
>
> While reviewing YARN-5677 and YARN-5694, I noticed we could make the
> curator-based election code cleaner. It is nicer to get this fixed in 2.8
> before we ship it, but this can be done at a later time as well.
> # By EmbeddedElector, we meant it was running as part of the RM daemon. Since
> the Curator-based elector is also running embedded, I feel the code should be
> checking for {{!curatorBased}} instead of {{isEmbeddedElector}}
> # {{LeaderElectorService}} should probably be named
> {{CuratorBasedEmbeddedElectorService}} or some such.
> # The code that initializes the elector should be at the same place
> irrespective of whether it is curator-based or not.
> # We seem to be caching the CuratorFramework instance in RM. It makes more
> sense for it to be in RMContext. If others are okay with it, we might even be
> better of having {{RMContext#getCurator()}} method to lazily create the
> curator framework and then cache it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3866) AM-RM protocol changes to support container resizing
[
https://issues.apache.org/jira/browse/YARN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752768#comment-15752768
]
Andrew Wang commented on YARN-3866:
---
Our compat guidelines available on the website, e.g.:
http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/Compatibility.html
I think this is what applies here:
{quote}
Public-Stable APIs must be deprecated for at least one major release prior to
their removal in a major release.
{quote}
Unless these APIs were deprecated in 2.1.0-beta, then we should wait to remove.
> AM-RM protocol changes to support container resizing
>
>
> Key: YARN-3866
> URL: https://issues.apache.org/jira/browse/YARN-3866
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api
>Reporter: MENG DING
>Assignee: MENG DING
>Priority: Blocker
> Fix For: 2.8.0, 3.0.0-alpha1
>
> Attachments: YARN-3866-YARN-1197.4.patch,
> YARN-3866-branch-2.8.addendum.1.patch, YARN-3866-branch-2.8.addendum.1.patch,
> YARN-3866-branch-2.8.addendum.2.patch, YARN-3866-branch-2.8.addendum.3.patch,
> YARN-3866.1.patch, YARN-3866.2.patch, YARN-3866.3.patch
>
>
> YARN-1447 and YARN-1448 are outdated.
> This ticket deals with AM-RM Protocol changes to support container resize
> according to the latest design in YARN-1197.
> 1) Add increase/decrease requests in AllocateRequest
> 2) Get approved increase/decrease requests from RM in AllocateResponse
> 3) Add relevant test cases
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5999) AMRMClientAsync will stop if any exceptions thrown on allocate call
[
https://issues.apache.org/jira/browse/YARN-5999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-5999:
--
Fix Version/s: 3.0.0-alpha2
> AMRMClientAsync will stop if any exceptions thrown on allocate call
>
>
> Key: YARN-5999
> URL: https://issues.apache.org/jira/browse/YARN-5999
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jian He
>Assignee: Jian He
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5999.1.patch
>
>
> Currently, for any exceptions thrown on the allocate call of AMRMClientAsync,
> it will stop both heartbeat thread and the callback handler thread, leaving
> AMRMClient in an unusable state. Caller has to instantiate a new AMRMClient.
> IMO, the threads should keep on running, it should be up to the caller
> whether to stop the AMRMClient or not.
> {code}
> try {
> response = client.allocate(progress);
> } catch (ApplicationAttemptNotFoundException e) {
> handler.onShutdownRequest();
> LOG.info("Shutdown requested. Stopping callback.");
> return;
> } catch (Throwable ex) {
> LOG.error("Exception on heartbeat", ex);
> savedException = ex;
> // interrupt handler thread in case it waiting on the queue
> handlerThread.interrupt();
> return;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5923) Unable to access logs for a running application if YARN_ACL_ENABLE is enabled
[ https://issues.apache.org/jira/browse/YARN-5923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5923: -- Fix Version/s: 3.0.0-alpha2 > Unable to access logs for a running application if YARN_ACL_ENABLE is enabled > - > > Key: YARN-5923 > URL: https://issues.apache.org/jira/browse/YARN-5923 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5923.1.patch, YARN-5923.2.patch, YARN-5923.3.patch > > > 2016-11-07 23:20:41,423 WARN webapp.GenericExceptionHandler > (GenericExceptionHandler.java:toResponse(98)) - INTERNAL_SERVER_ERROR > javax.ws.rs.WebApplicationException: > org.apache.hadoop.yarn.exceptions.YarnException: User [dr.who] is not > authorized to view the logs for application application_1478218837976_0068 > at > org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebServices.getContainerLogsInfo(NMWebServices.java:226) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) > at > com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185) > at > com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) > at > com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288) > at > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > at > com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) > at > com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) > at > com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469) > at > com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400) > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) > at > com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) > at > com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) > at > com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834) > at > org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebAppFilter.doFilter(NMWebAppFilter.java:72) > at > com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795) > at > com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163) > at > com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58) > at > com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118) > at > com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1294) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilte
[jira] [Updated] (YARN-6049) Graceful Decommission web link is broken, gives 404 Not Found
[ https://issues.apache.org/jira/browse/YARN-6049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-6049: -- Fix Version/s: (was: 3.0.0-alpha2) > Graceful Decommission web link is broken, gives 404 Not Found > - > > Key: YARN-6049 > URL: https://issues.apache.org/jira/browse/YARN-6049 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha1 >Reporter: Emre Sevinç >Priority: Minor > Labels: documentation, easyfix > > Graceful Decommission page, > http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html, > is broken. It gives "Not Found The requested URL > /docs/current/hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html was not > found on this server." error. > There are links to this problematic web page from all of the HTML pages in > http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6049) Graceful Decommission web link is broken, gives 404 Not Found
[ https://issues.apache.org/jira/browse/YARN-6049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-6049: -- Target Version/s: 3.0.0-alpha2 (was: 3.0.0-alpha1) > Graceful Decommission web link is broken, gives 404 Not Found > - > > Key: YARN-6049 > URL: https://issues.apache.org/jira/browse/YARN-6049 > Project: Hadoop YARN > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0-alpha1 >Reporter: Emre Sevinç >Priority: Minor > Labels: documentation, easyfix > > Graceful Decommission page, > http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html, > is broken. It gives "Not Found The requested URL > /docs/current/hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html was not > found on this server." error. > There are links to this problematic web page from all of the HTML pages in > http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5569) YARN logs out user when job is run when improperly configured
Andrew Wang created YARN-5569:
-
Summary: YARN logs out user when job is run when improperly
configured
Key: YARN-5569
URL: https://issues.apache.org/jira/browse/YARN-5569
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.7.3
Reporter: Andrew Wang
This is something that Vinay and I both hit while testing a 2.7.3 RC:
{quote}
I didn't have SSH to localhost set up (new laptop), and when I tried to run the
Pi job, it'd exit my window manager session. I feel there must be a more
developer-friendly solution here.
{quote}
{quote}
Faced same issues as Andrew wang, while running the WordCount job first time in
my new Ubuntu installation, without 'configuring the shuffle handler properly'.
Whole session got logged by closing all other applications open. After
configuring the shuffle handler properly, job was successful though.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5570) NodeManager blocks kill attempts during startup if ResourceManager is down
Andrew Wang created YARN-5570:
-
Summary: NodeManager blocks kill attempts during startup if
ResourceManager is down
Key: YARN-5570
URL: https://issues.apache.org/jira/browse/YARN-5570
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.7.3
Reporter: Andrew Wang
Found while testing a 2.7.3 RC:
{quote}
* If you start the NodeManager and not the RM, the NM has a handler for SIGTERM
and SIGINT that blocked my Ctrl-C and kill attempts during startup. I had to
kill -9 it.
{quote}
[~ste...@apache.org] tells me that this is potentially addressed by YARN-679,
though that's a large patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3225) New parameter or CLI for decommissioning node gracefully in RMAdmin CLI
[ https://issues.apache.org/jira/browse/YARN-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447509#comment-15447509 ] Andrew Wang commented on YARN-3225: --- FYI for git log greppers that the JIRA ID is missing in the commit message. > New parameter or CLI for decommissioning node gracefully in RMAdmin CLI > --- > > Key: YARN-3225 > URL: https://issues.apache.org/jira/browse/YARN-3225 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Junping Du >Assignee: Devaraj K > Fix For: 2.8.0 > > Attachments: YARN-3225-1.patch, YARN-3225-2.patch, YARN-3225-3.patch, > YARN-3225-4.patch, YARN-3225-5.patch, YARN-3225.patch, YARN-914.patch > > > New CLI (or existing CLI with parameters) should put each node on > decommission list to decommissioning status and track timeout to terminate > the nodes that haven't get finished. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-4072) ApplicationHistoryServer, WebAppProxyServer, NodeManager and ResourceManager to support JvmPauseMonitor as a service
[ https://issues.apache.org/jira/browse/YARN-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened YARN-4072: --- > ApplicationHistoryServer, WebAppProxyServer, NodeManager and ResourceManager > to support JvmPauseMonitor as a service > > > Key: YARN-4072 > URL: https://issues.apache.org/jira/browse/YARN-4072 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, resourcemanager >Affects Versions: 2.8.0 >Reporter: Sunil G >Assignee: Sunil G >Priority: Minor > Attachments: 0001-YARN-4072.patch, 0002-YARN-4072.patch, > HADOOP-12321-005-aggregated.patch, HADOOP-12407-001.patch > > > As JvmPauseMonitor is made as an AbstractService, subsequent method changes > are needed in all places which uses the monitor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-4072) ApplicationHistoryServer, WebAppProxyServer, NodeManager and ResourceManager to support JvmPauseMonitor as a service
[ https://issues.apache.org/jira/browse/YARN-4072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved YARN-4072. --- Resolution: Duplicate Fix Version/s: (was: 2.9.0) > ApplicationHistoryServer, WebAppProxyServer, NodeManager and ResourceManager > to support JvmPauseMonitor as a service > > > Key: YARN-4072 > URL: https://issues.apache.org/jira/browse/YARN-4072 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, resourcemanager >Affects Versions: 2.8.0 >Reporter: Sunil G >Assignee: Sunil G >Priority: Minor > Attachments: 0001-YARN-4072.patch, 0002-YARN-4072.patch, > HADOOP-12321-005-aggregated.patch, HADOOP-12407-001.patch > > > As JvmPauseMonitor is made as an AbstractService, subsequent method changes > are needed in all places which uses the monitor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4717) TestResourceLocalizationService.testPublicResourceInitializesLocalDir fails Intermittently due to IllegalArgumentException from cleanup
[ https://issues.apache.org/jira/browse/YARN-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447516#comment-15447516 ] Andrew Wang commented on YARN-4717: --- FYI for future git log greppers, this one is missing the JIRA #. > TestResourceLocalizationService.testPublicResourceInitializesLocalDir fails > Intermittently due to IllegalArgumentException from cleanup > --- > > Key: YARN-4717 > URL: https://issues.apache.org/jira/browse/YARN-4717 > Project: Hadoop YARN > Issue Type: Test > Components: nodemanager >Affects Versions: 2.6.0 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Fix For: 2.8.0, 2.7.4 > > Attachments: YARN-4717.001.patch > > > The same issue that was resolved by [~zxu] in YARN-3602 is back. Looks like > the commons-io package throws an IAE instead of an IOE now if the directory > doesn't exist. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4749) Generalize config file handling in container-executor
[
https://issues.apache.org/jira/browse/YARN-4749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447520#comment-15447520
]
Andrew Wang commented on YARN-4749:
---
FYI for future git log greppers, the JIRA # was typo'd as YARN-4245.
> Generalize config file handling in container-executor
> -
>
> Key: YARN-4749
> URL: https://issues.apache.org/jira/browse/YARN-4749
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
> Fix For: 2.9.0
>
> Attachments: YARN-4749.001.patch, YARN-4749.002.patch
>
>
> The current implementation of container-executor already supports parsing of
> key value pairs from a config file. However, it is currently restricted to
> {{container-executor.cfg}} and cannot be reused for parsing additional
> config/command files. Generalizing this is a required step for YARN-4245.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4815) ATS 1.5 timelineclient impl try to create attempt directory for every event call
[
https://issues.apache.org/jira/browse/YARN-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447527#comment-15447527
]
Andrew Wang commented on YARN-4815:
---
FYI for git log greppers, this JIRA # was typo'd as YARN-4814 (with the same
commit message as the actual YARN-4814). This is it the commit though:
{quote}
commit af1d125f9ce35ec69a610674a1c5c60cc17141a7
Author: Junping Du
Date: Wed Mar 23 08:57:16 2016 -0700
YARN-4814. ATS 1.5 timelineclient impl call flush after every event write.
Contributed by Xuan Gong.
{quote}
> ATS 1.5 timelineclient impl try to create attempt directory for every event
> call
>
>
> Key: YARN-4815
> URL: https://issues.apache.org/jira/browse/YARN-4815
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 2.8.0
>
> Attachments: YARN-4815.1.patch, YARN-4815.2.patch, YARN-4815.3.patch
>
>
> ATS 1.5 timelineclinet impl, try to create attempt directory for every event
> call. Since per attempt only one call to create directory is enough, this is
> causing perf issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4955) Add retry for SocketTimeoutException in TimelineClient
[
https://issues.apache.org/jira/browse/YARN-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447532#comment-15447532
]
Andrew Wang commented on YARN-4955:
---
FYI for git log greppers that this one is missing commit message, here's the
commit (msg a bit different from JIRA summary):
{quote}
commit 477003730e6a7c7eff11892f5cedf74073ca867b
Author: Vinod Kumar Vavilapalli
Date: Mon Apr 18 11:47:06 2016 -0700
Fixed TimelineClient to retry SocketTimeoutException too. Contributed by
Xuan Gong.
{quote}
> Add retry for SocketTimeoutException in TimelineClient
> --
>
> Key: YARN-4955
> URL: https://issues.apache.org/jira/browse/YARN-4955
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: YARN-4955.1.patch, YARN-4955.2.patch, YARN-4955.3.patch,
> YARN-4955.4-1.patch, YARN-4955.4.patch, YARN-4955.5.patch, YARN-4955.6.patch
>
>
> We saw this exception several times when we tried to getDelegationToken from
> ATS.
> java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> java.net.SocketTimeoutException: Read timed out
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$5.run(TimelineClientImpl.java:569)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$TimelineClientConnectionRetry.retryOn(TimelineClientImpl.java:234)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.operateDelegationToken(TimelineClientImpl.java:582)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.getDelegationToken(TimelineClientImpl.java:479)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getTimelineDelegationToken(YarnClientImpl.java:349)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.addTimelineDelegationToken(YarnClientImpl.java:330)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:250)
> at
> org.apache.hadoop.mapred.ResourceMgrDelegate.submitApplication(ResourceMgrDelegate.java:291)
> at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:290)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
> at
> org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:194)
> at java.lang.Thread.run(Thread.java:745)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:276)
> Caused by:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> java.net.SocketTimeoutException: Read timed out
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128)
> at
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:285)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:166)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:371)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$2.run(TimelineClientImpl.java:475)
> at
> org.a
[jira] [Commented] (YARN-5002) getApplicationReport call may raise NPE for removed queues
[
https://issues.apache.org/jira/browse/YARN-5002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447535#comment-15447535
]
Andrew Wang commented on YARN-5002:
---
FYI for git log greppers, this one is missing the JIRA ID in commit message.
> getApplicationReport call may raise NPE for removed queues
> --
>
> Key: YARN-5002
> URL: https://issues.apache.org/jira/browse/YARN-5002
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sumana Sathish
>Assignee: Jian He
>Priority: Critical
> Fix For: 2.9.0
>
> Attachments: YARN-5002.1.patch, YARN-5002.2.patch, YARN-5002.3.patch
>
>
> getApplicationReport call may raise NPE
> {code}
> Exception in thread "main" java.lang.NullPointerException:
> java.lang.NullPointerException
>
> org.apache.hadoop.yarn.server.resourcemanager.security.QueueACLsManager.checkAccess(QueueACLsManager.java:57)
>
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.checkAccess(ClientRMService.java:279)
>
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:760)
>
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.getApplications(ClientRMService.java:682)
>
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.getApplications(ApplicationClientProtocolPBServiceImpl.java:234)
>
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:425)
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2268)
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2264)
> java.security.AccessController.doPrivileged(Native Method)
> javax.security.auth.Subject.doAs(Subject.java:422)
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1708)
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:2262)
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:107)
>
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplications(ApplicationClientProtocolPBClientImpl.java:254)
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> java.lang.reflect.Method.invoke(Method.java:498)
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> com.sun.proxy.$Proxy18.getApplications(Unknown Source)
>
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplications(YarnClientImpl.java:479)
>
> org.apache.hadoop.mapred.ResourceMgrDelegate.getAllJobs(ResourceMgrDelegate.java:135)
> org.apache.hadoop.mapred.YARNRunner.getAllJobs(YARNRunner.java:167)
> org.apache.hadoop.mapreduce.Cluster.getAllJobStatuses(Cluster.java:294)
> org.apache.hadoop.mapreduce.tools.CLI.listJobs(CLI.java:553)
> org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:338)
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> org.apache.hadoop.mapred.JobClient.main(JobClient.java:1274)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5337) Dshell AM failed with "java.lang.OutOfMemoryError: GC overhead limit exceeded"
[
https://issues.apache.org/jira/browse/YARN-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447539#comment-15447539
]
Andrew Wang commented on YARN-5337:
---
Tracked this one down in git log, here's the commit:
{quote}
commit 9bdb5bebea1183ec2f697ee3e55392df4fe697bb
Author: Junping Du
Date: Sat Jul 9 05:09:53 2016 -0700
Fix OOM issue in DistributedShell. Contributed by Jian He.
{quote}
> Dshell AM failed with "java.lang.OutOfMemoryError: GC overhead limit exceeded"
> --
>
> Key: YARN-5337
> URL: https://issues.apache.org/jira/browse/YARN-5337
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Sumana Sathish
>Assignee: Jian He
> Fix For: 2.8.0
>
> Attachments: YARN-5337.1.patch
>
>
> Please find AM logs with the exception
> {code}
> INFO distributedshell.ApplicationMaster: Container completed successfully.,
> containerId=container_e49_1467633982200_0001_01_04
> Exception in thread "AMRM Callback Handler Thread"
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at
> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:312)
> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.lang.Object.clone(Native Method)
> at java.lang.reflect.Method.getParameterTypes(Method.java:264)
> at
> java.lang.reflect.Executable.getGenericParameterTypes(Executable.java:285)
> at java.lang.reflect.Method.getGenericParameterTypes(Method.java:283)
> at
> org.codehaus.jackson.map.introspect.AnnotatedMethod.getParameterTypes(AnnotatedMethod.java:143)
> at
> org.codehaus.jackson.map.introspect.AnnotatedMethod.getParameterCount(AnnotatedMethod.java:139)
> at
> org.codehaus.jackson.map.introspect.POJOPropertiesCollector._addMethods(POJOPropertiesCollector.java:427)
> at
> org.codehaus.jackson.map.introspect.POJOPropertiesCollector.collect(POJOPropertiesCollector.java:219)
> at
> org.codehaus.jackson.map.introspect.BasicClassIntrospector.collectProperties(BasicClassIntrospector.java:160)
> at
> org.codehaus.jackson.map.introspect.BasicClassIntrospector.forSerialization(BasicClassIntrospector.java:96)
> at
> org.codehaus.jackson.map.introspect.BasicClassIntrospector.forSerialization(BasicClassIntrospector.java:16)
> at
> org.codehaus.jackson.map.SerializationConfig.introspect(SerializationConfig.java:973)
> at
> org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:251)
> at
> org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782)
> at
> org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735)
> at
> org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344)
> at
> org.codehaus.jackson.map.ser.impl.PropertySerializerMap.findAndAddSerializer(PropertySerializerMap.java:39)
> at
> org.codehaus.jackson.map.ser.std.MapSerializer._findAndAddDynamic(MapSerializer.java:403)
> at
> org.codehaus.jackson.map.ser.std.MapSerializer.serializeFields(MapSerializer.java:257)
> at
> org.codehaus.jackson.map.ser.std.MapSerializer.serialize(MapSerializer.java:186)
> at
> org.codehaus.jackson.map.ser.std.MapSerializer.serialize(MapSerializer.java:23)
> at
> org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:446)
> at
> org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:150)
> at
> org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:112)
> at
> org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.serializeContents(StdContainerSerializers.java:122)
> at
> org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.serializeContents(StdContainerSerializers.java:71)
> at
> org.codehaus.jackson.map.ser.std.AsArraySerializerBase.serialize(AsArraySerializerBase.java:86)
> at
> org.codehaus.jackson.map.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:446)
> at
> org.codehaus.jackson.map.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:150)
> at
> org.codehaus.jackson.map.ser.BeanSerializer.serialize(BeanSerializer.java:112)
> at
> org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.serializeContents(StdContainerSerializers.java:122)
> at
> org.codehaus.jackson.map.ser.std.StdContainerSerializers$IndexedListSerializer.seria
[jira] [Commented] (YARN-5526) DrainDispacher#ServiceStop blocked if setDrainEventsOnStop invoked
[
https://issues.apache.org/jira/browse/YARN-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447542#comment-15447542
]
Andrew Wang commented on YARN-5526:
---
FYI for future git log greppers, this one was minorly typo'd as YARN=5526:
{quote}
commit 913a8951366a5d816bbae253b230250e0b74849e
Author: Varun Saxena
Date: Thu Aug 18 11:38:13 2016 +0530
YARN=5526. DrainDispacher#serviceStop blocked if setDrainEventsOnStop
invoked (sandflee via Varun Saxena)
{quote}
> DrainDispacher#ServiceStop blocked if setDrainEventsOnStop invoked
> --
>
> Key: YARN-5526
> URL: https://issues.apache.org/jira/browse/YARN-5526
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: sandflee
>Assignee: sandflee
> Fix For: 2.9.0
>
> Attachments: YARN-5526.01.patch, YARN-5526.02.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5567) Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus
[
https://issues.apache.org/jira/browse/YARN-5567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15456763#comment-15456763
]
Andrew Wang commented on YARN-5567:
---
Looking at git log, it looks like this will also be included in alpha1. I
rebranched right before sending the RC, and we picked up this JIRA as part of
that.
So, I think we're good?
> Fix script exit code checking in NodeHealthScriptRunner#reportHealthStatus
> --
>
> Key: YARN-5567
> URL: https://issues.apache.org/jira/browse/YARN-5567
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.8.0, 3.0.0-alpha1
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 3.0.0-alpha1
>
> Attachments: YARN-5567.001.patch
>
>
> In case of FAILED_WITH_EXIT_CODE, health status should be false.
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(true, "", now);
> break;
> {code}
> should be
> {code}
> case FAILED_WITH_EXIT_CODE:
> setHealthStatus(false, "", now);
> break;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5221) Expose UpdateResourceRequest API to allow AM to request for change in container properties
[ https://issues.apache.org/jira/browse/YARN-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5221: -- Fix Version/s: 3.0.0-alpha2 2.8.0 > Expose UpdateResourceRequest API to allow AM to request for change in > container properties > -- > > Key: YARN-5221 > URL: https://issues.apache.org/jira/browse/YARN-5221 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: YARN-5221-branch-2-v1.patch, > YARN-5221-branch-2.8-v1.patch, YARN-5221.001.patch, YARN-5221.002.patch, > YARN-5221.003.patch, YARN-5221.004.patch, YARN-5221.005.patch, > YARN-5221.006.patch, YARN-5221.007.patch, YARN-5221.008.patch, > YARN-5221.009.patch, YARN-5221.010.patch, YARN-5221.011.patch, > YARN-5221.012.patch, YARN-5221.013.patch > > > YARN-1197 introduced APIs to allow an AM to request for Increase and Decrease > of Container Resources after initial allocation. > YARN-5085 proposes to allow an AM to request for a change of Container > ExecutionType. > This JIRA proposes to unify both of the above into an Update Container API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5596) Fix failing unit test in TestDockerContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-5596:
--
Fix Version/s: 3.0.0-alpha2
> Fix failing unit test in TestDockerContainerRuntime
> ---
>
> Key: YARN-5596
> URL: https://issues.apache.org/jira/browse/YARN-5596
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5596.001.patch, YARN-5596.002.patch
>
>
> /sys/fs/cgroup doesn't exist on Mac OS X. And the tests seem to fail because
> of this.
> {code}
> Failed tests:
> TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks:456
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testContainerLaunchWithNetworkingDefaults:401
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testDockerContainerLaunch:297
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> Tests run: 19, Failures: 3, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5596) Fix failing unit test in TestDockerContainerRuntime
[
https://issues.apache.org/jira/browse/YARN-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15459673#comment-15459673
]
Andrew Wang commented on YARN-5596:
---
Gentle reminder to please include the appropriate 3.0.0 fix version when
committing to trunk, thanks!
> Fix failing unit test in TestDockerContainerRuntime
> ---
>
> Key: YARN-5596
> URL: https://issues.apache.org/jira/browse/YARN-5596
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, yarn
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: YARN-5596.001.patch, YARN-5596.002.patch
>
>
> /sys/fs/cgroup doesn't exist on Mac OS X. And the tests seem to fail because
> of this.
> {code}
> Failed tests:
> TestDockerContainerRuntime.testContainerLaunchWithCustomNetworks:456
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testContainerLaunchWithNetworkingDefaults:401
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> TestDockerContainerRuntime.testDockerContainerLaunch:297
> expected:<...ET_BIND_SERVICE -v /[sys/fs/cgroup:/sys/fs/cgroup:ro -v
> /]test_container_local...> but was:<...ET_BIND_SERVICE -v
> /[]test_container_local...>
> Tests run: 19, Failures: 3, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5555) Scheduler UI: "% of Queue" is inaccurate if leaf queue is hierarchically nested.
[
https://issues.apache.org/jira/browse/YARN-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated YARN-:
--
Fix Version/s: 3.0.0-alpha2
> Scheduler UI: "% of Queue" is inaccurate if leaf queue is hierarchically
> nested.
>
>
> Key: YARN-
> URL: https://issues.apache.org/jira/browse/YARN-
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: PctOfQueueIsInaccurate.jpg, YARN-.001.patch
>
>
> If a leaf queue is hierarchically nested (e.g., {{root.a.a1}},
> {{root.a.a2}}), the values in the "*% of Queue*" column in the apps section
> of the Scheduler UI is calculated as if the leaf queue ({{a1}}) were a direct
> child of {{root}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5221) Expose UpdateResourceRequest API to allow AM to request for change in container properties
[ https://issues.apache.org/jira/browse/YARN-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15459680#comment-15459680 ] Andrew Wang commented on YARN-5221: --- Gentle reminder to please include the appropriate 3.0.0 fix version when committing to trunk, thanks! I also set 2.8.0 since I think the target version was set by mistake instead. > Expose UpdateResourceRequest API to allow AM to request for change in > container properties > -- > > Key: YARN-5221 > URL: https://issues.apache.org/jira/browse/YARN-5221 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: YARN-5221-branch-2-v1.patch, > YARN-5221-branch-2.8-v1.patch, YARN-5221.001.patch, YARN-5221.002.patch, > YARN-5221.003.patch, YARN-5221.004.patch, YARN-5221.005.patch, > YARN-5221.006.patch, YARN-5221.007.patch, YARN-5221.008.patch, > YARN-5221.009.patch, YARN-5221.010.patch, YARN-5221.011.patch, > YARN-5221.012.patch, YARN-5221.013.patch > > > YARN-1197 introduced APIs to allow an AM to request for Increase and Decrease > of Container Resources after initial allocation. > YARN-5085 proposes to allow an AM to request for a change of Container > ExecutionType. > This JIRA proposes to unify both of the above into an Update Container API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5559) Analyse 2.8.0/3.0.0 jdiff reports and fix any issues
[ https://issues.apache.org/jira/browse/YARN-5559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5559: -- Target Version/s: 2.8.0, 3.0.0-alpha2 (was: 2.8.0, 3.0.0-alpha1) > Analyse 2.8.0/3.0.0 jdiff reports and fix any issues > > > Key: YARN-5559 > URL: https://issues.apache.org/jira/browse/YARN-5559 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Blocker > Attachments: YARN-5559.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
