[jira] [Commented] (YARN-6424) TimelineCollector is not stopped when an app finishes in RM
[ https://issues.apache.org/jira/browse/YARN-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958299#comment-15958299 ] Rohith Sharma K S commented on YARN-6424: - Yes it is bug!! I committed to trunk and YARN-5355-branch-2 also. > TimelineCollector is not stopped when an app finishes in RM > --- > > Key: YARN-6424 > URL: https://issues.apache.org/jira/browse/YARN-6424 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.0.0-alpha2 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Critical > Fix For: YARN-5355, YARN-5355-branch-2, 3.0.0-alpha3 > > Attachments: YARN-6424.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6424) TimelineCollector is not stopped when an app finishes in RM
[ https://issues.apache.org/jira/browse/YARN-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-6424: Fix Version/s: 3.0.0-alpha3 YARN-5355-branch-2 > TimelineCollector is not stopped when an app finishes in RM > --- > > Key: YARN-6424 > URL: https://issues.apache.org/jira/browse/YARN-6424 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.0.0-alpha2 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Critical > Fix For: YARN-5355, YARN-5355-branch-2, 3.0.0-alpha3 > > Attachments: YARN-6424.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6424) TimelineCollector is not stopped when an app finishes in RM
[ https://issues.apache.org/jira/browse/YARN-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-6424: Issue Type: Bug (was: Sub-task) Parent: (was: YARN-5355) > TimelineCollector is not stopped when an app finishes in RM > --- > > Key: YARN-6424 > URL: https://issues.apache.org/jira/browse/YARN-6424 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.0.0-alpha2 >Reporter: Varun Saxena >Assignee: Varun Saxena >Priority: Critical > Fix For: YARN-5355 > > Attachments: YARN-6424.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6403) Invalid local resource request can raise NPE and make NM exit
[
https://issues.apache.org/jira/browse/YARN-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958249#comment-15958249
]
Tao Yang commented on YARN-6403:
[~jlowe], thanks for review and committing!
> Invalid local resource request can raise NPE and make NM exit
> -
>
> Key: YARN-6403
> URL: https://issues.apache.org/jira/browse/YARN-6403
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.8.0
>Reporter: Tao Yang
>Assignee: Tao Yang
> Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3
>
> Attachments: YARN-6403.001.patch, YARN-6403.002.patch,
> YARN-6403.004.patch, YARN-6403.branch-2.8.003.patch,
> YARN-6403.branch-2.8.004.patch, YARN-6403.branch-2.8.004.patch
>
>
> Recently we found this problem on our testing environment. The app that
> caused this problem added a invalid local resource request(have no location)
> into ContainerLaunchContext like this:
> {code}
> localResources.put("test", LocalResource.newInstance(location,
> LocalResourceType.FILE, LocalResourceVisibility.PRIVATE, 100,
> System.currentTimeMillis()));
> ContainerLaunchContext amContainer =
> ContainerLaunchContext.newInstance(localResources, environment,
> vargsFinal, null, securityTokens, acls);
> {code}
> The actual value of location was null although app doesn't expect that. This
> mistake cause several NMs exited with the NPE below and can't restart until
> the nm recovery dirs were deleted.
> {code}
> FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourceRequest.(LocalResourceRequest.java:46)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$RequestResourcesTransition.transition(ContainerImpl.java:711)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$RequestResourcesTransition.transition(ContainerImpl.java:660)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:1320)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:88)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1293)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1286)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> NPE occured when created LocalResourceRequest instance for invalid resource
> request.
> {code}
> public LocalResourceRequest(LocalResource resource)
> throws URISyntaxException {
> this(resource.getResource().toPath(), //NPE occurred here
> resource.getTimestamp(),
> resource.getType(),
> resource.getVisibility(),
> resource.getPattern());
> }
> {code}
> We can't guarantee the validity of local resource request now, but we could
> avoid damaging the cluster. Perhaps we can verify the resource both in
> ContainerLaunchContext and LocalResourceRequest? Please feel free to give
> your suggestions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6453) fairscheduler-statedump.log gets generated regardless of configured scheduler
Allen Wittenauer created YARN-6453: -- Summary: fairscheduler-statedump.log gets generated regardless of configured scheduler Key: YARN-6453 URL: https://issues.apache.org/jira/browse/YARN-6453 Project: Hadoop YARN Issue Type: Bug Components: fairscheduler, scheduler Affects Versions: 3.0.0-alpha3 Reporter: Allen Wittenauer Priority: Minor An empty fairscheduler-statedump.log is getting generated despite fair share not being the configured scheduler. It should not be. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6091) the AppMaster register failed when use Docker on LinuxContainer
[ https://issues.apache.org/jira/browse/YARN-6091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958176#comment-15958176 ] zhengchenyu commented on YARN-6091: --- I just changed the judging condition of pclose's return code, then the problem is solved. > the AppMaster register failed when use Docker on LinuxContainer > > > Key: YARN-6091 > URL: https://issues.apache.org/jira/browse/YARN-6091 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, yarn >Affects Versions: 2.8.1 > Environment: CentOS >Reporter: zhengchenyu >Priority: Critical > Original Estimate: 336h > Remaining Estimate: 336h > > In some servers, When I use Docker on LinuxContainer, I found the aciton that > AppMaster register to Resourcemanager failed. But didn't happen in other > servers. > I found the pclose (in container-executor.c) return different value in > different server, even though the process which is launched by popen is > running normally. Some server return 0, and others return 13. > Because yarn regard the application as failed application when pclose return > nonzero, and yarn will remove the AMRMToken, then the AppMaster register > failed because Resourcemanager have removed this applicaiton's token. > In container-executor.c, the judgement condition is whether the return code > is zero. But man the pclose, the document tells that "pclose return -1" > represent wrong. So I change the judgement condition, then slove this > problem. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6368) Decommissioning an NM results in a -1 exit code
[
https://issues.apache.org/jira/browse/YARN-6368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958111#comment-15958111
]
Miklos Szegedi commented on YARN-6368:
--
I cannot repro this. It must be a flaky test. Moreover this is a change in the
exit code that cannot be tested (the test would exit), so it is unlikely that
it breaks any tests.
> Decommissioning an NM results in a -1 exit code
> ---
>
> Key: YARN-6368
> URL: https://issues.apache.org/jira/browse/YARN-6368
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Minor
> Attachments: YARN-6368.000.patch, YARN-6368.001.patch,
> YARN-6368.002.patch, YARN-6368.003.patch
>
>
> In NodeManager.java we should exit normally in case the RM shuts down the
> node:
> {code}
> } finally {
> if (shouldExitOnShutdownEvent
> && !ShutdownHookManager.get().isShutdownInProgress()) {
> ExitUtil.terminate(-1);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6452) test-container-executor should not be in bin in dist tarball
Allen Wittenauer created YARN-6452: -- Summary: test-container-executor should not be in bin in dist tarball Key: YARN-6452 URL: https://issues.apache.org/jira/browse/YARN-6452 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0-alpha3 Reporter: Allen Wittenauer Priority: Minor test-container-executor should probably be in sbin or libexec or not there at all. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6195) Export UsedCapacity and AbsoluteUsedCapacity to JMX
[ https://issues.apache.org/jira/browse/YARN-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958090#comment-15958090 ] Benson Qiu commented on YARN-6195: -- Version 5 outputs usedCapacity and absoluteUsedCapacity for the default partition. > Export UsedCapacity and AbsoluteUsedCapacity to JMX > --- > > Key: YARN-6195 > URL: https://issues.apache.org/jira/browse/YARN-6195 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler, metrics, yarn >Affects Versions: 3.0.0-alpha3 >Reporter: Benson Qiu >Assignee: Benson Qiu > Attachments: YARN-6195.001.patch, YARN-6195.002.patch, > YARN-6195.003.patch, YARN-6195.004.patch, YARN-6195.005.patch > > > `usedCapacity` and `absoluteUsedCapacity` are currently not available as JMX. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-5949) Add PluggableConfigurationPolicy interface as a component of MutableConfigurationManager
[ https://issues.apache.org/jira/browse/YARN-5949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung reassigned YARN-5949: --- Assignee: Jonathan Hung > Add PluggableConfigurationPolicy interface as a component of > MutableConfigurationManager > > > Key: YARN-5949 > URL: https://issues.apache.org/jira/browse/YARN-5949 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung > > The PluggableConfigurationPolicy component will allow different policies to > be applied to customize how configuration changes should be applied (for > example, a policy might restrict whether a configuration change by a certain > user is allowed). This will be enforced by the MutableConfigurationManager. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6344) Rethinking OFF_SWITCH locality in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958056#comment-15958056
]
Konstantinos Karanasos commented on YARN-6344:
--
Thanks for the feedback, [~jlowe].
Indeed setting the additional rack delay to 0 has the double effect of no
additional rack delay _and_ to _not_ fall back to the old behavior, while -1
just falls back to old behavior.
If we all agree, I wouldn't mind removing the old behavior completely. But as
you say, I don't know the exact reason it is there, so it seems a less
aggressive change to have the choice of falling back to the old behavior.
> Rethinking OFF_SWITCH locality in CapacityScheduler
> ---
>
> Key: YARN-6344
> URL: https://issues.apache.org/jira/browse/YARN-6344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6344.001.patch, YARN-6344.002.patch,
> YARN-6344.003.patch, YARN-6344.004.patch
>
>
> When relaxing locality from node to rack, the {{node-locality-parameter}} is
> used: when scheduling opportunities for a scheduler key are more than the
> value of this parameter, we relax locality and try to assign the container to
> a node in the corresponding rack.
> On the other hand, when relaxing locality to off-switch (i.e., assign the
> container anywhere in the cluster), we are using a {{localityWaitFactor}},
> which is computed based on the number of outstanding requests for a specific
> scheduler key, which is divided by the size of the cluster.
> In case of applications that request containers in big batches (e.g.,
> traditional MR jobs), and for relatively small clusters, the
> localityWaitFactor does not affect relaxing locality much.
> However, in case of applications that request containers in small batches,
> this load factor takes a very small value, which leads to assigning
> off-switch containers too soon. This situation is even more pronounced in big
> clusters.
> For example, if an application requests only one container per request, the
> locality will be relaxed after a single missed scheduling opportunity.
> The purpose of this JIRA is to rethink the way we are relaxing locality for
> off-switch assignments.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6381) FSAppAttempt has several variables that should be final
[
https://issues.apache.org/jira/browse/YARN-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958044#comment-15958044
]
Hudson commented on YARN-6381:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11536 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11536/])
YARN-6381. FSAppAttempt has several variables that should be final (templedf:
rev a2c57bb70d719f72d59413f33ae67781b583b9f8)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSAppAttempt.java
> FSAppAttempt has several variables that should be final
> ---
>
> Key: YARN-6381
> URL: https://issues.apache.org/jira/browse/YARN-6381
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha2
>Reporter: Daniel Templeton
>Assignee: Ameet Zaveri
> Labels: newbie
> Attachments: YARN-6381.001.patch
>
>
> These should all be final:
> {code}
> private long startTime;
> private Priority appPriority;
> private ResourceWeights resourceWeights;
> private FairScheduler scheduler;
> private Map reservations = new HashMap<>();
> private List blacklistNodeIds = new ArrayList<>();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6368) Decommissioning an NM results in a -1 exit code
[
https://issues.apache.org/jira/browse/YARN-6368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958009#comment-15958009
]
Robert Kanter commented on YARN-6368:
-
[~miklos.szeg...@cloudera.com], I was about to commit this but I ran the
modified unit test against the latest trunk and it failed:
{noformat}
Running org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerResync
Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 33.184 sec <<<
FAILURE! - in org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerResync
testNMSentContainerStatusOnResync(org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerResync)
Time elapsed: 0.73 sec <<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertFalse(Assert.java:64)
at org.junit.Assert.assertFalse(Assert.java:74)
at
org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerResync.testNMSentContainerStatusOnResync(TestNodeManagerResync.java:363)
{noformat}
Can you look into this?
> Decommissioning an NM results in a -1 exit code
> ---
>
> Key: YARN-6368
> URL: https://issues.apache.org/jira/browse/YARN-6368
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Minor
> Attachments: YARN-6368.000.patch, YARN-6368.001.patch,
> YARN-6368.002.patch, YARN-6368.003.patch
>
>
> In NodeManager.java we should exit normally in case the RM shuts down the
> node:
> {code}
> } finally {
> if (shouldExitOnShutdownEvent
> && !ShutdownHookManager.get().isShutdownInProgress()) {
> ExitUtil.terminate(-1);
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6448) Continuous scheduling thread crashes while sorting nodes
[
https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958001#comment-15958001
]
Hudson commented on YARN-6448:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11535 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11535/])
YARN-6448. Continuous scheduling thread crashes while sorting nodes. (kasha:
rev b4c4f365948d36b36942f912ef994c1c21ba59e3)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestContinuousScheduling.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNode.java
> Continuous scheduling thread crashes while sorting nodes
>
>
> Key: YARN-6448
> URL: https://issues.apache.org/jira/browse/YARN-6448
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.9.0
>
> Attachments: YARN-6448.001.patch, YARN-6448.002.patch,
> YARN-6448.003.patch, YARN-6448.004.patch
>
>
> YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
> breaks the order in comparison if nodes changes while sorting.
> {code}
> 2017-04-04 23:42:26,123 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.RMCriticalThreadUncaughtExceptionHandler:
> Critical thread FairSchedulerContinuousScheduling crashed!
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker.sortedNodeList(ClusterNodeTracker.java:306)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:884)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:316)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6439) Fix ReservationSystem creation of default ReservationQueue
[
https://issues.apache.org/jira/browse/YARN-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957993#comment-15957993
]
Hadoop QA commented on YARN-6439:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 38m 29s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6439 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862181/YARN-6439.v2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b88b9cd1c802 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7eea8d8 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/15545/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15545/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15545/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Fix ReservationSystem creation of default ReservationQueue
> --
>
> Key: YARN-6439
> URL: https://issues.apache.org/jira/browse/YARN-6439
>
[jira] [Resolved] (YARN-6448) Continuous scheduling thread crashes while sorting nodes
[
https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla resolved YARN-6448.
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.9.0
Just committed this to trunk and branch-2.
Thanks Yufei for reporting and fixing this.
> Continuous scheduling thread crashes while sorting nodes
>
>
> Key: YARN-6448
> URL: https://issues.apache.org/jira/browse/YARN-6448
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Fix For: 2.9.0
>
> Attachments: YARN-6448.001.patch, YARN-6448.002.patch,
> YARN-6448.003.patch, YARN-6448.004.patch
>
>
> YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
> breaks the order in comparison if nodes changes while sorting.
> {code}
> 2017-04-04 23:42:26,123 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.RMCriticalThreadUncaughtExceptionHandler:
> Critical thread FairSchedulerContinuousScheduling crashed!
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker.sortedNodeList(ClusterNodeTracker.java:306)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:884)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:316)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6195) Export UsedCapacity and AbsoluteUsedCapacity to JMX
[
https://issues.apache.org/jira/browse/YARN-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957958#comment-15957958
]
Hadoop QA commented on YARN-6195:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
17s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
0s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 3s{color} | {color:orange} root: The patch generated 7 new + 267 unchanged
- 5 fixed = 274 total (was 272) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m
31s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 39m
7s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}135m 21s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6195 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862149/YARN-6195.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 80c9a9a148fe 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7d963c4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/15534/artifact/patchprocess/diff-checkstyle-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15534/testReport/ |
| modules | C: hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U: . |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15534/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT
[jira] [Updated] (YARN-6448) Continuous scheduling thread crashes while sorting nodes
[
https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated YARN-6448:
---
Summary: Continuous scheduling thread crashes while sorting nodes (was:
Continuous scheduling thread crash while sorting nodes)
> Continuous scheduling thread crashes while sorting nodes
>
>
> Key: YARN-6448
> URL: https://issues.apache.org/jira/browse/YARN-6448
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6448.001.patch, YARN-6448.002.patch,
> YARN-6448.003.patch, YARN-6448.004.patch
>
>
> YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
> breaks the order in comparison if nodes changes while sorting.
> {code}
> 2017-04-04 23:42:26,123 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.RMCriticalThreadUncaughtExceptionHandler:
> Critical thread FairSchedulerContinuousScheduling crashed!
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker.sortedNodeList(ClusterNodeTracker.java:306)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:884)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:316)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6448) Continuous scheduling thread crash while sorting nodes
[
https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957939#comment-15957939
]
Karthik Kambatla commented on YARN-6448:
+1. Committing this based on the Jenkins run against v2 patch. v4 differs only
in the name of the test and fixes the checkstyle issue.
> Continuous scheduling thread crash while sorting nodes
> --
>
> Key: YARN-6448
> URL: https://issues.apache.org/jira/browse/YARN-6448
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6448.001.patch, YARN-6448.002.patch,
> YARN-6448.003.patch, YARN-6448.004.patch
>
>
> YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
> breaks the order in comparison if nodes changes while sorting.
> {code}
> 2017-04-04 23:42:26,123 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.RMCriticalThreadUncaughtExceptionHandler:
> Critical thread FairSchedulerContinuousScheduling crashed!
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker.sortedNodeList(ClusterNodeTracker.java:306)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:884)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:316)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6448) Continuous scheduling thread crash while sorting nodes
[
https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-6448:
---
Attachment: YARN-6448.004.patch
> Continuous scheduling thread crash while sorting nodes
> --
>
> Key: YARN-6448
> URL: https://issues.apache.org/jira/browse/YARN-6448
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6448.001.patch, YARN-6448.002.patch,
> YARN-6448.003.patch, YARN-6448.004.patch
>
>
> YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
> breaks the order in comparison if nodes changes while sorting.
> {code}
> 2017-04-04 23:42:26,123 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.RMCriticalThreadUncaughtExceptionHandler:
> Critical thread FairSchedulerContinuousScheduling crashed!
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker.sortedNodeList(ClusterNodeTracker.java:306)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:884)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:316)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6448) Continuous scheduling thread crash while sorting nodes
[
https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957933#comment-15957933
]
Hadoop QA commented on YARN-6448:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 22s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 50 unchanged - 0 fixed = 51 total (was 50) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 54s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6448 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862171/YARN-6448.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 23e533af6577 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7d963c4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/15541/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/15541/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15541/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15541/console |
|
[jira] [Commented] (YARN-5797) Add metrics to the node manager for cleaning the PUBLIC and PRIVATE caches
[
https://issues.apache.org/jira/browse/YARN-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957929#comment-15957929
]
Chris Trezzo commented on YARN-5797:
Branch-2 test failures are unrelated. From my perspective the patch is good to
go for trunk and branch-2. Thanks!
> Add metrics to the node manager for cleaning the PUBLIC and PRIVATE caches
> --
>
> Key: YARN-5797
> URL: https://issues.apache.org/jira/browse/YARN-5797
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Attachments: YARN-5797-branch-2.001.patch, YARN-5797-trunk.002.patch,
> YARN-5797-trunk.003.patch, YARN-5797-trunk-v1.patch
>
>
> Add new metrics to the node manager around the local cache sizes and how much
> is being cleaned from them on a regular bases. For example, we can expose
> information contained in the {{LocalCacheCleanerStats}} class.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6448) Continuous scheduling thread crash while sorting nodes
[
https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-6448:
---
Attachment: YARN-6448.003.patch
Patch v3 change the unit test name.
> Continuous scheduling thread crash while sorting nodes
> --
>
> Key: YARN-6448
> URL: https://issues.apache.org/jira/browse/YARN-6448
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6448.001.patch, YARN-6448.002.patch,
> YARN-6448.003.patch
>
>
> YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
> breaks the order in comparison if nodes changes while sorting.
> {code}
> 2017-04-04 23:42:26,123 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.RMCriticalThreadUncaughtExceptionHandler:
> Critical thread FairSchedulerContinuousScheduling crashed!
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker.sortedNodeList(ClusterNodeTracker.java:306)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:884)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:316)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5797) Add metrics to the node manager for cleaning the PUBLIC and PRIVATE caches
[
https://issues.apache.org/jira/browse/YARN-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957915#comment-15957915
]
Hadoop QA commented on YARN-5797:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} branch-2 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 0 new + 303 unchanged - 6 fixed = 303 total (was 309)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 58s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 41s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_121 Failed junit tests |
hadoop.yarn.server.nodemanager.webapp.TestNMWebServer |
| JDK v1.7.0_121 Failed junit tests |
hadoop.yarn.server.nodemanager.webapp.TestNMWebServer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | YARN-5797 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862174/YARN-5797-branch-2.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 11f03bcec26c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64
[jira] [Updated] (YARN-6439) Fix ReservationSystem creation of default ReservationQueue
[ https://issues.apache.org/jira/browse/YARN-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-6439: --- Attachment: YARN-6439.v2.patch > Fix ReservationSystem creation of default ReservationQueue > -- > > Key: YARN-6439 > URL: https://issues.apache.org/jira/browse/YARN-6439 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6439.v0.patch, YARN-6439.v1.patch, > YARN-6439.v2.patch > > > The ReservationSystem does not create the default ReservationQueue till the > PlanFollower runs the first time. This is problematic for SLS runs, and might > show up in some other corner case. Inlining creation of the queue as part of > PlanQueue creation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6439) Fix ReservationSystem creation of default ReservationQueue
[ https://issues.apache.org/jira/browse/YARN-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957856#comment-15957856 ] Carlo Curino commented on YARN-6439: Fixed the checkstyle... The TestRMRestart.testFinishedAppRemovalAfterRMRestart(..) also fails in trunk per YARN-5632 and will be fixed as part of YARN-5548. > Fix ReservationSystem creation of default ReservationQueue > -- > > Key: YARN-6439 > URL: https://issues.apache.org/jira/browse/YARN-6439 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6439.v0.patch, YARN-6439.v1.patch > > > The ReservationSystem does not create the default ReservationQueue till the > PlanFollower runs the first time. This is problematic for SLS runs, and might > show up in some other corner case. Inlining creation of the queue as part of > PlanQueue creation. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6363) Extending SLS: Synthetic Load Generator
[
https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957850#comment-15957850
]
Hadoop QA commented on YARN-6363:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-6363 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6363 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862179/YARN-6363.v7.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15544/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Extending SLS: Synthetic Load Generator
> ---
>
> Key: YARN-6363
> URL: https://issues.apache.org/jira/browse/YARN-6363
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Attachments: YARN-6363 overview.pdf, YARN-6363.v0.patch,
> YARN-6363.v1.patch, YARN-6363.v2.patch, YARN-6363.v3.patch,
> YARN-6363.v4.patch, YARN-6363.v5.patch, YARN-6363.v6.patch, YARN-6363.v7.patch
>
>
> This JIRA tracks the introduction of a synthetic load generator in the SLS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6344) Rethinking OFF_SWITCH locality in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957849#comment-15957849
]
Jason Lowe commented on YARN-6344:
--
I'd prefer a configured rack locality delay of zero means no additional rack
delay, but I see that is semantically different than disabling it altogether.
Specifying a rack locality delay of zero means it will _not_ scale the node
locality delay based on the request/cluster sizes like it does today, whereas
setting it to -1 will. In that sense it's not purely an additional delay.
Given I don't know the complete backstory on the reasoning behind why it
behaves the way it does for node locality delay, I can see the desire to leave
the existing behavior unchanged when this new setting isn't configured.
Patch looks good to me.
> Rethinking OFF_SWITCH locality in CapacityScheduler
> ---
>
> Key: YARN-6344
> URL: https://issues.apache.org/jira/browse/YARN-6344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6344.001.patch, YARN-6344.002.patch,
> YARN-6344.003.patch, YARN-6344.004.patch
>
>
> When relaxing locality from node to rack, the {{node-locality-parameter}} is
> used: when scheduling opportunities for a scheduler key are more than the
> value of this parameter, we relax locality and try to assign the container to
> a node in the corresponding rack.
> On the other hand, when relaxing locality to off-switch (i.e., assign the
> container anywhere in the cluster), we are using a {{localityWaitFactor}},
> which is computed based on the number of outstanding requests for a specific
> scheduler key, which is divided by the size of the cluster.
> In case of applications that request containers in big batches (e.g.,
> traditional MR jobs), and for relatively small clusters, the
> localityWaitFactor does not affect relaxing locality much.
> However, in case of applications that request containers in small batches,
> this load factor takes a very small value, which leads to assigning
> off-switch containers too soon. This situation is even more pronounced in big
> clusters.
> For example, if an application requests only one container per request, the
> locality will be relaxed after a single missed scheduling opportunity.
> The purpose of this JIRA is to rethink the way we are relaxing locality for
> off-switch assignments.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6091) the AppMaster register failed when use Docker on LinuxContainer
[ https://issues.apache.org/jira/browse/YARN-6091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957837#comment-15957837 ] Eric Badger commented on YARN-6091: --- I think the issue is that the container-executor is calling pclose() before the command is finished. According to http://man7.org/linux/man-pages/man3/pclose.3p.html pclose() will close the stream before waiting for the command executed in popen() to terminate. So if the command that was executed in popen() is still executing when we call pclose() and is trying to write to stdout, the command will return a SIGPIPE (signal number 13). Since this depends on how long it takes to execute the command, it is non-deterministic between nodes and won't always return the SIGPIPE. I believe a fix would be to redirect stdout to /dev/null on the instances of popen() where we do not read the stdout of the command. I already tested locally that the stderr of the underlying command from popen() will be passed to the stderr of the process running popen(). This will end up having stderr print to stderr of the NM, but I think that is better than silently swallowing stderr altogether. I'll test out this approach locally and then put up a patch once I'm done. > the AppMaster register failed when use Docker on LinuxContainer > > > Key: YARN-6091 > URL: https://issues.apache.org/jira/browse/YARN-6091 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, yarn >Affects Versions: 2.8.1 > Environment: CentOS >Reporter: zhengchenyu >Priority: Critical > Original Estimate: 336h > Remaining Estimate: 336h > > In some servers, When I use Docker on LinuxContainer, I found the aciton that > AppMaster register to Resourcemanager failed. But didn't happen in other > servers. > I found the pclose (in container-executor.c) return different value in > different server, even though the process which is launched by popen is > running normally. Some server return 0, and others return 13. > Because yarn regard the application as failed application when pclose return > nonzero, and yarn will remove the AMRMToken, then the AppMaster register > failed because Resourcemanager have removed this applicaiton's token. > In container-executor.c, the judgement condition is whether the return code > is zero. But man the pclose, the document tells that "pclose return -1" > represent wrong. So I change the judgement condition, then slove this > problem. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5797) Add metrics to the node manager for cleaning the PUBLIC and PRIVATE caches
[
https://issues.apache.org/jira/browse/YARN-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957836#comment-15957836
]
Hadoop QA commented on YARN-5797:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 0 new + 284 unchanged - 6 fixed = 284 total (was 290)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
52s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 59s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5797 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862167/YARN-5797-trunk.003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 10996f70376d 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7d963c4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15540/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15540/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add metrics to the node manager for cleaning the PUBLIC and PRIVATE caches
> --
>
> Key: YARN-5797
> URL: https://issues.apache.org/jira/browse/YARN-5797
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Chris Trezzo
>
[jira] [Updated] (YARN-6448) Continuous scheduling thread crash while sorting nodes
[
https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-6448:
---
Summary: Continuous scheduling thread crash while sorting nodes (was: Add
back the lock in continuous scheduling while sorting nodes)
> Continuous scheduling thread crash while sorting nodes
> --
>
> Key: YARN-6448
> URL: https://issues.apache.org/jira/browse/YARN-6448
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6448.001.patch, YARN-6448.002.patch
>
>
> YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
> breaks the order in comparison if nodes changes while sorting.
> {code}
> 2017-04-04 23:42:26,123 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.RMCriticalThreadUncaughtExceptionHandler:
> Critical thread FairSchedulerContinuousScheduling crashed!
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker.sortedNodeList(ClusterNodeTracker.java:306)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:884)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:316)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6363) Extending SLS: Synthetic Load Generator
[ https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-6363: --- Attachment: YARN-6363.v7.patch (missed one method rename) > Extending SLS: Synthetic Load Generator > --- > > Key: YARN-6363 > URL: https://issues.apache.org/jira/browse/YARN-6363 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6363 overview.pdf, YARN-6363.v0.patch, > YARN-6363.v1.patch, YARN-6363.v2.patch, YARN-6363.v3.patch, > YARN-6363.v4.patch, YARN-6363.v5.patch, YARN-6363.v6.patch, YARN-6363.v7.patch > > > This JIRA tracks the introduction of a synthetic load generator in the SLS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6439) Fix ReservationSystem creation of default ReservationQueue
[
https://issues.apache.org/jira/browse/YARN-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957835#comment-15957835
]
Hadoop QA commented on YARN-6439:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 41 unchanged - 0 fixed = 42 total (was 41) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 11s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6439 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861978/YARN-6439.v1.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b595ce3d9722 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7d963c4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/15539/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/15539/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15539/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15539/console |
|
[jira] [Updated] (YARN-6448) Add back the lock in continuous scheduling while sorting nodes
[
https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-6448:
---
Description:
YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
breaks the order in comparison if nodes changes while sorting.
2017-04-04 23:42:26,123 FATAL
org.apache.hadoop.yarn.server.resourcemanager.RMCriticalThreadUncaughtExceptionHandler:
Critical thread FairSchedulerContinuousScheduling crashed!
{code}
java.lang.IllegalArgumentException: Comparison method violates its general
contract!
at java.util.TimSort.mergeHi(TimSort.java:899)
at java.util.TimSort.mergeAt(TimSort.java:516)
at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
at java.util.TimSort.sort(TimSort.java:254)
at java.util.Arrays.sort(Arrays.java:1512)
at java.util.ArrayList.sort(ArrayList.java:1454)
at java.util.Collections.sort(Collections.java:175)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker.sortedNodeList(ClusterNodeTracker.java:306)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:884)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:316)
{code}
was:YARN-4719 remove the lock in continuous scheduling while sorting nodes.
It breaks the order in comparison if nodes changes while sorting.
> Add back the lock in continuous scheduling while sorting nodes
> --
>
> Key: YARN-6448
> URL: https://issues.apache.org/jira/browse/YARN-6448
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6448.001.patch, YARN-6448.002.patch
>
>
> YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
> breaks the order in comparison if nodes changes while sorting.
> 2017-04-04 23:42:26,123 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.RMCriticalThreadUncaughtExceptionHandler:
> Critical thread FairSchedulerContinuousScheduling crashed!
> {code}
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker.sortedNodeList(ClusterNodeTracker.java:306)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:884)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:316)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6448) Add back the lock in continuous scheduling while sorting nodes
[
https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yufei Gu updated YARN-6448:
---
Description:
YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
breaks the order in comparison if nodes changes while sorting.
{code}
2017-04-04 23:42:26,123 FATAL
org.apache.hadoop.yarn.server.resourcemanager.RMCriticalThreadUncaughtExceptionHandler:
Critical thread FairSchedulerContinuousScheduling crashed!
java.lang.IllegalArgumentException: Comparison method violates its general
contract!
at java.util.TimSort.mergeHi(TimSort.java:899)
at java.util.TimSort.mergeAt(TimSort.java:516)
at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
at java.util.TimSort.sort(TimSort.java:254)
at java.util.Arrays.sort(Arrays.java:1512)
at java.util.ArrayList.sort(ArrayList.java:1454)
at java.util.Collections.sort(Collections.java:175)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker.sortedNodeList(ClusterNodeTracker.java:306)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:884)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:316)
{code}
was:
YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
breaks the order in comparison if nodes changes while sorting.
2017-04-04 23:42:26,123 FATAL
org.apache.hadoop.yarn.server.resourcemanager.RMCriticalThreadUncaughtExceptionHandler:
Critical thread FairSchedulerContinuousScheduling crashed!
{code}
java.lang.IllegalArgumentException: Comparison method violates its general
contract!
at java.util.TimSort.mergeHi(TimSort.java:899)
at java.util.TimSort.mergeAt(TimSort.java:516)
at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
at java.util.TimSort.sort(TimSort.java:254)
at java.util.Arrays.sort(Arrays.java:1512)
at java.util.ArrayList.sort(ArrayList.java:1454)
at java.util.Collections.sort(Collections.java:175)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker.sortedNodeList(ClusterNodeTracker.java:306)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:884)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:316)
{code}
> Add back the lock in continuous scheduling while sorting nodes
> --
>
> Key: YARN-6448
> URL: https://issues.apache.org/jira/browse/YARN-6448
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6448.001.patch, YARN-6448.002.patch
>
>
> YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
> breaks the order in comparison if nodes changes while sorting.
> {code}
> 2017-04-04 23:42:26,123 FATAL
> org.apache.hadoop.yarn.server.resourcemanager.RMCriticalThreadUncaughtExceptionHandler:
> Critical thread FairSchedulerContinuousScheduling crashed!
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.TimSort.mergeHi(TimSort.java:899)
> at java.util.TimSort.mergeAt(TimSort.java:516)
> at java.util.TimSort.mergeForceCollapse(TimSort.java:457)
> at java.util.TimSort.sort(TimSort.java:254)
> at java.util.Arrays.sort(Arrays.java:1512)
> at java.util.ArrayList.sort(ArrayList.java:1454)
> at java.util.Collections.sort(Collections.java:175)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.ClusterNodeTracker.sortedNodeList(ClusterNodeTracker.java:306)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.continuousSchedulingAttempt(FairScheduler.java:884)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler$ContinuousSchedulingThread.run(FairScheduler.java:316)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6363) Extending SLS: Synthetic Load Generator
[
https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957826#comment-15957826
]
Hadoop QA commented on YARN-6363:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-6363 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6363 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862177/YARN-6363.v6.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15543/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Extending SLS: Synthetic Load Generator
> ---
>
> Key: YARN-6363
> URL: https://issues.apache.org/jira/browse/YARN-6363
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Attachments: YARN-6363 overview.pdf, YARN-6363.v0.patch,
> YARN-6363.v1.patch, YARN-6363.v2.patch, YARN-6363.v3.patch,
> YARN-6363.v4.patch, YARN-6363.v5.patch, YARN-6363.v6.patch
>
>
> This JIRA tracks the introduction of a synthetic load generator in the SLS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6363) Extending SLS: Synthetic Load Generator
[ https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-6363: --- Attachment: YARN-6363.v6.patch Fixing dirty shutdown issue > Extending SLS: Synthetic Load Generator > --- > > Key: YARN-6363 > URL: https://issues.apache.org/jira/browse/YARN-6363 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6363 overview.pdf, YARN-6363.v0.patch, > YARN-6363.v1.patch, YARN-6363.v2.patch, YARN-6363.v3.patch, > YARN-6363.v4.patch, YARN-6363.v5.patch, YARN-6363.v6.patch > > > This JIRA tracks the introduction of a synthetic load generator in the SLS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6443) Allow for Priority order relaxing in favor of improved node/rack locality
[ https://issues.apache.org/jira/browse/YARN-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh reassigned YARN-6443: - Assignee: Hitesh Sharma (was: Arun Suresh) > Allow for Priority order relaxing in favor of improved node/rack locality > -- > > Key: YARN-6443 > URL: https://issues.apache.org/jira/browse/YARN-6443 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, fairscheduler >Reporter: Arun Suresh >Assignee: Hitesh Sharma > > Currently the Schedulers examine an applications pending Requests in Priority > order. This JIRA proposes to introduce a flag (either via the > ApplicationMasterService::registerApplication() or via some Scheduler > configuration) to favor an ordering that is baised to the node that is > currently heartbeating by relaxing the priority constraint. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6448) Add back the lock in continuous scheduling while sorting nodes
[
https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957810#comment-15957810
]
Hadoop QA commented on YARN-6448:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 39m
32s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6448 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862155/YARN-6448.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c7141e621a82 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7d963c4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15537/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15537/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add back the lock in continuous scheduling while sorting nodes
> --
>
> Key: YARN-6448
> URL: https://issues.apache.org/jira/browse/YARN-6448
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>
[jira] [Comment Edited] (YARN-5797) Add metrics to the node manager for cleaning the PUBLIC and PRIVATE caches
[
https://issues.apache.org/jira/browse/YARN-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957794#comment-15957794
]
Chris Trezzo edited comment on YARN-5797 at 4/5/17 9:43 PM:
Also attached is a v1 branch-2 patch. The only conflict was in
TestResourceLocalizationService.java and it was due to a difference in
formatting between the branches on one line.
was (Author: ctrezzo):
Also attached is a v1 branch-2 patch. The only conflict was in
TestResourceLocalization and it was due to a difference in formatting between
the branches on one line.
> Add metrics to the node manager for cleaning the PUBLIC and PRIVATE caches
> --
>
> Key: YARN-5797
> URL: https://issues.apache.org/jira/browse/YARN-5797
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Attachments: YARN-5797-branch-2.001.patch, YARN-5797-trunk.002.patch,
> YARN-5797-trunk.003.patch, YARN-5797-trunk-v1.patch
>
>
> Add new metrics to the node manager around the local cache sizes and how much
> is being cleaned from them on a regular bases. For example, we can expose
> information contained in the {{LocalCacheCleanerStats}} class.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6451) Create a monitor to check whether we maintain RM (scheduling) invariants
Carlo Curino created YARN-6451: -- Summary: Create a monitor to check whether we maintain RM (scheduling) invariants Key: YARN-6451 URL: https://issues.apache.org/jira/browse/YARN-6451 Project: Hadoop YARN Issue Type: Bug Reporter: Carlo Curino Assignee: Carlo Curino For SLS runs, as well as for live test clusters (and maybe prod), it would be useful to have a mechanism to continuously check whether core invariants of the RM/Scheduler are respected (e.g., no priority inversions, fairness mostly respected, certain latencies within expected range, etc..) -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5797) Add metrics to the node manager for cleaning the PUBLIC and PRIVATE caches
[
https://issues.apache.org/jira/browse/YARN-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated YARN-5797:
---
Attachment: YARN-5797-branch-2.001.patch
Also attached is a v1 branch-2 patch. The only conflict was in
TestResourceLocalization and it was due to a difference in formatting between
the branches on one line.
> Add metrics to the node manager for cleaning the PUBLIC and PRIVATE caches
> --
>
> Key: YARN-5797
> URL: https://issues.apache.org/jira/browse/YARN-5797
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Attachments: YARN-5797-branch-2.001.patch, YARN-5797-trunk.002.patch,
> YARN-5797-trunk.003.patch, YARN-5797-trunk-v1.patch
>
>
> Add new metrics to the node manager around the local cache sizes and how much
> is being cleaned from them on a regular bases. For example, we can expose
> information contained in the {{LocalCacheCleanerStats}} class.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6448) Add back the lock in continuous scheduling while sorting nodes
[ https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-6448: --- Attachment: YARN-6448.002.patch > Add back the lock in continuous scheduling while sorting nodes > -- > > Key: YARN-6448 > URL: https://issues.apache.org/jira/browse/YARN-6448 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6448.001.patch, YARN-6448.002.patch > > > YARN-4719 remove the lock in continuous scheduling while sorting nodes. It > breaks the order in comparison if nodes changes while sorting. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6448) Add back the lock in continuous scheduling while sorting nodes
[ https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957768#comment-15957768 ] Yufei Gu commented on YARN-6448: Patch v2 add a unit test. > Add back the lock in continuous scheduling while sorting nodes > -- > > Key: YARN-6448 > URL: https://issues.apache.org/jira/browse/YARN-6448 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6448.001.patch, YARN-6448.002.patch > > > YARN-4719 remove the lock in continuous scheduling while sorting nodes. It > breaks the order in comparison if nodes changes while sorting. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3760) FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close()
[ https://issues.apache.org/jira/browse/YARN-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957749#comment-15957749 ] Miklos Szegedi commented on YARN-3760: -- +1 (non-binding) > FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close() > > > Key: YARN-3760 > URL: https://issues.apache.org/jira/browse/YARN-3760 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.4.0 >Reporter: Daryn Sharp >Assignee: Miklos Szegedi >Priority: Critical > Attachments: YARN-3760.00.patch, YARN-3760.01.patch > > > The aggregated log file does not appear to be properly closed when writes > fail. This leaves a lease renewer active in the NM that spams the NN with > lease renewals. If the token is marked not to be cancelled, the renewals > appear to continue until the token expires. If the token is cancelled, the > periodic renew spam turns into a flood of failed connections until the lease > renewer gives up. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-3760) FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close()
[ https://issues.apache.org/jira/browse/YARN-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi reassigned YARN-3760: Assignee: Miklos Szegedi (was: Haibo Chen) > FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close() > > > Key: YARN-3760 > URL: https://issues.apache.org/jira/browse/YARN-3760 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.4.0 >Reporter: Daryn Sharp >Assignee: Miklos Szegedi >Priority: Critical > Attachments: YARN-3760.00.patch, YARN-3760.01.patch > > > The aggregated log file does not appear to be properly closed when writes > fail. This leaves a lease renewer active in the NM that spams the NN with > lease renewals. If the token is marked not to be cancelled, the renewals > appear to continue until the token expires. If the token is cancelled, the > periodic renew spam turns into a flood of failed connections until the lease > renewer gives up. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5797) Add metrics to the node manager for cleaning the PUBLIC and PRIVATE caches
[
https://issues.apache.org/jira/browse/YARN-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated YARN-5797:
---
Attachment: YARN-5797-trunk.003.patch
Attached is a v3 patch that is rebased on trunk.
> Add metrics to the node manager for cleaning the PUBLIC and PRIVATE caches
> --
>
> Key: YARN-5797
> URL: https://issues.apache.org/jira/browse/YARN-5797
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
> Attachments: YARN-5797-trunk.002.patch, YARN-5797-trunk.003.patch,
> YARN-5797-trunk-v1.patch
>
>
> Add new metrics to the node manager around the local cache sizes and how much
> is being cleaned from them on a regular bases. For example, we can expose
> information contained in the {{LocalCacheCleanerStats}} class.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6445) Improve YARN-3926 performance with respect to SLS
[
https://issues.apache.org/jira/browse/YARN-6445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957705#comment-15957705
]
Hadoop QA commented on YARN-6445:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
37s{color} | {color:green} YARN-3926 passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 5m
14s{color} | {color:red} hadoop-yarn in YARN-3926 failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
19s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
21s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} YARN-3926 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 5m
40s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 40s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 59s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 7 new + 50 unchanged - 42 fixed = 57 total (was 92) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
43s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
55s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 44m
29s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}111m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6445 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862046/YARN-6445-YARN-3926.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux db7c26be16a7 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-3926 / f3dc4ca |
| Default Java | 1.8.0_121 |
| compile |
https://builds.apache.org/job/PreCommit-YARN-Build/15533/artifact/patchprocess/branch-compile-hadoop-yarn-project_hadoop-yarn.txt
|
| findbugs | v3.0.0 |
| compile |
[jira] [Commented] (YARN-3760) FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close()
[
https://issues.apache.org/jira/browse/YARN-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957698#comment-15957698
]
Hadoop QA commented on YARN-3760:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
18s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-3760 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862152/YARN-3760.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 61bc087986d4 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7d963c4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15538/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15538/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close()
>
>
> Key: YARN-3760
> URL: https://issues.apache.org/jira/browse/YARN-3760
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.4.0
>Reporter: Daryn Sharp
>Assignee: Haibo Chen
>Priority: Critical
>
[jira] [Commented] (YARN-6449) Enable YARN to accept jobs with < 1 core allocations
[ https://issues.apache.org/jira/browse/YARN-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957687#comment-15957687 ] Karthik Kambatla commented on YARN-6449: Which scheduler are you using? FairScheduler already supports this. > Enable YARN to accept jobs with < 1 core allocations > > > Key: YARN-6449 > URL: https://issues.apache.org/jira/browse/YARN-6449 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Daniel Tomes > Labels: features, performance > > Product Enhancement Request > In Spark/HIVE/etc. I often need to complete work for which an entire core is > overkill such as managing a JDBC connection or doing a simple map/transform; > however, when I do this on large datasets, 1 core X 500 partitions/mappers > winds up with quite the cluster level footprint even though most of those > processor cycles are idle. > I propose that we enable YARN to allow a user to submit jobs that "allocate < > 1 core". Under the covers, the JVM will still receive one core but YARN/ZK > could keep track of the fractions of cores being used and allow other jobs to > consume the same core twice provided that both jobs were submitted with <= .5 > cores. Now, YARN can more effectively utilize multi-threading and decrease > CPU idle for the power users. > Obviously this can ultimately result in very bad outcomes, but if we also > enable security controls then customers can configure such that only > admins/gates can submit with < 1 full core and ultimately resulting in a > cluster that can do more. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6363) Extending SLS: Synthetic Load Generator
[ https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957655#comment-15957655 ] Carlo Curino commented on YARN-6363: [~wangda] I believe I addressed your concerns, and added the scheduler as one of the parameters so we now run for both CS and FS all three formats SLS, RUMEN, and SYNTH. In the process I fixed the shutdown hooks (it didn't work properly, and since we inherit from the main schedulers, I think it is enough to implement serviceStop() and call super.serviceStop() please check if this logic makes sense to you). I am not very familiar with the SLS input, so I used the example form the documentation, feel free to change it if there is any better sls inputs to use for this test. > Extending SLS: Synthetic Load Generator > --- > > Key: YARN-6363 > URL: https://issues.apache.org/jira/browse/YARN-6363 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6363 overview.pdf, YARN-6363.v0.patch, > YARN-6363.v1.patch, YARN-6363.v2.patch, YARN-6363.v3.patch, > YARN-6363.v4.patch, YARN-6363.v5.patch > > > This JIRA tracks the introduction of a synthetic load generator in the SLS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6363) Extending SLS: Synthetic Load Generator
[ https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957657#comment-15957657 ] Carlo Curino commented on YARN-6363: QA rightfully gripes because we depend on YARN-6439. > Extending SLS: Synthetic Load Generator > --- > > Key: YARN-6363 > URL: https://issues.apache.org/jira/browse/YARN-6363 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6363 overview.pdf, YARN-6363.v0.patch, > YARN-6363.v1.patch, YARN-6363.v2.patch, YARN-6363.v3.patch, > YARN-6363.v4.patch, YARN-6363.v5.patch > > > This JIRA tracks the introduction of a synthetic load generator in the SLS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6344) Rethinking OFF_SWITCH locality in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957652#comment-15957652
]
Konstantinos Karanasos commented on YARN-6344:
--
Thanks for checking the patch, [~leftnoteasy].
{{getActualRackLocalityDelay}} is only called by the {{canAssign}} method, and
within there I am falling back to the old behavior in case rack locality is set
to -1, so what you are observing will not happen.
I was also thinking we can push this logic inside the
{{getActualRackLocalityDelay}}, but didn't do so to avoid evaluating the
{{getActualRackLocalityDelay}} in case rack locality is set to -1 (existing
behavior).
Let me know what you think.
> Rethinking OFF_SWITCH locality in CapacityScheduler
> ---
>
> Key: YARN-6344
> URL: https://issues.apache.org/jira/browse/YARN-6344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6344.001.patch, YARN-6344.002.patch,
> YARN-6344.003.patch, YARN-6344.004.patch
>
>
> When relaxing locality from node to rack, the {{node-locality-parameter}} is
> used: when scheduling opportunities for a scheduler key are more than the
> value of this parameter, we relax locality and try to assign the container to
> a node in the corresponding rack.
> On the other hand, when relaxing locality to off-switch (i.e., assign the
> container anywhere in the cluster), we are using a {{localityWaitFactor}},
> which is computed based on the number of outstanding requests for a specific
> scheduler key, which is divided by the size of the cluster.
> In case of applications that request containers in big batches (e.g.,
> traditional MR jobs), and for relatively small clusters, the
> localityWaitFactor does not affect relaxing locality much.
> However, in case of applications that request containers in small batches,
> this load factor takes a very small value, which leads to assigning
> off-switch containers too soon. This situation is even more pronounced in big
> clusters.
> For example, if an application requests only one container per request, the
> locality will be relaxed after a single missed scheduling opportunity.
> The purpose of this JIRA is to rethink the way we are relaxing locality for
> off-switch assignments.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6435) [ATSv2] Can't retrieve more than 1000 versions of metrics in time series
[
https://issues.apache.org/jira/browse/YARN-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated YARN-6435:
---
Issue Type: Sub-task (was: Bug)
Parent: YARN-5355
> [ATSv2] Can't retrieve more than 1000 versions of metrics in time series
>
>
> Key: YARN-6435
> URL: https://issues.apache.org/jira/browse/YARN-6435
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Rohith Sharma K S
>Assignee: Rohith Sharma K S
>Priority: Critical
> Attachments: YARN-6435.0001.patch
>
>
> It is observed that, even though *metricslimit* is set to 1500, maximum
> number of metrics values retrieved is 1000.
> This is due to, while creating EntityTable, metrics column family max version
> is specified as 1000 which is hardcoded in
> {{EntityTable#DEFAULT_METRICS_MAX_VERSIONS}}. So, HBase will return max
> version with following {{MIN(cf max version , user provided max version)}}.
> This behavior is contradicting the documentation which claims that
> {code}
> metricslimit - If specified, defines the number of metrics to return.
> Considered only if fields contains METRICS/ALL or metricstoretrieve is
> specified. Ignored otherwise. The maximum possible value for metricslimit can
> be maximum value of Integer. If it is not specified or has a value less than
> 1, and metrics have to be retrieved, then metricslimit will be considered as
> 1 i.e. latest single value of metric(s) will be returned.
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6363) Extending SLS: Synthetic Load Generator
[
https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957638#comment-15957638
]
Hadoop QA commented on YARN-6363:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red} YARN-6363 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6363 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862153/YARN-6363.v5.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15535/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Extending SLS: Synthetic Load Generator
> ---
>
> Key: YARN-6363
> URL: https://issues.apache.org/jira/browse/YARN-6363
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Attachments: YARN-6363 overview.pdf, YARN-6363.v0.patch,
> YARN-6363.v1.patch, YARN-6363.v2.patch, YARN-6363.v3.patch,
> YARN-6363.v4.patch, YARN-6363.v5.patch
>
>
> This JIRA tracks the introduction of a synthetic load generator in the SLS.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6437) TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently
[ https://issues.apache.org/jira/browse/YARN-6437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957640#comment-15957640 ] Hudson commented on YARN-6437: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11532 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11532/]) YARN-6437. TestSignalContainer#testSignalRequestDeliveryToNM fails (varunsaxena: rev 7d963c477ab997e0daad760dd3c568d7010fe511) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSignalContainer.java > TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently > -- > > Key: YARN-6437 > URL: https://issues.apache.org/jira/browse/YARN-6437 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Affects Versions: 2.8.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3 > > Attachments: YARN-6437.001.patch > > > testSignalRequestDeliveryToNM can fail if the containers are returned across > multiple scheduling heartbeats. The loop waiting for all the containers > should be accumulating the containers but instead is smashing the same list > of containers with whatever the allocate call returns. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6448) Add back the lock in continuous scheduling while sorting nodes
[ https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yufei Gu updated YARN-6448: --- Attachment: YARN-6448.001.patch > Add back the lock in continuous scheduling while sorting nodes > -- > > Key: YARN-6448 > URL: https://issues.apache.org/jira/browse/YARN-6448 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.9.0, 3.0.0-alpha2 >Reporter: Yufei Gu >Assignee: Yufei Gu > Attachments: YARN-6448.001.patch > > > YARN-4719 remove the lock in continuous scheduling while sorting nodes. It > breaks the order in comparison if nodes changes while sorting. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6450) TestContainerManagerWithLCE requires override for each new test added to ContainerManagerTest
[
https://issues.apache.org/jira/browse/YARN-6450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated YARN-6450:
-
Attachment: YARN-6450.001.patch
Using {{Assume.assumeTrue(shouldRunTest())}} in the existing setup function
will automatically skip the tests if the LCE hasn't been configured. Then we
don't need to revisit this code every time a new unit test is added to
ContainerManagerTest.
Attaching a patch that implements this approach.
> TestContainerManagerWithLCE requires override for each new test added to
> ContainerManagerTest
> -
>
> Key: YARN-6450
> URL: https://issues.apache.org/jira/browse/YARN-6450
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: YARN-6450.001.patch
>
>
> Every test in TestContainerManagerWithLCE looks like this:
> {code}
> @Override
> public void testSomething() throws Exception {
> // Don't run the test if the binary is not available.
> if (!shouldRunTest()) {
> LOG.info("LCE binary path is not passed. Not running the test");
> return;
> }
> LOG.info("Running something");
> super.testSomething();
> }
> {code}
> If a new test is added to ContainerManagerTest then by default
> ContainerManagerTestWithLCE will fail when the LCE has not been configured.
> This is an unnecessary maintenance burden.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6450) TestContainerManagerWithLCE requires override for each new test added to ContainerManagerTest
Jason Lowe created YARN-6450:
Summary: TestContainerManagerWithLCE requires override for each
new test added to ContainerManagerTest
Key: YARN-6450
URL: https://issues.apache.org/jira/browse/YARN-6450
Project: Hadoop YARN
Issue Type: Bug
Components: test
Reporter: Jason Lowe
Assignee: Jason Lowe
Every test in TestContainerManagerWithLCE looks like this:
{code}
@Override
public void testSomething() throws Exception {
// Don't run the test if the binary is not available.
if (!shouldRunTest()) {
LOG.info("LCE binary path is not passed. Not running the test");
return;
}
LOG.info("Running something");
super.testSomething();
}
{code}
If a new test is added to ContainerManagerTest then by default
ContainerManagerTestWithLCE will fail when the LCE has not been configured.
This is an unnecessary maintenance burden.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6363) Extending SLS: Synthetic Load Generator
[ https://issues.apache.org/jira/browse/YARN-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-6363: --- Attachment: YARN-6363.v5.patch > Extending SLS: Synthetic Load Generator > --- > > Key: YARN-6363 > URL: https://issues.apache.org/jira/browse/YARN-6363 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-6363 overview.pdf, YARN-6363.v0.patch, > YARN-6363.v1.patch, YARN-6363.v2.patch, YARN-6363.v3.patch, > YARN-6363.v4.patch, YARN-6363.v5.patch > > > This JIRA tracks the introduction of a synthetic load generator in the SLS. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3760) FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close()
[ https://issues.apache.org/jira/browse/YARN-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-3760: - Attachment: YARN-3760.01.patch Updated the patch to address Miklos' comment per offline discussion. > FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close() > > > Key: YARN-3760 > URL: https://issues.apache.org/jira/browse/YARN-3760 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.4.0 >Reporter: Daryn Sharp >Assignee: Haibo Chen >Priority: Critical > Attachments: YARN-3760.00.patch, YARN-3760.01.patch > > > The aggregated log file does not appear to be properly closed when writes > fail. This leaves a lease renewer active in the NM that spams the NN with > lease renewals. If the token is marked not to be cancelled, the renewals > appear to continue until the token expires. If the token is cancelled, the > periodic renew spam turns into a flood of failed connections until the lease > renewer gives up. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6448) Add back the lock in continuous scheduling while sorting nodes
[
https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957606#comment-15957606
]
Yufei Gu commented on YARN-6448:
According to my tests, any changes to {{unallocatedResource}} of a node may
break the method {{ClusterNodeTracker#sortedNodeList}}, so it's not enough to
add a write lock in it since {{unallocatedResource}} could be changed outside
of class {{ClusterNodeTracker}}. We should mark
{{ClusterNodeTracker#sortedNodeList}} not thread safe, and solve the issue in
continuous scheduling by adding a lock out side of {{List
nodeIdList =  nodeTracker.sortedNodeList(nodeAvailableResourceComparator);}}
> Add back the lock in continuous scheduling while sorting nodes
> --
>
> Key: YARN-6448
> URL: https://issues.apache.org/jira/browse/YARN-6448
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
> breaks the order in comparison if nodes changes while sorting.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6195) Export UsedCapacity and AbsoluteUsedCapacity to JMX
[ https://issues.apache.org/jira/browse/YARN-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benson Qiu updated YARN-6195: - Attachment: YARN-6195.005.patch > Export UsedCapacity and AbsoluteUsedCapacity to JMX > --- > > Key: YARN-6195 > URL: https://issues.apache.org/jira/browse/YARN-6195 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler, metrics, yarn >Affects Versions: 3.0.0-alpha3 >Reporter: Benson Qiu >Assignee: Benson Qiu > Attachments: YARN-6195.001.patch, YARN-6195.002.patch, > YARN-6195.003.patch, YARN-6195.004.patch, YARN-6195.005.patch > > > `usedCapacity` and `absoluteUsedCapacity` are currently not available as JMX. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5602) Utils for Federation State and Policy Store
[ https://issues.apache.org/jira/browse/YARN-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957540#comment-15957540 ] Subru Krishnan commented on YARN-5602: -- Thanks [~giovanni.fumarola] for addressing my comments. +1 on the latest version (v6) of the patch, will be committing it shortly. > Utils for Federation State and Policy Store > --- > > Key: YARN-5602 > URL: https://issues.apache.org/jira/browse/YARN-5602 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Labels: oct16-medium > Attachments: YARN-5602-YARN-2915.v1.patch, > YARN-5602-YARN-2915.v2.patch, YARN-5602-YARN-2915.v3.patch, > YARN-5602-YARN-2915.v4.patch, YARN-5602-YARN-2915.v5.patch, > YARN-5602-YARN-2915.v6.patch > > > This JIRA tracks the creation of utils for Federation State and Policy Store > such as Error Codes, Exceptions... -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6381) FSAppAttempt has several variables that should be final
[
https://issues.apache.org/jira/browse/YARN-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957539#comment-15957539
]
Hadoop QA commented on YARN-6381:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 42m
12s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6381 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861796/YARN-6381.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8e3ccc690e15 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 87e2ef8 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15532/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15532/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> FSAppAttempt has several variables that should be final
> ---
>
> Key: YARN-6381
> URL: https://issues.apache.org/jira/browse/YARN-6381
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha2
>Reporter:
[jira] [Commented] (YARN-6195) Export UsedCapacity and AbsoluteUsedCapacity to JMX
[ https://issues.apache.org/jira/browse/YARN-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957532#comment-15957532 ] Benson Qiu commented on YARN-6195: -- I will fix the findbugs issue from HadoopQA. > Export UsedCapacity and AbsoluteUsedCapacity to JMX > --- > > Key: YARN-6195 > URL: https://issues.apache.org/jira/browse/YARN-6195 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler, metrics, yarn >Affects Versions: 3.0.0-alpha3 >Reporter: Benson Qiu >Assignee: Benson Qiu > Attachments: YARN-6195.001.patch, YARN-6195.002.patch, > YARN-6195.003.patch, YARN-6195.004.patch > > > `usedCapacity` and `absoluteUsedCapacity` are currently not available as JMX. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6449) Enable YARN to accept jobs with < 1 core allocations
[ https://issues.apache.org/jira/browse/YARN-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957495#comment-15957495 ] Miklos Szegedi commented on YARN-6449: -- vcore is a setting per container. Another option is to use opportunistic containers in newer versions. > Enable YARN to accept jobs with < 1 core allocations > > > Key: YARN-6449 > URL: https://issues.apache.org/jira/browse/YARN-6449 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Daniel Tomes > Labels: features, performance > > Product Enhancement Request > In Spark/HIVE/etc. I often need to complete work for which an entire core is > overkill such as managing a JDBC connection or doing a simple map/transform; > however, when I do this on large datasets, 1 core X 500 partitions/mappers > winds up with quite the cluster level footprint even though most of those > processor cycles are idle. > I propose that we enable YARN to allow a user to submit jobs that "allocate < > 1 core". Under the covers, the JVM will still receive one core but YARN/ZK > could keep track of the fractions of cores being used and allow other jobs to > consume the same core twice provided that both jobs were submitted with <= .5 > cores. Now, YARN can more effectively utilize multi-threading and decrease > CPU idle for the power users. > Obviously this can ultimately result in very bad outcomes, but if we also > enable security controls then customers can configure such that only > admins/gates can submit with < 1 full core and ultimately resulting in a > cluster that can do more. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6438) Code can be improved in ContainersMonitorImpl.java
[ https://issues.apache.org/jira/browse/YARN-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957488#comment-15957488 ] Miklos Szegedi commented on YARN-6438: -- Thank you, [~yufeigu]. > Code can be improved in ContainersMonitorImpl.java > -- > > Key: YARN-6438 > URL: https://issues.apache.org/jira/browse/YARN-6438 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Labels: newbie > Attachments: YARN-6438.000.patch, YARN-6438.001.patch, > YARN-6438.002.patch > > > I noticed two code blocks that can be improved in ContainersMonitorImpl.java. > cpuUsagePercentPerCoreByAllContainers and cpuUsageTotalCoresByAllContainers > track the same value and CHANGE_MONITORING_CONTAINER_RESOURCE is checked > twice along with two calls to changeContainerResource. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6449) Enable YARN to accept jobs with < 1 core allocations
[ https://issues.apache.org/jira/browse/YARN-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957484#comment-15957484 ] Daniel Tomes commented on YARN-6449: [~miklos.szeg...@cloudera.com] That would work in theory but that would be a global change, right? Most users are not experienced enough to do this correctly and should not be allowed or we would surely run into issues. If we were to allow a small group of admins/schedulers in the enterprise tweak this for certain jobs we could fit a lot more into our clusters. This request comes from numerous clients starting to use SPARK on YARN for ETL. 90% of the simple ETL I see does not require anywhere near a full processor to manage the ETL pipeline. Thanks for the quick response. > Enable YARN to accept jobs with < 1 core allocations > > > Key: YARN-6449 > URL: https://issues.apache.org/jira/browse/YARN-6449 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Daniel Tomes > Labels: features, performance > > Product Enhancement Request > In Spark/HIVE/etc. I often need to complete work for which an entire core is > overkill such as managing a JDBC connection or doing a simple map/transform; > however, when I do this on large datasets, 1 core X 500 partitions/mappers > winds up with quite the cluster level footprint even though most of those > processor cycles are idle. > I propose that we enable YARN to allow a user to submit jobs that "allocate < > 1 core". Under the covers, the JVM will still receive one core but YARN/ZK > could keep track of the fractions of cores being used and allow other jobs to > consume the same core twice provided that both jobs were submitted with <= .5 > cores. Now, YARN can more effectively utilize multi-threading and decrease > CPU idle for the power users. > Obviously this can ultimately result in very bad outcomes, but if we also > enable security controls then customers can configure such that only > admins/gates can submit with < 1 full core and ultimately resulting in a > cluster that can do more. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6449) Enable YARN to accept jobs with < 1 core allocations
[ https://issues.apache.org/jira/browse/YARN-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957478#comment-15957478 ] Miklos Szegedi commented on YARN-6449: -- [~Jrtorres42], thank you for the suggestion. Why do not you split your nodes into multiple virtual cores per core with yarn.nodemanager.resource.cpu-vcores and assign based on a vcore (=fraction of a core)? > Enable YARN to accept jobs with < 1 core allocations > > > Key: YARN-6449 > URL: https://issues.apache.org/jira/browse/YARN-6449 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Daniel Tomes > Labels: features, performance > > Product Enhancement Request > In Spark/HIVE/etc. I often need to complete work for which an entire core is > overkill such as managing a JDBC connection or doing a simple map/transform; > however, when I do this on large datasets, 1 core X 500 partitions/mappers > winds up with quite the cluster level footprint even though most of those > processor cycles are idle. > I propose that we enable YARN to allow a user to submit jobs that "allocate < > 1 core". Under the covers, the JVM will still receive one core but YARN/ZK > could keep track of the fractions of cores being used and allow other jobs to > consume the same core twice provided that both jobs were submitted with <= .5 > cores. Now, YARN can more effectively utilize multi-threading and decrease > CPU idle for the power users. > Obviously this can ultimately result in very bad outcomes, but if we also > enable security controls then customers can configure such that only > admins/gates can submit with < 1 full core and ultimately resulting in a > cluster that can do more. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6449) Enable YARN to accept jobs with < 1 core allocations
[ https://issues.apache.org/jira/browse/YARN-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957479#comment-15957479 ] Daniel Tomes commented on YARN-6449: I realize this topic has been discussed in detail but thought it might warrant another thought as we've grown up a lot since the past issue was filed. > Enable YARN to accept jobs with < 1 core allocations > > > Key: YARN-6449 > URL: https://issues.apache.org/jira/browse/YARN-6449 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Daniel Tomes > Labels: features, performance > > Product Enhancement Request > In Spark/HIVE/etc. I often need to complete work for which an entire core is > overkill such as managing a JDBC connection or doing a simple map/transform; > however, when I do this on large datasets, 1 core X 500 partitions/mappers > winds up with quite the cluster level footprint even though most of those > processor cycles are idle. > I propose that we enable YARN to allow a user to submit jobs that "allocate < > 1 core". Under the covers, the JVM will still receive one core but YARN/ZK > could keep track of the fractions of cores being used and allow other jobs to > consume the same core twice provided that both jobs were submitted with <= .5 > cores. Now, YARN can more effectively utilize multi-threading and decrease > CPU idle for the power users. > Obviously this can ultimately result in very bad outcomes, but if we also > enable security controls then customers can configure such that only > admins/gates can submit with < 1 full core and ultimately resulting in a > cluster that can do more. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Add support for toggling the removal of completed and failed docker containers
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957458#comment-15957458 ] Vinod Kumar Vavilapalli commented on YARN-5366: --- bq. While it is possible to do a docker kill --signal SIGQUIT this is limited in it usefulness and may result in unexpected behavior. The signal is always sent to PID 1 in the container. Depending on the image or app type, this may not be the process we want to catch that signal. Alternatively, users can specify the STOPSIGNAL in the Dockerfile and the user likely has a better understanding of the implications for that application/image type. Thoughts on how this should be handled? Should we just ignore the Signal.QUIT? This is kind of the same expectation in the non-docker case where there is a process-tree and the signal is simply just delivered without the NM interpreting it. I think we should just pass any signal that is not 0, KILL and TERM along to the container via "docker kill -s ". bq. IMO, moving more of this logic into c-e complicates matters and doesn't follow what we've done so far. Nearly all existing DockerCommands execute via c-e as a single Docker CLI command. It's a performance optimization. We can come back to this later if it indeed becomes a bottleneck. > Add support for toggling the removal of completed and failed docker containers > -- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Labels: oct16-medium > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch > > > Currently, completed and failed docker containers are removed by > container-executor. Add a job level environment variable to > DockerLinuxContainerRuntime to allow the user to toggle whether they want the > container deleted or not and remove the logic from container-executor. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6449) Enable YARN to accept jobs with < 1 core allocations
[ https://issues.apache.org/jira/browse/YARN-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6449: --- Issue Type: Improvement (was: Bug) > Enable YARN to accept jobs with < 1 core allocations > > > Key: YARN-6449 > URL: https://issues.apache.org/jira/browse/YARN-6449 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Daniel Tomes > Labels: features, performance > > Product Enhancement Request > In Spark/HIVE/etc. I often need to complete work for which an entire core is > overkill such as managing a JDBC connection or doing a simple map/transform; > however, when I do this on large datasets, 1 core X 500 partitions/mappers > winds up with quite the cluster level footprint even though most of those > processor cycles are idle. > I propose that we enable YARN to allow a user to submit jobs that "allocate < > 1 core". Under the covers, the JVM will still receive one core but YARN/ZK > could keep track of the fractions of cores being used and allow other jobs to > consume the same core twice provided that both jobs were submitted with <= .5 > cores. Now, YARN can more effectively utilize multi-threading and decrease > CPU idle for the power users. > Obviously this can ultimately result in very bad outcomes, but if we also > enable security controls then customers can configure such that only > admins/gates can submit with < 1 full core and ultimately resulting in a > cluster that can do more. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6449) Enable YARN to accept jobs with < 1 core allocations
Daniel Tomes created YARN-6449: -- Summary: Enable YARN to accept jobs with < 1 core allocations Key: YARN-6449 URL: https://issues.apache.org/jira/browse/YARN-6449 Project: Hadoop YARN Issue Type: Bug Components: yarn Reporter: Daniel Tomes Product Enhancement Request In Spark/HIVE/etc. I often need to complete work for which an entire core is overkill such as managing a JDBC connection or doing a simple map/transform; however, when I do this on large datasets, 1 core X 500 partitions/mappers winds up with quite the cluster level footprint even though most of those processor cycles are idle. I propose that we enable YARN to allow a user to submit jobs that "allocate < 1 core". Under the covers, the JVM will still receive one core but YARN/ZK could keep track of the fractions of cores being used and allow other jobs to consume the same core twice provided that both jobs were submitted with <= .5 cores. Now, YARN can more effectively utilize multi-threading and decrease CPU idle for the power users. Obviously this can ultimately result in very bad outcomes, but if we also enable security controls then customers can configure such that only admins/gates can submit with < 1 full core and ultimately resulting in a cluster that can do more. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6448) Add back the lock in continuous scheduling while sorting nodes
[
https://issues.apache.org/jira/browse/YARN-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957436#comment-15957436
]
Yufei Gu commented on YARN-6448:
Seems better to add a write lock in {{ClusterNodeTracker#sortedNodeList}}.
> Add back the lock in continuous scheduling while sorting nodes
> --
>
> Key: YARN-6448
> URL: https://issues.apache.org/jira/browse/YARN-6448
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> YARN-4719 remove the lock in continuous scheduling while sorting nodes. It
> breaks the order in comparison if nodes changes while sorting.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6406) Remove SchedulerRequestKeys when no more pending ResourceRequest
[ https://issues.apache.org/jira/browse/YARN-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957432#comment-15957432 ] Hudson commented on YARN-6406: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11531 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11531/]) YARN-6406. Remove SchedulerRequestKeys when no more pending (wangda: rev 87e2ef8c985bb72a916477e8783359f2859f7890) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/LocalitySchedulingPlacementSet.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java > Remove SchedulerRequestKeys when no more pending ResourceRequest > > > Key: YARN-6406 > URL: https://issues.apache.org/jira/browse/YARN-6406 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha2 >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6406.001.patch, YARN-6406.002.patch > > > YARN-5540 introduced some optimizations to remove satisfied SchedulerKeys > from the AppScheduleingInfo. It looks like after YARN-6040, > ScedulerRequestKeys are removed only if the Application sends a 0 > numContainers requests. While earlier, the outstanding schedulerKeys were > also remove as soon as a container is allocated as well. > An additional optimization we were hoping to include is to remove the > ResourceRequests itself once the numContainers == 0, since we see in our > clusters that the RM heap space consumption increases drastically due to a > large number of ResourceRequests with 0 num containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5602) Utils for Federation State and Policy Store
[ https://issues.apache.org/jira/browse/YARN-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957431#comment-15957431 ] Giovanni Matteo Fumarola commented on YARN-5602: Thanks [~subru] for reviewing the patch. About your comments: 1) I added a comment on why Capability and HeartBeat are not used for comparison. 2) I removed the HikariCP dependency. I will add it when we will add the FederationStateStore SQL implementation. 3) I edited the FederationStateStoreErrorCode to match with the APIs. 4) I added the missing javadocs. > Utils for Federation State and Policy Store > --- > > Key: YARN-5602 > URL: https://issues.apache.org/jira/browse/YARN-5602 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Labels: oct16-medium > Attachments: YARN-5602-YARN-2915.v1.patch, > YARN-5602-YARN-2915.v2.patch, YARN-5602-YARN-2915.v3.patch, > YARN-5602-YARN-2915.v4.patch, YARN-5602-YARN-2915.v5.patch, > YARN-5602-YARN-2915.v6.patch > > > This JIRA tracks the creation of utils for Federation State and Policy Store > such as Error Codes, Exceptions... -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6195) Export UsedCapacity and AbsoluteUsedCapacity to JMX
[
https://issues.apache.org/jira/browse/YARN-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957423#comment-15957423
]
Hadoop QA commented on YARN-6195:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
11s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
2m 11s{color} | {color:orange} root: The patch generated 7 new + 267 unchanged
- 4 fixed = 274 total (was 271) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
29s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
27s{color} | {color:green} hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 41m 42s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 0s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| | Dead store to nodeLabels in
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.updateUsedCapacity(ResourceCalculator,
Resource, String, AbstractCSQueue) At
CSQueueUtils.java:org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.updateUsedCapacity(ResourceCalculator,
Resource, String, AbstractCSQueue) At CSQueueUtils.java:[line 190] |
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6195 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862102/YARN-6195.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux d29b7f7bcb8d 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64
[jira] [Commented] (YARN-6344) Rethinking OFF_SWITCH locality in CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957412#comment-15957412
]
Wangda Tan commented on YARN-6344:
--
Thanks [~kkaranasos],
{code}
281 private int getActualRackLocalityDelay() {
282 return Math.min(rmContext.getScheduler().getNumClusterNodes(),
283 application.getCSLeafQueue().getNodeLocalityDelay()
284 +
application.getCSLeafQueue().getRackLocalityAdditionalDelay());
285 }
{code}
Will this cause the real_rack_locality_delay = node_locality_delay - 1 when
getRackLocalityAdditionalDelay is not set?
[~jlowe]: I think it's better to get review from you as well.
Thanks,
> Rethinking OFF_SWITCH locality in CapacityScheduler
> ---
>
> Key: YARN-6344
> URL: https://issues.apache.org/jira/browse/YARN-6344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Attachments: YARN-6344.001.patch, YARN-6344.002.patch,
> YARN-6344.003.patch, YARN-6344.004.patch
>
>
> When relaxing locality from node to rack, the {{node-locality-parameter}} is
> used: when scheduling opportunities for a scheduler key are more than the
> value of this parameter, we relax locality and try to assign the container to
> a node in the corresponding rack.
> On the other hand, when relaxing locality to off-switch (i.e., assign the
> container anywhere in the cluster), we are using a {{localityWaitFactor}},
> which is computed based on the number of outstanding requests for a specific
> scheduler key, which is divided by the size of the cluster.
> In case of applications that request containers in big batches (e.g.,
> traditional MR jobs), and for relatively small clusters, the
> localityWaitFactor does not affect relaxing locality much.
> However, in case of applications that request containers in small batches,
> this load factor takes a very small value, which leads to assigning
> off-switch containers too soon. This situation is even more pronounced in big
> clusters.
> For example, if an application requests only one container per request, the
> locality will be relaxed after a single missed scheduling opportunity.
> The purpose of this JIRA is to rethink the way we are relaxing locality for
> off-switch assignments.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6438) Code can be improved in ContainersMonitorImpl.java
[ https://issues.apache.org/jira/browse/YARN-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957408#comment-15957408 ] Yufei Gu commented on YARN-6438: +1 (non-binding) > Code can be improved in ContainersMonitorImpl.java > -- > > Key: YARN-6438 > URL: https://issues.apache.org/jira/browse/YARN-6438 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Labels: newbie > Attachments: YARN-6438.000.patch, YARN-6438.001.patch, > YARN-6438.002.patch > > > I noticed two code blocks that can be improved in ContainersMonitorImpl.java. > cpuUsagePercentPerCoreByAllContainers and cpuUsageTotalCoresByAllContainers > track the same value and CHANGE_MONITORING_CONTAINER_RESOURCE is checked > twice along with two calls to changeContainerResource. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6448) Add back the lock in continuous scheduling while sorting nodes
Yufei Gu created YARN-6448: -- Summary: Add back the lock in continuous scheduling while sorting nodes Key: YARN-6448 URL: https://issues.apache.org/jira/browse/YARN-6448 Project: Hadoop YARN Issue Type: Bug Affects Versions: 3.0.0-alpha2, 2.9.0 Reporter: Yufei Gu Assignee: Yufei Gu YARN-4719 remove the lock in continuous scheduling while sorting nodes. It breaks the order in comparison if nodes changes while sorting. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5602) Utils for Federation State and Policy Store
[
https://issues.apache.org/jira/browse/YARN-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957391#comment-15957391
]
Hadoop QA commented on YARN-5602:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
36s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
52s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} YARN-2915 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
14s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 23m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5602 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862121/YARN-5602-YARN-2915.v6.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 2e2a42ab092b 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | YARN-2915 / 6e34518 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15531/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15531/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Utils for Federation State and Policy Store
> ---
>
> Key: YARN-5602
> URL: https://issues.apache.org/jira/browse/YARN-5602
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Giovanni Matteo Fumarola
>Assignee: Giovanni Matteo Fumarola
> Labels: oct16-medium
> Attachments: YARN-5602-YARN-2915.v1.patch,
>
[jira] [Commented] (YARN-6406) Remove SchedulerRequestKeys when no more pending ResourceRequest
[ https://issues.apache.org/jira/browse/YARN-6406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957387#comment-15957387 ] Wangda Tan commented on YARN-6406: -- [~asuresh], I committed but forgot push to trunk yesterday. Just pushed to trunk. > Remove SchedulerRequestKeys when no more pending ResourceRequest > > > Key: YARN-6406 > URL: https://issues.apache.org/jira/browse/YARN-6406 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.8.0, 2.7.3, 3.0.0-alpha2 >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 3.0.0-alpha3 > > Attachments: YARN-6406.001.patch, YARN-6406.002.patch > > > YARN-5540 introduced some optimizations to remove satisfied SchedulerKeys > from the AppScheduleingInfo. It looks like after YARN-6040, > ScedulerRequestKeys are removed only if the Application sends a 0 > numContainers requests. While earlier, the outstanding schedulerKeys were > also remove as soon as a container is allocated as well. > An additional optimization we were hoping to include is to remove the > ResourceRequests itself once the numContainers == 0, since we see in our > clusters that the RM heap space consumption increases drastically due to a > large number of ResourceRequests with 0 num containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6004) Refactor TestResourceLocalizationService#testDownloadingResourcesOnContainer so that it is less than 150 lines
[ https://issues.apache.org/jira/browse/YARN-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957380#comment-15957380 ] Chris Trezzo commented on YARN-6004: Thanks [~mingma]! > Refactor TestResourceLocalizationService#testDownloadingResourcesOnContainer > so that it is less than 150 lines > -- > > Key: YARN-6004 > URL: https://issues.apache.org/jira/browse/YARN-6004 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Reporter: Chris Trezzo >Assignee: Chris Trezzo >Priority: Trivial > Labels: newbie > Fix For: 2.9.0, 3.0.0-alpha3 > > Attachments: YARN-6004-branch-2.001.patch, YARN-6004-trunk.001.patch, > YARN-6004-trunk.002.patch > > > The TestResourceLocalizationService#testDownloadingResourcesOnContainerKill > method is over 150 lines: > bq. > ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestResourceLocalizationService.java:1128: > @Test(timeout = 2):3: Method length is 242 lines (max allowed is 150). > This method needs to be refactored and broken up into smaller methods. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6381) FSAppAttempt has several variables that should be final
[
https://issues.apache.org/jira/browse/YARN-6381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957363#comment-15957363
]
Daniel Templeton commented on YARN-6381:
LGTM +1 I'll commit in a bit.
> FSAppAttempt has several variables that should be final
> ---
>
> Key: YARN-6381
> URL: https://issues.apache.org/jira/browse/YARN-6381
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.0.0-alpha2
>Reporter: Daniel Templeton
>Assignee: Ameet Zaveri
> Labels: newbie
> Attachments: YARN-6381.001.patch
>
>
> These should all be final:
> {code}
> private long startTime;
> private Priority appPriority;
> private ResourceWeights resourceWeights;
> private FairScheduler scheduler;
> private Map reservations = new HashMap<>();
> private List blacklistNodeIds = new ArrayList<>();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6438) Code can be improved in ContainersMonitorImpl.java
[
https://issues.apache.org/jira/browse/YARN-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957335#comment-15957335
]
Hadoop QA commented on YARN-6438:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 12m
48s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-6438 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12862115/YARN-6438.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a2755228bfa8 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 34ab8e7 |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/15530/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/15530/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Code can be improved in ContainersMonitorImpl.java
> --
>
> Key: YARN-6438
> URL: https://issues.apache.org/jira/browse/YARN-6438
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>
[jira] [Updated] (YARN-5602) Utils for Federation State and Policy Store
[ https://issues.apache.org/jira/browse/YARN-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-5602: --- Attachment: (was: YARN-5602-YARN-2915.v6.patch) > Utils for Federation State and Policy Store > --- > > Key: YARN-5602 > URL: https://issues.apache.org/jira/browse/YARN-5602 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Labels: oct16-medium > Attachments: YARN-5602-YARN-2915.v1.patch, > YARN-5602-YARN-2915.v2.patch, YARN-5602-YARN-2915.v3.patch, > YARN-5602-YARN-2915.v4.patch, YARN-5602-YARN-2915.v5.patch, > YARN-5602-YARN-2915.v6.patch > > > This JIRA tracks the creation of utils for Federation State and Policy Store > such as Error Codes, Exceptions... -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5602) Utils for Federation State and Policy Store
[ https://issues.apache.org/jira/browse/YARN-5602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-5602: --- Attachment: YARN-5602-YARN-2915.v6.patch > Utils for Federation State and Policy Store > --- > > Key: YARN-5602 > URL: https://issues.apache.org/jira/browse/YARN-5602 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Labels: oct16-medium > Attachments: YARN-5602-YARN-2915.v1.patch, > YARN-5602-YARN-2915.v2.patch, YARN-5602-YARN-2915.v3.patch, > YARN-5602-YARN-2915.v4.patch, YARN-5602-YARN-2915.v5.patch, > YARN-5602-YARN-2915.v6.patch > > > This JIRA tracks the creation of utils for Federation State and Policy Store > such as Error Codes, Exceptions... -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6403) Invalid local resource request can raise NPE and make NM exit
[
https://issues.apache.org/jira/browse/YARN-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957322#comment-15957322
]
Hudson commented on YARN-6403:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11530 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/11530/])
YARN-6403. Invalid local resource request can raise NPE and make NM (jlowe: rev
e8071aa249c7b21b1de084ee5a9ca2a44efd3bf0)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerLaunchContextPBImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/api/records/impl/pb/TestApplicationClientProtocolRecords.java
> Invalid local resource request can raise NPE and make NM exit
> -
>
> Key: YARN-6403
> URL: https://issues.apache.org/jira/browse/YARN-6403
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.8.0
>Reporter: Tao Yang
>Assignee: Tao Yang
> Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3
>
> Attachments: YARN-6403.001.patch, YARN-6403.002.patch,
> YARN-6403.004.patch, YARN-6403.branch-2.8.003.patch,
> YARN-6403.branch-2.8.004.patch, YARN-6403.branch-2.8.004.patch
>
>
> Recently we found this problem on our testing environment. The app that
> caused this problem added a invalid local resource request(have no location)
> into ContainerLaunchContext like this:
> {code}
> localResources.put("test", LocalResource.newInstance(location,
> LocalResourceType.FILE, LocalResourceVisibility.PRIVATE, 100,
> System.currentTimeMillis()));
> ContainerLaunchContext amContainer =
> ContainerLaunchContext.newInstance(localResources, environment,
> vargsFinal, null, securityTokens, acls);
> {code}
> The actual value of location was null although app doesn't expect that. This
> mistake cause several NMs exited with the NPE below and can't restart until
> the nm recovery dirs were deleted.
> {code}
> FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourceRequest.(LocalResourceRequest.java:46)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$RequestResourcesTransition.transition(ContainerImpl.java:711)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$RequestResourcesTransition.transition(ContainerImpl.java:660)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:1320)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:88)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1293)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1286)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> NPE occured when created LocalResourceRequest instance for invalid resource
> request.
> {code}
> public LocalResourceRequest(LocalResource resource)
> throws URISyntaxException {
> this(resource.getResource().toPath(), //NPE occurred here
> resource.getTimestamp(),
> resource.getType(),
>
[jira] [Commented] (YARN-6443) Allow for Priority order relaxing in favor of improved node/rack locality
[ https://issues.apache.org/jira/browse/YARN-6443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957308#comment-15957308 ] Arun Suresh commented on YARN-6443: --- (+ [~hrsharma] , [~roniburd] since they've been investigating very related stuff lately) > Allow for Priority order relaxing in favor of improved node/rack locality > -- > > Key: YARN-6443 > URL: https://issues.apache.org/jira/browse/YARN-6443 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, fairscheduler >Reporter: Arun Suresh >Assignee: Arun Suresh > > Currently the Schedulers examine an applications pending Requests in Priority > order. This JIRA proposes to introduce a flag (either via the > ApplicationMasterService::registerApplication() or via some Scheduler > configuration) to favor an ordering that is baised to the node that is > currently heartbeating by relaxing the priority constraint. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6403) Invalid local resource request can raise NPE and make NM exit
[
https://issues.apache.org/jira/browse/YARN-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957290#comment-15957290
]
Jason Lowe commented on YARN-6403:
--
+1 for the latest trunk and 2.8 patches. Committing this.
> Invalid local resource request can raise NPE and make NM exit
> -
>
> Key: YARN-6403
> URL: https://issues.apache.org/jira/browse/YARN-6403
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.8.0
>Reporter: Tao Yang
>Assignee: Tao Yang
> Attachments: YARN-6403.001.patch, YARN-6403.002.patch,
> YARN-6403.004.patch, YARN-6403.branch-2.8.003.patch,
> YARN-6403.branch-2.8.004.patch, YARN-6403.branch-2.8.004.patch
>
>
> Recently we found this problem on our testing environment. The app that
> caused this problem added a invalid local resource request(have no location)
> into ContainerLaunchContext like this:
> {code}
> localResources.put("test", LocalResource.newInstance(location,
> LocalResourceType.FILE, LocalResourceVisibility.PRIVATE, 100,
> System.currentTimeMillis()));
> ContainerLaunchContext amContainer =
> ContainerLaunchContext.newInstance(localResources, environment,
> vargsFinal, null, securityTokens, acls);
> {code}
> The actual value of location was null although app doesn't expect that. This
> mistake cause several NMs exited with the NPE below and can't restart until
> the nm recovery dirs were deleted.
> {code}
> FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.LocalResourceRequest.(LocalResourceRequest.java:46)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$RequestResourcesTransition.transition(ContainerImpl.java:711)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$RequestResourcesTransition.transition(ContainerImpl.java:660)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:1320)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:88)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1293)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1286)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> NPE occured when created LocalResourceRequest instance for invalid resource
> request.
> {code}
> public LocalResourceRequest(LocalResource resource)
> throws URISyntaxException {
> this(resource.getResource().toPath(), //NPE occurred here
> resource.getTimestamp(),
> resource.getType(),
> resource.getVisibility(),
> resource.getPattern());
> }
> {code}
> We can't guarantee the validity of local resource request now, but we could
> avoid damaging the cluster. Perhaps we can verify the resource both in
> ContainerLaunchContext and LocalResourceRequest? Please feel free to give
> your suggestions.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6438) Code can be improved in ContainersMonitorImpl.java
[ https://issues.apache.org/jira/browse/YARN-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-6438: - Attachment: YARN-6438.002.patch Addressing comments > Code can be improved in ContainersMonitorImpl.java > -- > > Key: YARN-6438 > URL: https://issues.apache.org/jira/browse/YARN-6438 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Labels: newbie > Attachments: YARN-6438.000.patch, YARN-6438.001.patch, > YARN-6438.002.patch > > > I noticed two code blocks that can be improved in ContainersMonitorImpl.java. > cpuUsagePercentPerCoreByAllContainers and cpuUsageTotalCoresByAllContainers > track the same value and CHANGE_MONITORING_CONTAINER_RESOURCE is checked > twice along with two calls to changeContainerResource. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6438) Code can be improved in ContainersMonitorImpl.java
[
https://issues.apache.org/jira/browse/YARN-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957278#comment-15957278
]
Yufei Gu commented on YARN-6438:
Patch looks good. Only one nit, this comments should be rewrite.
{code}
// if machine has 6 cores and 3 are used,
// cpuUsagePercentPerCore should be 300% and
// cpuUsageTotalCoresPercentage should be 50%
{code}
> Code can be improved in ContainersMonitorImpl.java
> --
>
> Key: YARN-6438
> URL: https://issues.apache.org/jira/browse/YARN-6438
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Reporter: Miklos Szegedi
>Assignee: Miklos Szegedi
>Priority: Minor
> Labels: newbie
> Attachments: YARN-6438.000.patch, YARN-6438.001.patch
>
>
> I noticed two code blocks that can be improved in ContainersMonitorImpl.java.
> cpuUsagePercentPerCoreByAllContainers and cpuUsageTotalCoresByAllContainers
> track the same value and CHANGE_MONITORING_CONTAINER_RESOURCE is checked
> twice along with two calls to changeContainerResource.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6438) Code can be improved in ContainersMonitorImpl.java
[ https://issues.apache.org/jira/browse/YARN-6438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-6438: - Attachment: YARN-6438.001.patch Thank you for the review [~yufeigu]. Addressing the comments > Code can be improved in ContainersMonitorImpl.java > -- > > Key: YARN-6438 > URL: https://issues.apache.org/jira/browse/YARN-6438 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Minor > Labels: newbie > Attachments: YARN-6438.000.patch, YARN-6438.001.patch > > > I noticed two code blocks that can be improved in ContainersMonitorImpl.java. > cpuUsagePercentPerCoreByAllContainers and cpuUsageTotalCoresByAllContainers > track the same value and CHANGE_MONITORING_CONTAINER_RESOURCE is checked > twice along with two calls to changeContainerResource. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3760) FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close()
[ https://issues.apache.org/jira/browse/YARN-3760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957202#comment-15957202 ] Haibo Chen commented on YARN-3760: -- [~miklos.szeg...@cloudera.com] Both the writer and the fsDataOuptutStream need to be closed. > FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close() > > > Key: YARN-3760 > URL: https://issues.apache.org/jira/browse/YARN-3760 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.4.0 >Reporter: Daryn Sharp >Assignee: Haibo Chen >Priority: Critical > Attachments: YARN-3760.00.patch > > > The aggregated log file does not appear to be properly closed when writes > fail. This leaves a lease renewer active in the NM that spams the NN with > lease renewals. If the token is marked not to be cancelled, the renewals > appear to continue until the token expires. If the token is cancelled, the > periodic renew spam turns into a flood of failed connections until the lease > renewer gives up. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6403) Invalid local resource request can raise NPE and make NM exit
[
https://issues.apache.org/jira/browse/YARN-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957174#comment-15957174
]
Hadoop QA commented on YARN-6403:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
44s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
0s{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} branch-2.8 passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} branch-2.8 passed with JDK v1.7.0_121 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
11s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 2 new + 75 unchanged - 0 fixed = 77 total (was 75) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed with JDK v1.8.0_121 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} the patch passed with JDK v1.7.0_121 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
30s{color} | {color:green} hadoop-yarn-common in the patch passed with JDK
v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
51s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed
with JDK v1.7.0_121. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 62m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:5af2af1 |
| JIRA Issue | YARN-6403 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861974/YARN-6403.branch-2.8.004.patch
[jira] [Commented] (YARN-5531) UnmanagedAM pool manager for federating application across clusters
[
https://issues.apache.org/jira/browse/YARN-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957131#comment-15957131
]
Botong Huang commented on YARN-5531:
Thanks [~kasha] for the review! For 2, sure I will add more logs. For 1, to me
this class is more like a utility rather than a service, which hides the fact
that this is an UAM in the remote Yarn cluster, and present an aync version of
{ApplicationMasterProtocol} RM interface to user (the actual AM). We can modify
it into a service as well if needed.
[~subru]: along this line of thought, maybe we should call this class
UnmanagedApplicationMasterProxy? Thanks!
> UnmanagedAM pool manager for federating application across clusters
> ---
>
> Key: YARN-5531
> URL: https://issues.apache.org/jira/browse/YARN-5531
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Botong Huang
> Attachments: YARN-5531-YARN-2915.v1.patch,
> YARN-5531-YARN-2915.v2.patch, YARN-5531-YARN-2915.v3.patch
>
>
> One of the main tenets the YARN Federation is to *transparently* scale
> applications across multiple clusters. This is achieved by running UAMs on
> behalf of the application on other clusters. This JIRA tracks the addition of
> a UnmanagedAM pool manager for federating application across clusters which
> will be used the FederationInterceptor (YARN-3666) which is part of the
> AMRMProxy pipeline introduced in YARN-2884.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6195) Export UsedCapacity and AbsoluteUsedCapacity to JMX
[ https://issues.apache.org/jira/browse/YARN-6195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benson Qiu updated YARN-6195: - Attachment: YARN-6195.004.patch > Export UsedCapacity and AbsoluteUsedCapacity to JMX > --- > > Key: YARN-6195 > URL: https://issues.apache.org/jira/browse/YARN-6195 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler, metrics, yarn >Affects Versions: 3.0.0-alpha3 >Reporter: Benson Qiu >Assignee: Benson Qiu > Attachments: YARN-6195.001.patch, YARN-6195.002.patch, > YARN-6195.003.patch, YARN-6195.004.patch > > > `usedCapacity` and `absoluteUsedCapacity` are currently not available as JMX. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5892) Capacity Scheduler: Support user-specific minimum user limit percent
[
https://issues.apache.org/jira/browse/YARN-5892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957125#comment-15957125
]
Hadoop QA commented on YARN-5892:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
8s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 7s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 15 new + 572 unchanged - 1 fixed = 587 total (was 573) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
30s{color} | {color:red}
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
generated 4 new + 880 unchanged - 0 fixed = 884 total (was 880) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
36s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 40m 16s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
14s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}109m 13s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5892 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12861980/YARN-5892.007.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname
[jira] [Commented] (YARN-6091) the AppMaster register failed when use Docker on LinuxContainer
[ https://issues.apache.org/jira/browse/YARN-6091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957094#comment-15957094 ] Eric Badger commented on YARN-6091: --- [~zhengchenyu], do you have logs for this issue or are you able to reproduce it? Also, do you know which pclose failed inside of container-executor.c? > the AppMaster register failed when use Docker on LinuxContainer > > > Key: YARN-6091 > URL: https://issues.apache.org/jira/browse/YARN-6091 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, yarn >Affects Versions: 2.8.1 > Environment: CentOS >Reporter: zhengchenyu >Priority: Critical > Original Estimate: 336h > Remaining Estimate: 336h > > In some servers, When I use Docker on LinuxContainer, I found the aciton that > AppMaster register to Resourcemanager failed. But didn't happen in other > servers. > I found the pclose (in container-executor.c) return different value in > different server, even though the process which is launched by popen is > running normally. Some server return 0, and others return 13. > Because yarn regard the application as failed application when pclose return > nonzero, and yarn will remove the AMRMToken, then the AppMaster register > failed because Resourcemanager have removed this applicaiton's token. > In container-executor.c, the judgement condition is whether the return code > is zero. But man the pclose, the document tells that "pclose return -1" > represent wrong. So I change the judgement condition, then slove this > problem. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
