[jira] [Commented] (YARN-763) AMRMClientAsync should stop heartbeating after receiving shutdown from RM
[ https://issues.apache.org/jira/browse/YARN-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705515#comment-13705515 ] Xuan Gong commented on YARN-763: Oh, You are right. If we want to let CallBackThread to call asyncClient.stop(), we might need to add this part of code inside the CallBackThread.run(). In that case, we may need to create a new test class, such as mockAMRMClientAsync, and re-write CallBackThread.run(). Any other ideas ? > AMRMClientAsync should stop heartbeating after receiving shutdown from RM > - > > Key: YARN-763 > URL: https://issues.apache.org/jira/browse/YARN-763 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-763.1.patch, YARN-763.2.patch, YARN-763.3.patch, > YARN-763.4.patch, YARN-763.5.patch, YARN-763.6.patch, YARN-763.7.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-347) YARN node CLI should also show CPU info as memory info in node status
[ https://issues.apache.org/jira/browse/YARN-347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705512#comment-13705512 ] Junping Du commented on YARN-347: - Hi, [~acmurthy] The patch is rebase to latest trunk. Would you help to review it again? Thx! > YARN node CLI should also show CPU info as memory info in node status > - > > Key: YARN-347 > URL: https://issues.apache.org/jira/browse/YARN-347 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-347.patch, YARN-347-v2.patch > > > With YARN-2 checked in, CPU info are taken into consideration in resource > scheduling. yarn node -status should show CPU used and capacity info > as memory info. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-879) Fix NPE in test/o.a.h.y.server.resourcemanager.Application.getResources()
[ https://issues.apache.org/jira/browse/YARN-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705501#comment-13705501 ] Junping Du commented on YARN-879: - Sure. Vinod. I will try to make tests work. Thanks for sharing background here. :) > Fix NPE in test/o.a.h.y.server.resourcemanager.Application.getResources() > - > > Key: YARN-879 > URL: https://issues.apache.org/jira/browse/YARN-879 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-879.patch > > > getResources() will return a list of containers that allocated by RM. > However, it is now return null directly. The worse thing is: if LOG.debug is > enabled, then it will definitely cause NPE exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-865) RM webservices can't query on application Types
[
https://issues.apache.org/jira/browse/YARN-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705477#comment-13705477
]
Hitesh Shah commented on YARN-865:
--
[~xgong] why is the following done for each rm app ( i.e inside the for loop )?
{code}
+ if (appTypesQuery != null && !appTypesQuery.isEmpty()) {
+String[] appTypes = appTypesQuery.split(",");
+Set applicationTypes = new HashSet();
+for (String appType : appTypes) {
+ if (!appType.trim().isEmpty()) {
+applicationTypes.add(appType);
+ }
+}
{code}
> RM webservices can't query on application Types
> ---
>
> Key: YARN-865
> URL: https://issues.apache.org/jira/browse/YARN-865
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: MR-5337.1.patch, YARN-865.1.patch, YARN-865.2.patch
>
>
> The resource manager web service api to get the list of apps doesn't have a
> query parameter for appTypes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-654) AMRMClient: Perform sanity checks for parameters of public methods
[
https://issues.apache.org/jira/browse/YARN-654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705470#comment-13705470
]
Hadoop QA commented on YARN-654:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591792/YARN-654.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client:
org.apache.hadoop.yarn.client.api.impl.TestNMClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1457//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1457//console
This message is automatically generated.
> AMRMClient: Perform sanity checks for parameters of public methods
> --
>
> Key: YARN-654
> URL: https://issues.apache.org/jira/browse/YARN-654
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bikas Saha
>Assignee: Xuan Gong
> Fix For: 2.1.0-beta
>
> Attachments: YARN-654.1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-763) AMRMClientAsync should stop heartbeating after receiving shutdown from RM
[ https://issues.apache.org/jira/browse/YARN-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705464#comment-13705464 ] Jian He commented on YARN-763: -- testCallAMRMClientAsyncStopFromCallbackHandler: In the test case, asyncClient.stop() is still called from the main test thread, not from callback thread. we need a way to simulate callback thread can actually call asyncClient.stop() we should also prevent the main test thread from exiting before the callback thread actually completes calling asyncClient.stop(). > AMRMClientAsync should stop heartbeating after receiving shutdown from RM > - > > Key: YARN-763 > URL: https://issues.apache.org/jira/browse/YARN-763 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-763.1.patch, YARN-763.2.patch, YARN-763.3.patch, > YARN-763.4.patch, YARN-763.5.patch, YARN-763.6.patch, YARN-763.7.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-654) AMRMClient: Perform sanity checks for parameters of public methods
[ https://issues.apache.org/jira/browse/YARN-654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-654: --- Attachment: YARN-654.1.patch Do the sanity checks(null and negative) for public methods in AMRMClient api. No testcase added > AMRMClient: Perform sanity checks for parameters of public methods > -- > > Key: YARN-654 > URL: https://issues.apache.org/jira/browse/YARN-654 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Xuan Gong > Fix For: 2.1.0-beta > > Attachments: YARN-654.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-661) NM fails to cleanup local directories for users
[
https://issues.apache.org/jira/browse/YARN-661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705410#comment-13705410
]
Hadoop QA commented on YARN-661:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591780/YARN-661-20130710.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1455//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/1455//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1455//console
This message is automatically generated.
> NM fails to cleanup local directories for users
> ---
>
> Key: YARN-661
> URL: https://issues.apache.org/jira/browse/YARN-661
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.1.0-beta, 0.23.8
>Reporter: Jason Lowe
>Assignee: Omkar Vinit Joshi
> Attachments: YARN-661-20130701.patch, YARN-661-20130708.patch,
> YARN-661-20130710.1.patch
>
>
> YARN-71 added deletion of local directories on startup, but in practice it
> fails to delete the directories because of permission problems. The
> top-level usercache directory is owned by the user but is in a directory that
> is not writable by the user. Therefore the deletion of the user's usercache
> directory, as the user, fails due to lack of permissions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-521) Augment AM - RM client module to be able to request containers only at specific locations
[
https://issues.apache.org/jira/browse/YARN-521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705408#comment-13705408
]
Hadoop QA commented on YARN-521:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591779/YARN-521-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1456//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1456//console
This message is automatically generated.
> Augment AM - RM client module to be able to request containers only at
> specific locations
> -
>
> Key: YARN-521
> URL: https://issues.apache.org/jira/browse/YARN-521
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api
>Affects Versions: 2.0.3-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: YARN-521-1.patch, YARN-521-2.patch, YARN-521-2.patch,
> YARN-521-3.patch, YARN-521.patch
>
>
> When YARN-392 and YARN-398 are completed, it would be good for AMRMClient to
> offer an easy way to access their functionality
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-661) NM fails to cleanup local directories for users
[ https://issues.apache.org/jira/browse/YARN-661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated YARN-661: --- Attachment: YARN-661-20130710.1.patch > NM fails to cleanup local directories for users > --- > > Key: YARN-661 > URL: https://issues.apache.org/jira/browse/YARN-661 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.1.0-beta, 0.23.8 >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi > Attachments: YARN-661-20130701.patch, YARN-661-20130708.patch, > YARN-661-20130710.1.patch > > > YARN-71 added deletion of local directories on startup, but in practice it > fails to delete the directories because of permission problems. The > top-level usercache directory is owned by the user but is in a directory that > is not writable by the user. Therefore the deletion of the user's usercache > directory, as the user, fails due to lack of permissions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-659) RMStateStore's removeApplication APIs should just take an applicationId
[ https://issues.apache.org/jira/browse/YARN-659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla reassigned YARN-659: - Assignee: Karthik Kambatla > RMStateStore's removeApplication APIs should just take an applicationId > --- > > Key: YARN-659 > URL: https://issues.apache.org/jira/browse/YARN-659 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Karthik Kambatla > > There is no need to give in the whole state for removal - just an ID should > be enough when an app finishes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-521) Augment AM - RM client module to be able to request containers only at specific locations
[ https://issues.apache.org/jira/browse/YARN-521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705395#comment-13705395 ] Sandy Ryza commented on YARN-521: - Uploaded a patch that addresses Bikas's comments > Augment AM - RM client module to be able to request containers only at > specific locations > - > > Key: YARN-521 > URL: https://issues.apache.org/jira/browse/YARN-521 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-521-1.patch, YARN-521-2.patch, YARN-521-2.patch, > YARN-521-3.patch, YARN-521.patch > > > When YARN-392 and YARN-398 are completed, it would be good for AMRMClient to > offer an easy way to access their functionality -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-521) Augment AM - RM client module to be able to request containers only at specific locations
[ https://issues.apache.org/jira/browse/YARN-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated YARN-521: Attachment: YARN-521-3.patch > Augment AM - RM client module to be able to request containers only at > specific locations > - > > Key: YARN-521 > URL: https://issues.apache.org/jira/browse/YARN-521 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-521-1.patch, YARN-521-2.patch, YARN-521-2.patch, > YARN-521-3.patch, YARN-521.patch > > > When YARN-392 and YARN-398 are completed, it would be good for AMRMClient to > offer an easy way to access their functionality -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-661) NM fails to cleanup local directories for users
[
https://issues.apache.org/jira/browse/YARN-661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705393#comment-13705393
]
Omkar Vinit Joshi commented on YARN-661:
Thanks [~zjshen] .. uploading updated patch..
bq. Not sure getCancelIfAnyDependentTaskFailedFlag is necessary. If deletion of
the sub-dir fails, should deletion of the root-dir fail automatically?
In this scenario we want to cancel this but may not be the case always. I will
remove it but that would have been an added control for user.
bq. Instead of synchronizing over HashMap, how about using ConcurrentHashMap
instead?
yeah fixed it..
{code}
+List dependentTaskList =
+this.deletionTaskDependencyMap.putIfAbsent(
+parentTask, new ArrayList());
{code}
there is a problem with this as we end up creating new objects for every call.
It is present at lot of places in yarn and need to be fixed. the containsKey
check even though looks complicated avoids extra object creation on heap.
bq. WRT the dependency, I'd like rather call task pair predecessor and
successor instead of parent and child, because it's a dag not a tree. Moreover,
how about defining public void
populateFileDeletionTaskDependency(List, FileDeletion), and
populateFileDeletionTaskDependency(List, List)
wraps it. In fact, the former one seems to be enough for the patch.
yeah renaming parent -> predecessor and child -> successor. I think current
(List<>, List<>) api is more flexible and we can keep it that way.
bq. How about calling it delete as well, just overloading the method?
yeah wanted to overload only but is not helping me in junit tests.
MockitTo.verify is not smart enough to distinguish between overloaded
methods...hence giving different name...
{code}
verify(delService, times(1)).deleteHelper(
argThat(new FileDeletionInclude(user, null,
new String[] { destinationFile })));
verify(delService, times(1)).deleteHelper(
argThat(new FileDeletionInclude(null, ContainerLocalizer.USERCACHE
+ "_DEL_", new String[] {})));
{code}
bq. There's one finbugs warning to fix.
Hopefully fixed it.. I think it is complaining because I am exposing final
array. This too I need it for testing. Let me see if findbug complains again
then will fix it in some other way.
bq. Please use LocalResource.newInstance instead.
not related to this patch ..was just reformatting..but fixed it ..trivial fix.
bq. How about refactoring the code in the following pattern, which should be
more clear.
makes sense... fixed it..
> NM fails to cleanup local directories for users
> ---
>
> Key: YARN-661
> URL: https://issues.apache.org/jira/browse/YARN-661
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.1.0-beta, 0.23.8
>Reporter: Jason Lowe
>Assignee: Omkar Vinit Joshi
> Attachments: YARN-661-20130701.patch, YARN-661-20130708.patch
>
>
> YARN-71 added deletion of local directories on startup, but in practice it
> fails to delete the directories because of permission problems. The
> top-level usercache directory is owned by the user but is in a directory that
> is not writable by the user. Therefore the deletion of the user's usercache
> directory, as the user, fails due to lack of permissions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-865) RM webservices can't query on application Types
[
https://issues.apache.org/jira/browse/YARN-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705391#comment-13705391
]
Hadoop QA commented on YARN-865:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591770/YARN-865.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1454//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1454//console
This message is automatically generated.
> RM webservices can't query on application Types
> ---
>
> Key: YARN-865
> URL: https://issues.apache.org/jira/browse/YARN-865
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: MR-5337.1.patch, YARN-865.1.patch, YARN-865.2.patch
>
>
> The resource manager web service api to get the list of apps doesn't have a
> query parameter for appTypes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-521) Augment AM - RM client module to be able to request containers only at specific locations
[
https://issues.apache.org/jira/browse/YARN-521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705387#comment-13705387
]
Sandy Ryza commented on YARN-521:
-
Thinking about this a little more, I think a specific host should be able to be
mixed with the rack it resides on. This means "I'd prefer the host, but
anything on the rack is acceptable". Here's the documentation I wrote, which
includes this:
{code}
* The way that YARN handles requesting containers in specific locations is
* somewhat subtle. Containers may be requested on specific nodes and/or on
* specific racks. By default, locations requested are only suggestions and
* the scheduler may return containers outside of them. A request may also
* be submitted with locality relaxation turned off, in which case containers
* for it will never be assigned on nodes and racks other than the ones
* explicitly specified in it. A request with locality relaxation off that
* specifies both a node and the rack it resides on means that the node is
* preferred, but another node on the rack is still acceptable.
*
* A couple restrictions apply to requests with locality relaxation turned
off.
* A request without locality relaxation may not be submitted at the same
* priority as one with locality relaxation turned on. Additionally, a
request
* without locality relaxation that explicitly includes a node, but not the
rack
* that the node resides on, may not be submitted at the same priority as a
* request without locality relaxation that explicitly asks for that rack.
{code}
> Augment AM - RM client module to be able to request containers only at
> specific locations
> -
>
> Key: YARN-521
> URL: https://issues.apache.org/jira/browse/YARN-521
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api
>Affects Versions: 2.0.3-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: YARN-521-1.patch, YARN-521-2.patch, YARN-521-2.patch,
> YARN-521.patch
>
>
> When YARN-392 and YARN-398 are completed, it would be good for AMRMClient to
> offer an easy way to access their functionality
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-569) CapacityScheduler: support for preemption (using a capacity monitor)
[ https://issues.apache.org/jira/browse/YARN-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705385#comment-13705385 ] Hudson commented on YARN-569: - Integrated in Hadoop-trunk-Commit #4065 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4065/]) YARN-569. Add support for requesting and enforcing preemption requests via a capacity monitor. Contributed by Carlo Curino, Chris Douglas (Revision 1502083) Result = SUCCESS cdouglas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1502083 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/SchedulingEditPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/SchedulingMonitor.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ContainerPreemptEvent.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ContainerPreemptEventType.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/PreemptableResourceScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerNode.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java > CapacityScheduler: support for preemption (using a capacity monitor) > > > Key: YARN-569 > URL: https://issues.apache.org/jira/browse/YARN-569 >
[jira] [Commented] (YARN-873) YARNClient.getApplicationReport(unknownAppId) returns a null report
[ https://issues.apache.org/jira/browse/YARN-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705370#comment-13705370 ] Xuan Gong commented on YARN-873: OR, add String type diagnosis in those Response. If we query the unknownAppId, just set the diagnosis as "Application with id '" + applicationId + "' doesn't exist in RM.". When we output applicationReport, we print out diagnosis, too > YARNClient.getApplicationReport(unknownAppId) returns a null report > --- > > Key: YARN-873 > URL: https://issues.apache.org/jira/browse/YARN-873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Xuan Gong > > How can the client find out that app does not exist? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-569) CapacityScheduler: support for preemption (using a capacity monitor)
[
https://issues.apache.org/jira/browse/YARN-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705355#comment-13705355
]
Hadoop QA commented on YARN-569:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591768/YARN-569.11.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1453//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1453//console
This message is automatically generated.
> CapacityScheduler: support for preemption (using a capacity monitor)
>
>
> Key: YARN-569
> URL: https://issues.apache.org/jira/browse/YARN-569
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Attachments: 3queues.pdf, CapScheduler_with_preemption.pdf,
> preemption.2.patch, YARN-569.10.patch, YARN-569.11.patch, YARN-569.1.patch,
> YARN-569.2.patch, YARN-569.3.patch, YARN-569.4.patch, YARN-569.5.patch,
> YARN-569.6.patch, YARN-569.8.patch, YARN-569.9.patch, YARN-569.patch,
> YARN-569.patch
>
>
> There is a tension between the fast-pace reactive role of the
> CapacityScheduler, which needs to respond quickly to
> applications resource requests, and node updates, and the more introspective,
> time-based considerations
> needed to observe and correct for capacity balance. To this purpose we opted
> instead of hacking the delicate
> mechanisms of the CapacityScheduler directly to add support for preemption by
> means of a "Capacity Monitor",
> which can be run optionally as a separate service (much like the
> NMLivelinessMonitor).
> The capacity monitor (similarly to equivalent functionalities in the fairness
> scheduler) operates running on intervals
> (e.g., every 3 seconds), observe the state of the assignment of resources to
> queues from the capacity scheduler,
> performs off-line computation to determine if preemption is needed, and how
> best to "edit" the current schedule to
> improve capacity, and generates events that produce four possible actions:
> # Container de-reservations
> # Resource-based preemptions
> # Container-based preemptions
> # Container killing
> The actions listed above are progressively more costly, and it is up to the
> policy to use them as desired to achieve the rebalancing goals.
> Note that due to the "lag" in the effect of these actions the policy should
> operate at the macroscopic level (e.g., preempt tens of containers
> from a queue) and not trying to tightly and consistently micromanage
> container allocations.
> - Preemption policy (ProportionalCapacityPreemptionPolicy):
> -
> Preemption policies are by design pluggable, in the following we present an
> initial policy (ProportionalCapacityPreemptionPolicy) we have been
> experimenting with. The ProportionalCapacityPreemptionPolicy behaves as
> follows:
> # it gathers from the scheduler the state of the queues, in particular, their
> current capacity, guaranteed capacity and pending requests (*)
> # if there are pending requests from queues that are under capacity it
> computes a new ideal balanced state (**)
> # it computes the set of preemptions needed to repair the current schedule
> and achieve capacity balance (accounting for natural completion rates, and
> respecting bounds on the amount of preemption we allow for each round)
> # it selects which applications to preempt from each over-capacity queue (the
> last one in the FIFO order)
> # it remove reservations from the most recently assigned app until the amount
> of resource to
[jira] [Commented] (YARN-865) RM webservices can't query on application Types
[ https://issues.apache.org/jira/browse/YARN-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705349#comment-13705349 ] Xuan Gong commented on YARN-865: 1.add the test case for empty type string 2.add Documentation on how to use applicationTypes to filter the applications in ResourceManagerRest.apt.vm > RM webservices can't query on application Types > --- > > Key: YARN-865 > URL: https://issues.apache.org/jira/browse/YARN-865 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: MR-5337.1.patch, YARN-865.1.patch, YARN-865.2.patch > > > The resource manager web service api to get the list of apps doesn't have a > query parameter for appTypes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-915) Apps Completed metrics on web UI is not correct after RM restart
[ https://issues.apache.org/jira/browse/YARN-915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-915: - Description: negative Apps completed metrics can show on web UI > Apps Completed metrics on web UI is not correct after RM restart > > > Key: YARN-915 > URL: https://issues.apache.org/jira/browse/YARN-915 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He > Attachments: screen shot.png > > > negative Apps completed metrics can show on web UI -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-915) Apps Completed metrics on web UI is not correct after RM restart
[ https://issues.apache.org/jira/browse/YARN-915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-915: - Attachment: screen shot.png Attach an image which shows the incorrect App complemented metrics. > Apps Completed metrics on web UI is not correct after RM restart > > > Key: YARN-915 > URL: https://issues.apache.org/jira/browse/YARN-915 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Jian He >Assignee: Jian He > Attachments: screen shot.png > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-873) YARNClient.getApplicationReport(unknownAppId) returns a null report
[ https://issues.apache.org/jira/browse/YARN-873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705345#comment-13705345 ] Xuan Gong commented on YARN-873: YARNClient.getApplicationReport(unknownAppId) and YARNClient.killApplication(unknowAppId) has very similar issue. But they handle it differently. One throws out a YarnException, the other one returns null. We might need to figure out a common way to do it. > YARNClient.getApplicationReport(unknownAppId) returns a null report > --- > > Key: YARN-873 > URL: https://issues.apache.org/jira/browse/YARN-873 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Bikas Saha >Assignee: Xuan Gong > > How can the client find out that app does not exist? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-915) Apps Completed metrics on web UI is not correct after RM restart
Jian He created YARN-915: Summary: Apps Completed metrics on web UI is not correct after RM restart Key: YARN-915 URL: https://issues.apache.org/jira/browse/YARN-915 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-521) Augment AM - RM client module to be able to request containers only at specific locations
[ https://issues.apache.org/jira/browse/YARN-521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705341#comment-13705341 ] Sandy Ryza commented on YARN-521: - bq. Shouldnt this also check that the inferred rack1 has relaxLocality set to false? This was meant to test the case where the inferred rack is also explicitly specified. Looks like I messed up NetworkTopology.DEFAULT_RACK vs. the one that MyResolver returns. Will fix this. bq. It would be more clear if the second request had a different host. Will make this change. bq. testLocalityRelaxationDifferentLevels() tests that specific host cannot be mixed with non-specific rack for the rack of that host. Can specific host be mixed with specific rack for the same rack as the host? Probably not. I meant for the documentation to cover this case: "If true, containers for this request may be assigned on hosts and racks other than the ones explicitly requested.", meaning that a container may be placed on any host on the rack, and that the host is redundant. I am OK with throwing an exception in this case if you think that is better. bq. Can specific node host1 be mixed with specific rack on a different rack (rack2)? If yes, if a container is allocated on host2 in rack2, then what prevents getMachingRequests from returning the requests for host1 and rack2 instead of only the request for rack2? The way I see it, getMatchingRequests(priority1, * ) should give me all the requests at priority1, independent of locality relaxation. bq. Now that we have a client and server pieces done, has this been tested with a real cluster to see that it works for in practice? I haven't done this and won't have a chance in the next couple days. If possible I would like to get this into 2.1.0-beta as there have been recent requests on the Hadoop user list for it, and including YARN-392 but omitting this will lead them away from AMRMClient (ref: http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201307.mbox/%3ccahg+sbmenvpsp4t5tbl3mi1osrjk-6+nxrafeko4ldelhps...@mail.gmail.com%3E). bq. I am concerned that we are implementing something pretty subtle and we need to get it right logically before implementing the code. I understand the concern and agree that some of the cases are pretty subtle. I have thought about this quite a bit and am convinced that the current approach is as coherent as we can get within the resource request model. I will try to lay it out more clearly in the javadoc and copy it on the JIRA to make it easier for posterity to reference. Ideally we would add it to the "Writing YARN Applications" documentation, but as it is very out of date and does not even reference AMRMClient, I think that is out of the scope of this JIRA. bq. The RM allows relaxLocality to be re-enabled on a location and priority. This allows users to change their minds about strict locality if they are unable to get those contabqiners for a long time. The logic in the patch does not allow this to happen. In fact testDifferentLocalityRelaxationSamePriority() shows that it is not possible to do this with the current patch. My intended behavior is that to do this the user must remove the original container request before adding a new one with different locality relaxation. Am I misunderstanding how removeContainerRequest works again? bq. Minor nits. Will make these changes. > Augment AM - RM client module to be able to request containers only at > specific locations > - > > Key: YARN-521 > URL: https://issues.apache.org/jira/browse/YARN-521 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-521-1.patch, YARN-521-2.patch, YARN-521-2.patch, > YARN-521.patch > > > When YARN-392 and YARN-398 are completed, it would be good for AMRMClient to > offer an easy way to access their functionality -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-763) AMRMClientAsync should stop heartbeating after receiving shutdown from RM
[
https://issues.apache.org/jira/browse/YARN-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705336#comment-13705336
]
Hadoop QA commented on YARN-763:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591769/YARN-763.7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1452//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1452//console
This message is automatically generated.
> AMRMClientAsync should stop heartbeating after receiving shutdown from RM
> -
>
> Key: YARN-763
> URL: https://issues.apache.org/jira/browse/YARN-763
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bikas Saha
>Assignee: Xuan Gong
> Attachments: YARN-763.1.patch, YARN-763.2.patch, YARN-763.3.patch,
> YARN-763.4.patch, YARN-763.5.patch, YARN-763.6.patch, YARN-763.7.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-883) Expose Fair Scheduler-specific queue metrics
[ https://issues.apache.org/jira/browse/YARN-883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705323#comment-13705323 ] Hudson commented on YARN-883: - Integrated in Hadoop-trunk-Commit #4064 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4064/]) updating CHANGES.txt after committing MAPREDUCE-5333,HADOOP-9661,HADOOP-9355,HADOOP-9673,HADOOP-9414,HADOOP-9416,HDFS-4797,YARN-866,YARN-736,YARN-883 to 2.1-beta branch (Revision 1502075) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1502075 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Expose Fair Scheduler-specific queue metrics > > > Key: YARN-883 > URL: https://issues.apache.org/jira/browse/YARN-883 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.5-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.1.0-beta > > Attachments: YARN-883-1.patch, YARN-883-1.patch, YARN-883.patch > > > When the Fair Scheduler is enabled, QueueMetrics should include fair share, > minimum share, and maximum share. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-866) Add test for class ResourceWeights
[ https://issues.apache.org/jira/browse/YARN-866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705326#comment-13705326 ] Hudson commented on YARN-866: - Integrated in Hadoop-trunk-Commit #4064 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4064/]) updating CHANGES.txt after committing MAPREDUCE-5333,HADOOP-9661,HADOOP-9355,HADOOP-9673,HADOOP-9414,HADOOP-9416,HDFS-4797,YARN-866,YARN-736,YARN-883 to 2.1-beta branch (Revision 1502075) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1502075 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Add test for class ResourceWeights > -- > > Key: YARN-866 > URL: https://issues.apache.org/jira/browse/YARN-866 > Project: Hadoop YARN > Issue Type: Test >Affects Versions: 2.1.0-beta >Reporter: Wei Yan >Assignee: Wei Yan > Fix For: 2.1.0-beta > > Attachments: Yarn-866.patch, Yarn-866.patch, YARN-866.patch > > > Add test case for the class ResourceWeights -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705324#comment-13705324 ] Hudson commented on YARN-736: - Integrated in Hadoop-trunk-Commit #4064 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4064/]) updating CHANGES.txt after committing MAPREDUCE-5333,HADOOP-9661,HADOOP-9355,HADOOP-9673,HADOOP-9414,HADOOP-9416,HDFS-4797,YARN-866,YARN-736,YARN-883 to 2.1-beta branch (Revision 1502075) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1502075 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.1.0-beta > > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736-4.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-865) RM webservices can't query on application Types
[ https://issues.apache.org/jira/browse/YARN-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-865: --- Attachment: YARN-865.2.patch > RM webservices can't query on application Types > --- > > Key: YARN-865 > URL: https://issues.apache.org/jira/browse/YARN-865 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: MR-5337.1.patch, YARN-865.1.patch, YARN-865.2.patch > > > The resource manager web service api to get the list of apps doesn't have a > query parameter for appTypes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-763) AMRMClientAsync should stop heartbeating after receiving shutdown from RM
[ https://issues.apache.org/jira/browse/YARN-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705290#comment-13705290 ] Xuan Gong commented on YARN-763: Add new patch to address all the comments > AMRMClientAsync should stop heartbeating after receiving shutdown from RM > - > > Key: YARN-763 > URL: https://issues.apache.org/jira/browse/YARN-763 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-763.1.patch, YARN-763.2.patch, YARN-763.3.patch, > YARN-763.4.patch, YARN-763.5.patch, YARN-763.6.patch, YARN-763.7.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-763) AMRMClientAsync should stop heartbeating after receiving shutdown from RM
[ https://issues.apache.org/jira/browse/YARN-763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-763: --- Attachment: YARN-763.7.patch > AMRMClientAsync should stop heartbeating after receiving shutdown from RM > - > > Key: YARN-763 > URL: https://issues.apache.org/jira/browse/YARN-763 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-763.1.patch, YARN-763.2.patch, YARN-763.3.patch, > YARN-763.4.patch, YARN-763.5.patch, YARN-763.6.patch, YARN-763.7.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-661) NM fails to cleanup local directories for users
[
https://issues.apache.org/jira/browse/YARN-661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705285#comment-13705285
]
Zhijie Shen commented on YARN-661:
--
I agree with the idea, and the patch looks generally good. Bellow are some
comments.
* Not sure getCancelIfAnyDependentTaskFailedFlag is necessary. If deletion of
the sub-dir fails, should deletion of the root-dir fail automatically?
* Instead of synchronizing over HashMap, how about using ConcurrentHashMap
instead?
{code}
+if (!this.deletionTaskDependencyMap.containsKey(parentTask)) {
+ this.deletionTaskDependencyMap.put(parentTask,
+new ArrayList());
+}
+List dependentTaskList =
+this.deletionTaskDependencyMap.get(parentTask);
{code}
can be simplified as
{code}
+List dependentTaskList =
+this.deletionTaskDependencyMap.putIfAbsent(
+parentTask, new ArrayList());
{code}
* WRT the dependency, I'd like rather call task pair predecessor and successor
instead of parent and child, because it's a dag not a tree. Moreover, how about
defining public void populateFileDeletionTaskDependency(List,
FileDeletion), and populateFileDeletionTaskDependency(List,
List) wraps it. In fact, the former one seems to be enough for
the patch.
{code}
+ public void populateFileDeletionTaskDependency(List
parentTasks,
+ List childDependentTasks) {
{code}
* How about calling it delete as well, just overloading the method?
{code}
+ public void deleteHelper(FileDeletion fileDeletion) {
{code}
* Please use LocalResource.newInstance instead.
{code}
+LocalResource localResource = Records.newRecord(LocalResource.class);
{code}
* There's one finbugs warning to fix.
* In the following method
{code}
+public boolean matches(Object o) {
{code}
How about refactoring the code in the following pattern, which should be more
clear.
{code}
if (obj1 == null && obj2 != null) {
return false;
} else if (obj1 != null && obj2 == null) {
return false;
} else if (obj1 != null && obj2 != null) {
// your logic
}
{code}
> NM fails to cleanup local directories for users
> ---
>
> Key: YARN-661
> URL: https://issues.apache.org/jira/browse/YARN-661
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 2.1.0-beta, 0.23.8
>Reporter: Jason Lowe
>Assignee: Omkar Vinit Joshi
> Attachments: YARN-661-20130701.patch, YARN-661-20130708.patch
>
>
> YARN-71 added deletion of local directories on startup, but in practice it
> fails to delete the directories because of permission problems. The
> top-level usercache directory is owned by the user but is in a directory that
> is not writable by the user. Therefore the deletion of the user's usercache
> directory, as the user, fails due to lack of permissions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-521) Augment AM - RM client module to be able to request containers only at specific locations
[
https://issues.apache.org/jira/browse/YARN-521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705283#comment-13705283
]
Bikas Saha commented on YARN-521:
-
Shouldnt this also check that the inferred rack1 has relaxLocality set to false?
{code}
+ContainerRequest bothLevelRequest =
+new ContainerRequest(capability, new String[] {"host3", "host4"},
+new String[] {NetworkTopology.DEFAULT_RACK, "/otherrack"},
+Priority.newInstance(3), 4, false);
+client.addContainerRequest(bothLevelRequest);
+
+verifyResourceRequest(client, bothLevelRequest, ResourceRequest.ANY,
false);
+verifyResourceRequest(client, bothLevelRequest,
NetworkTopology.DEFAULT_RACK,
+true);
+verifyResourceRequest(client, bothLevelRequest, "/otherrack",
+true);
+verifyResourceRequest(client, bothLevelRequest, "host3", true);
+verifyResourceRequest(client, bothLevelRequest, "host4", true);
+
{code}
The RM allows relaxLocality to be re-enabled on a location and priority. This
allows users to change their minds about strict locality if they are unable to
get those containers for a long time. The logic in the patch does not allow
this to happen. In fact testDifferentLocalityRelaxationSamePriority() shows
that it is not possible to do this with the current patch.
testDifferentLocalityRelaxationSamePriority() looks to have been written to
test for the case where host1 is specific resulting in rack1 to be false. Then
host3 in rack1 cannot be made non-specific since the flag on rack1 will be
inconsistent. It would be more clear if the second request had a different host.
testLocalityRelaxationDifferentLevels() tests that specific host cannot be
mixed with non-specific rack for the rack of that host. Can specific host be
mixed with specific rack for the same rack as the host? Probably not.
Can specific node host1 be mixed with specific rack on a different rack
(rack2)? If yes, if a container is allocated on host2 in rack2, then what
prevents getMachingRequests(*) from returning the requests for host1 and rack2
instead of only the request for rack2?
Now that we have a client and server pieces done, has this been tested with a
real cluster to see that it works for in practice?
It would really help if we had a clear document (even javadoc) that clarifies
what is legal and what is not. This will really help in code review, it will
make sure we have better test coverage and in general will help users
understand how to use the API. I am concerned that we are implementing
something pretty subtle and we need to get it right logically before
implementing the code. I havent even started thinking about how this interacts
with the blacklisting logic that we have added to the RM. How specific requests
interact with blacklist on the RM. And how both interact on the AMRMClient. And
how both interact across RM and AMRMClient.
Minor nits.
How about "inferredRacks" instead of "filledInRacks"?
How about JAVA standard library Collections.singletonList() instead of invoking
google common Lists.newArrayList?
> Augment AM - RM client module to be able to request containers only at
> specific locations
> -
>
> Key: YARN-521
> URL: https://issues.apache.org/jira/browse/YARN-521
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api
>Affects Versions: 2.0.3-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: YARN-521-1.patch, YARN-521-2.patch, YARN-521-2.patch,
> YARN-521.patch
>
>
> When YARN-392 and YARN-398 are completed, it would be good for AMRMClient to
> offer an easy way to access their functionality
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-569) CapacityScheduler: support for preemption (using a capacity monitor)
[ https://issues.apache.org/jira/browse/YARN-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated YARN-569: --- Attachment: YARN-569.11.patch Rebase. > CapacityScheduler: support for preemption (using a capacity monitor) > > > Key: YARN-569 > URL: https://issues.apache.org/jira/browse/YARN-569 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: 3queues.pdf, CapScheduler_with_preemption.pdf, > preemption.2.patch, YARN-569.10.patch, YARN-569.11.patch, YARN-569.1.patch, > YARN-569.2.patch, YARN-569.3.patch, YARN-569.4.patch, YARN-569.5.patch, > YARN-569.6.patch, YARN-569.8.patch, YARN-569.9.patch, YARN-569.patch, > YARN-569.patch > > > There is a tension between the fast-pace reactive role of the > CapacityScheduler, which needs to respond quickly to > applications resource requests, and node updates, and the more introspective, > time-based considerations > needed to observe and correct for capacity balance. To this purpose we opted > instead of hacking the delicate > mechanisms of the CapacityScheduler directly to add support for preemption by > means of a "Capacity Monitor", > which can be run optionally as a separate service (much like the > NMLivelinessMonitor). > The capacity monitor (similarly to equivalent functionalities in the fairness > scheduler) operates running on intervals > (e.g., every 3 seconds), observe the state of the assignment of resources to > queues from the capacity scheduler, > performs off-line computation to determine if preemption is needed, and how > best to "edit" the current schedule to > improve capacity, and generates events that produce four possible actions: > # Container de-reservations > # Resource-based preemptions > # Container-based preemptions > # Container killing > The actions listed above are progressively more costly, and it is up to the > policy to use them as desired to achieve the rebalancing goals. > Note that due to the "lag" in the effect of these actions the policy should > operate at the macroscopic level (e.g., preempt tens of containers > from a queue) and not trying to tightly and consistently micromanage > container allocations. > - Preemption policy (ProportionalCapacityPreemptionPolicy): > - > Preemption policies are by design pluggable, in the following we present an > initial policy (ProportionalCapacityPreemptionPolicy) we have been > experimenting with. The ProportionalCapacityPreemptionPolicy behaves as > follows: > # it gathers from the scheduler the state of the queues, in particular, their > current capacity, guaranteed capacity and pending requests (*) > # if there are pending requests from queues that are under capacity it > computes a new ideal balanced state (**) > # it computes the set of preemptions needed to repair the current schedule > and achieve capacity balance (accounting for natural completion rates, and > respecting bounds on the amount of preemption we allow for each round) > # it selects which applications to preempt from each over-capacity queue (the > last one in the FIFO order) > # it remove reservations from the most recently assigned app until the amount > of resource to reclaim is obtained, or until no more reservations exits > # (if not enough) it issues preemptions for containers from the same > applications (reverse chronological order, last assigned container first) > again until necessary or until no containers except the AM container are left, > # (if not enough) it moves onto unreserve and preempt from the next > application. > # containers that have been asked to preempt are tracked across executions. > If a containers is among the one to be preempted for more than a certain > time, the container is moved in a the list of containers to be forcibly > killed. > Notes: > (*) at the moment, in order to avoid double-counting of the requests, we only > look at the "ANY" part of pending resource requests, which means we might not > preempt on behalf of AMs that ask only for specific locations but not any. > (**) The ideal balance state is one in which each queue has at least its > guaranteed capacity, and the spare capacity is distributed among queues (that > wants some) as a weighted fair share. Where the weighting is based on the > guaranteed capacity of a queue, and the function runs to a fix point. > Tunables of the ProportionalCapacityPreemptionPolicy: > # observe-only mode (i.e., log the actions it would take, but behave as > read-only) > # how frequently to run the policy > # how long to wait between preemption and kill of a container > # which fraction of the containers I would like to
[jira] [Updated] (YARN-883) Expose Fair Scheduler-specific queue metrics
[ https://issues.apache.org/jira/browse/YARN-883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated YARN-883: Fix Version/s: (was: 2.2.0) 2.1.0-beta > Expose Fair Scheduler-specific queue metrics > > > Key: YARN-883 > URL: https://issues.apache.org/jira/browse/YARN-883 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.5-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.1.0-beta > > Attachments: YARN-883-1.patch, YARN-883-1.patch, YARN-883.patch > > > When the Fair Scheduler is enabled, QueueMetrics should include fair share, > minimum share, and maximum share. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-866) Add test for class ResourceWeights
[ https://issues.apache.org/jira/browse/YARN-866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated YARN-866: Fix Version/s: (was: 2.2.0) 2.1.0-beta > Add test for class ResourceWeights > -- > > Key: YARN-866 > URL: https://issues.apache.org/jira/browse/YARN-866 > Project: Hadoop YARN > Issue Type: Test >Affects Versions: 2.1.0-beta >Reporter: Wei Yan >Assignee: Wei Yan > Fix For: 2.1.0-beta > > Attachments: Yarn-866.patch, Yarn-866.patch, YARN-866.patch > > > Add test case for the class ResourceWeights -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-736) Add a multi-resource fair sharing metric
[ https://issues.apache.org/jira/browse/YARN-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated YARN-736: Fix Version/s: (was: 2.2.0) 2.1.0-beta > Add a multi-resource fair sharing metric > > > Key: YARN-736 > URL: https://issues.apache.org/jira/browse/YARN-736 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.1.0-beta > > Attachments: YARN-736-1.patch, YARN-736-2.patch, YARN-736-3.patch, > YARN-736-4.patch, YARN-736.patch > > > Currently, at a regular interval, the fair scheduler computes a fair memory > share for each queue and application inside it. This fair share is not used > for scheduling decisions, but is displayed in the web UI, exposed as a > metric, and used for preemption decisions. > With DRF and multi-resource scheduling, assigning a memory share as the fair > share metric to every queue no longer makes sense. It's not obvious what the > replacement should be, but probably something like fractional fairness within > a queue, or distance from an ideal cluster state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-914) Support graceful decommission of nodemanager
Luke Lu created YARN-914: Summary: Support graceful decommission of nodemanager Key: YARN-914 URL: https://issues.apache.org/jira/browse/YARN-914 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Luke Lu Assignee: Junping Du When NNs are decommissioned for non-fault reasons (capacity change etc.), it's desirable to minimize the impact to running applications. Currently if a NN is decommissioned, all running containers on the NN need to be rescheduled on other NNs. Further more, for finished map tasks, if their map output are not fetched by the reducers of the job, these map tasks will need to be rerun as well. We propose to introduce a mechanism to optionally gracefully decommission a node manager. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-389) Infinitely assigning containers when the required resource exceeds the cluster's absolute capacity
[ https://issues.apache.org/jira/browse/YARN-389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi reassigned YARN-389: -- Assignee: Omkar Vinit Joshi (was: Zhijie Shen) > Infinitely assigning containers when the required resource exceeds the > cluster's absolute capacity > -- > > Key: YARN-389 > URL: https://issues.apache.org/jira/browse/YARN-389 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Omkar Vinit Joshi > > I've run wordcount example on branch-2 and trunk. I've set > yarn.nodemanager.resource.memory-mb to 1G and > yarn.app.mapreduce.am.resource.mb to 1.5G. Therefore, resourcemanager is to > assign a 2G AM container for AM. However, the nodemanager doesn't have enough > memory to assign the container. The problem is that the assignment operation > will be repeated infinitely, if the assignment cannot be accomplished. Logs > follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-763) AMRMClientAsync should stop heartbeating after receiving shutdown from RM
[ https://issues.apache.org/jira/browse/YARN-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705144#comment-13705144 ] Bikas Saha commented on YARN-763: - we also need to remove the check in serviceStop() that it is being called on the callback handler thread. not needed anymore since we dont join() on that thread. we can add a test that we can call asyncClient.stop() on same thread as callback from asyncClient. > AMRMClientAsync should stop heartbeating after receiving shutdown from RM > - > > Key: YARN-763 > URL: https://issues.apache.org/jira/browse/YARN-763 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bikas Saha >Assignee: Xuan Gong > Attachments: YARN-763.1.patch, YARN-763.2.patch, YARN-763.3.patch, > YARN-763.4.patch, YARN-763.5.patch, YARN-763.6.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-763) AMRMClientAsync should stop heartbeating after receiving shutdown from RM
[
https://issues.apache.org/jira/browse/YARN-763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13705104#comment-13705104
]
Bikas Saha commented on YARN-763:
-
Changes mostly look good.
This assert is not 100% stable because the test code thread can come to this
point before the heartbeat thread manages to exit. These are all on different
threads that may lost the cpu at arbitrary times. We can remove it. I am not
too paranoid given that we are already checking for a single allocate call in
the mockClient. We can avoid adding this extra test only method to the main
class.
{code}
+Assert.assertFalse(((AMRMClientAsyncImpl) asyncClient)
+.isHeartbeatThreadAlive());
{code}
If we change the heartbeat thread sleep interval to 1ms and sleep 5ms after we
get the reboot notification in the test code, then we probably get 100% surety
in less time overall compared to the 20ms heartbeat interval being used
currently.
bq. If we set the callback as daemon thread, calling join() on the heartBeater
thread will be fine.
I dont think thats correct. join() will still get stuck. So we should continue
to interrupt the callback thread but not call join() on it. The original patch
for Async client did not have a join() in it. My bad in having it added.
> AMRMClientAsync should stop heartbeating after receiving shutdown from RM
> -
>
> Key: YARN-763
> URL: https://issues.apache.org/jira/browse/YARN-763
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Bikas Saha
>Assignee: Xuan Gong
> Attachments: YARN-763.1.patch, YARN-763.2.patch, YARN-763.3.patch,
> YARN-763.4.patch, YARN-763.5.patch, YARN-763.6.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-523) Container localization failures aren't reported from NM to RM
[ https://issues.apache.org/jira/browse/YARN-523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi reassigned YARN-523: -- Assignee: Jian He (was: Omkar Vinit Joshi) > Container localization failures aren't reported from NM to RM > - > > Key: YARN-523 > URL: https://issues.apache.org/jira/browse/YARN-523 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Jian He > Attachments: YARN-523.patch > > > This is mainly a pain on crashing AMs, but once we fix this, containers also > can benefit - same fix for both. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-541) getAllocatedContainers() is not returning all the allocated containers
[
https://issues.apache.org/jira/browse/YARN-541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704935#comment-13704935
]
Omkar Vinit Joshi commented on YARN-541:
[~write2kishore] I just took a look at nm logs and I can see that
"container_1366096597608_0001_01_06" container was allocated by RM and AM
made a start container request for it on NM. I think there is some problem in
the AM logs. Can you take a look at your AM code again? Looks like something is
getting missed there.. If it is still occurring then can you print the logs
when AM makes a start container request to NM?? probably something is getting
missed there..
{code}
2013-04-16 03:29:57,681 INFO [IPC Server handler 4 on 34660]
containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:startContainer(402)) - Start request for
container_1366096597608_0001_01_06 by user dsadm
2013-04-16 03:29:57,684 INFO [IPC Server handler 4 on 34660]
nodemanager.NMAuditLogger (NMAuditLogger.java:logSuccess(89)) - USER=dsadm
IP=127.0.1.1OPERATION=Start Container Request
TARGET=ContainerManageImpl RESULT=SUCCESS
APPID=application_1366096597608_0001
CONTAINERID=container_1366096597608_0001_01_06
2013-04-16 03:29:57,687 INFO [AsyncDispatcher event handler]
application.Application (ApplicationImpl.java:transition(255)) - Adding
container_1366096597608_0001_01_06 to application
application_1366096597608_0001
2013-04-16 03:29:57,689 INFO [AsyncDispatcher event handler]
container.Container (ContainerImpl.java:handle(835)) - Container
container_1366096597608_0001_01_06 transitioned from NEW to LOCALIZED
2013-04-16 03:29:57,952 INFO [AsyncDispatcher event handler]
container.Container (ContainerImpl.java:handle(835)) - Container
container_1366096597608_0001_01_06 transitioned from LOCALIZED to RUNNING
2013-04-16 03:29:58,475 INFO [Node Status Updater]
nodemanager.NodeStatusUpdaterImpl
(NodeStatusUpdaterImpl.java:getNodeStatus(249)) - Sending out status for
container: container_id {, app_attempt_id {, application_id {, id: 1,
cluster_timestamp: 1366096597608, }, attemptId: 1, }, id: 1, }, state:
C_RUNNING, diagnostics: "", exit_status: -1000,
2013-04-16 03:29:58,478 INFO [Node Status Updater]
nodemanager.NodeStatusUpdaterImpl
(NodeStatusUpdaterImpl.java:getNodeStatus(249)) - Sending out status for
container: container_id {, app_attempt_id {, application_id {, id: 1,
cluster_timestamp: 1366096597608, }, attemptId: 1, }, id: 5, }, state:
C_RUNNING, diagnostics: "", exit_status: -1000,
2013-04-16 03:29:58,481 INFO [Node Status Updater]
nodemanager.NodeStatusUpdaterImpl
(NodeStatusUpdaterImpl.java:getNodeStatus(249)) - Sending out status for
container: container_id {, app_attempt_id {, application_id {, id: 1,
cluster_timestamp: 1366096597608, }, attemptId: 1, }, id: 6, }, state:
C_RUNNING, diagnostics: "", exit_status: -1000,
2013-04-16 03:29:58,489 INFO [ContainersLauncher #2]
nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:launchContainer(113)) - launchContainer: [bash,
/tmp/nm-local-dir/usercache/dsadm/appcache/application_1366096597608_0001/container_1366096597608_0001_01_06/default_container_executor.sh]
2013-04-16 03:29:58,638 INFO [ContainersLauncher #1] launcher.ContainerLaunch
(ContainerLaunch.java:call(282)) - Container
container_1366096597608_0001_01_05 succeeded
2013-04-16 03:29:58,639 INFO [ContainersLauncher #2] launcher.ContainerLaunch
(ContainerLaunch.java:call(282)) - Container
container_1366096597608_0001_01_06 succeeded
2013-04-16 03:29:58,643 INFO [AsyncDispatcher event handler]
container.Container (ContainerImpl.java:handle(835)) - Container
container_1366096597608_0001_01_05 transitioned from RUNNING to
EXITED_WITH_SUCCESS
2013-04-16 03:29:58,644 INFO [AsyncDispatcher event handler]
container.Container (ContainerImpl.java:handle(835)) - Container
container_1366096597608_0001_01_06 transitioned from RUNNING to
EXITED_WITH_SUCCESS
2013-04-16 03:29:58,644 INFO [AsyncDispatcher event handler]
launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(300)) -
Cleaning up container container_1366096597608_0001_01_05
2013-04-16 03:29:58,693 INFO [AsyncDispatcher event handler]
launcher.ContainerLaunch (ContainerLaunch.java:cleanupContainer(300)) -
Cleaning up container container_1366096597608_0001_01_06
{code}
> getAllocatedContainers() is not returning all the allocated containers
> --
>
> Key: YARN-541
> URL: https://issues.apache.org/jira/browse/YARN-541
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.0.3-alpha
> Environment: Redhat Linux 64-bit
>Reporter: Krishna Kishore Bonag
[jira] [Assigned] (YARN-541) getAllocatedContainers() is not returning all the allocated containers
[ https://issues.apache.org/jira/browse/YARN-541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi reassigned YARN-541: -- Assignee: Omkar Vinit Joshi > getAllocatedContainers() is not returning all the allocated containers > -- > > Key: YARN-541 > URL: https://issues.apache.org/jira/browse/YARN-541 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.0.3-alpha > Environment: Redhat Linux 64-bit >Reporter: Krishna Kishore Bonagiri >Assignee: Omkar Vinit Joshi > Attachments: AppMaster.stdout, yarn-dsadm-nodemanager-isredeng.out, > yarn-dsadm-resourcemanager-isredeng.out > > > I am running an application that was written and working well with the > hadoop-2.0.0-alpha but when I am running the same against 2.0.3-alpha, the > getAllocatedContainers() method called on AMResponse is not returning all the > containers allocated sometimes. For example, I request for 10 containers and > this method gives me only 9 containers sometimes, and when I looked at the > log of Resource Manager, the 10th container is also allocated. It happens > only sometimes randomly and works fine all other times. If I send one more > request for the remaining container to RM after it failed to give them the > first time(and before releasing already acquired ones), it could allocate > that container. I am running only one application at a time, but 1000s of > them one after another. > My main worry is, even though the RM's log is saying that all 10 requested > containers are allocated, the getAllocatedContainers() method is not > returning me all of them, it returned only 9 surprisingly. I never saw this > kind of issue in the previous version, i.e. hadoop-2.0.0-alpha. > Thanks, > Kishore > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-913) Add a way to register long-lived services in a YARN cluster
[ https://issues.apache.org/jira/browse/YARN-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704925#comment-13704925 ] Steve Loughran commented on YARN-913: - You can list running apps today, but # anyone can start >1 app with the same name # The AM can only register a URL for the AM, and its own RPC service -not any other RPC ports opened, web pages served, etc. At the very least, I'd like to be able to launch an AM with a given app-type & app-name and be confident that no other YARN application with the same (type, name) was running or being started in my name. With the guarantee of a unique (user, type, name) I could use the RM as a bit of a service registry, albeit one limited in what it can register. > Add a way to register long-lived services in a YARN cluster > --- > > Key: YARN-913 > URL: https://issues.apache.org/jira/browse/YARN-913 > Project: Hadoop YARN > Issue Type: New Feature > Components: api >Affects Versions: 3.0.0 >Reporter: Steve Loughran > > In a YARN cluster you can't predict where services will come up -or on what > ports. The services need to work those things out as they come up and then > publish them somewhere. > Applications need to be able to find the service instance they are to bond to > -and not any others in the cluster. > Some kind of service registry -in the RM, in ZK, could do this. If the RM > held the write access to the ZK nodes, it would be more secure than having > apps register with ZK themselves. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-913) Add a way to register long-lived services in a YARN cluster
Steve Loughran created YARN-913: --- Summary: Add a way to register long-lived services in a YARN cluster Key: YARN-913 URL: https://issues.apache.org/jira/browse/YARN-913 Project: Hadoop YARN Issue Type: New Feature Components: api Affects Versions: 3.0.0 Reporter: Steve Loughran In a YARN cluster you can't predict where services will come up -or on what ports. The services need to work those things out as they come up and then publish them somewhere. Applications need to be able to find the service instance they are to bond to -and not any others in the cluster. Some kind of service registry -in the RM, in ZK, could do this. If the RM held the write access to the ZK nodes, it would be more secure than having apps register with ZK themselves. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-523) Container localization failures aren't reported from NM to RM
[
https://issues.apache.org/jira/browse/YARN-523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704910#comment-13704910
]
Jian He commented on YARN-523:
--
tested on single node cluster to throw exception("") in the
FSDownload.
I can see error message from command line as follows:
{code}
Job job_1373480985367_0001 failed with state FAILED due to:
Application application_1373480985367_0001 failed 10 times due to AM Container
for appattempt_1373480985367_0001_10 exited with exitCode: -1000 due to:
111
{code}
And error message from web UI:
{code}
Application application_1373480985367_0001 failed 10 times due to AM Container
for appattempt_1373480985367_0001_10 exited with exitCode: -1000 due to:
111
.Failing this attempt.. Failing the application.
{code}
> Container localization failures aren't reported from NM to RM
> -
>
> Key: YARN-523
> URL: https://issues.apache.org/jira/browse/YARN-523
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Omkar Vinit Joshi
> Attachments: YARN-523.patch
>
>
> This is mainly a pain on crashing AMs, but once we fix this, containers also
> can benefit - same fix for both.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-149) ResourceManager (RM) High-Availability (HA)
[ https://issues.apache.org/jira/browse/YARN-149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated YARN-149: -- Attachment: rm-ha-phase1-draft2.pdf Uploading draft-2 with more details on the wrapper approach. > ResourceManager (RM) High-Availability (HA) > --- > > Key: YARN-149 > URL: https://issues.apache.org/jira/browse/YARN-149 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Harsh J >Assignee: Bikas Saha > Attachments: rm-ha-phase1-approach-draft1.pdf, rm-ha-phase1-draft2.pdf > > > This jira tracks work needed to be done to support one RM instance failing > over to another RM instance so that we can have RM HA. Work includes leader > election, transfer of control to leader and client re-direction to new leader. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-890) The roundup for memory values on resource manager UI is misleading
[ https://issues.apache.org/jira/browse/YARN-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tassapol Athiapinya updated YARN-890: - Attachment: Screen Shot 2013-07-10 at 10.43.34 AM.png Attached screenshot. Memory showed in the UI is 5 GB. In fact, the memory given in configuration is little above 4 GB. That is why original reporter says rounding up is misleading. It should be better to round it to 4 GB. > The roundup for memory values on resource manager UI is misleading > -- > > Key: YARN-890 > URL: https://issues.apache.org/jira/browse/YARN-890 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Trupti Dhavle > Attachments: Screen Shot 2013-07-10 at 10.43.34 AM.png > > > From the yarn-site.xml, I see following values- > > yarn.nodemanager.resource.memory-mb > 4192 > > > yarn.scheduler.maximum-allocation-mb > 4192 > > > yarn.scheduler.minimum-allocation-mb > 1024 > > However the resourcemanager UI shows total memory as 5MB -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-649) Make container logs available over HTTP in plain text
[ https://issues.apache.org/jira/browse/YARN-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704804#comment-13704804 ] Alejandro Abdelnur commented on YARN-649: - [~vinodkv], just noticed i've missed your 0:45 comment, i've just seen the 0:47. [~sandyr], please take adress those in a new patch. > Make container logs available over HTTP in plain text > - > > Key: YARN-649 > URL: https://issues.apache.org/jira/browse/YARN-649 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-649-2.patch, YARN-649-3.patch, YARN-649-4.patch, > YARN-649-5.patch, YARN-649.patch, YARN-752-1.patch > > > It would be good to make container logs available over the REST API for > MAPREDUCE-4362 and so that they can be accessed programatically in general. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-649) Make container logs available over HTTP in plain text
[
https://issues.apache.org/jira/browse/YARN-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704787#comment-13704787
]
Alejandro Abdelnur commented on YARN-649:
-
The streaming is done using an StreamingOutput instance in the NMWebServices,
that is a javax.ws interface. The buffer is defined there. While we are at
this, [~sandyr], would you lower the buffer size to 8K, no need to use 64K. if
the underlying streams impls have larger buffers the only additional cost is
extra looping.
On YARN-911, you are right, i've missed that {{pre._(new String(cbuf, 0,
len));}} is actually streaming out. closing YARN-911 as invalid.
[~sandyr], would you address [~vinodkv] concerns?
> Make container logs available over HTTP in plain text
> -
>
> Key: YARN-649
> URL: https://issues.apache.org/jira/browse/YARN-649
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Affects Versions: 2.0.4-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: YARN-649-2.patch, YARN-649-3.patch, YARN-649-4.patch,
> YARN-649-5.patch, YARN-649.patch, YARN-752-1.patch
>
>
> It would be good to make container logs available over the REST API for
> MAPREDUCE-4362 and so that they can be accessed programatically in general.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (YARN-911) NM web ui log page loads log file fully in memory to create the html
[ https://issues.apache.org/jira/browse/YARN-911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur resolved YARN-911. - Resolution: Invalid > NM web ui log page loads log file fully in memory to create the html > > > Key: YARN-911 > URL: https://issues.apache.org/jira/browse/YARN-911 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.0.5-alpha >Reporter: Alejandro Abdelnur > > this could bloat the memory of the NM, we should either stream, paginate or > cap the log size, it depends what the web ui framework allows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-879) Fix NPE in test/o.a.h.y.server.resourcemanager.Application.getResources()
[ https://issues.apache.org/jira/browse/YARN-879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-879: - Summary: Fix NPE in test/o.a.h.y.server.resourcemanager.Application.getResources() (was: Fix NPE in Application.getResources()) > Fix NPE in test/o.a.h.y.server.resourcemanager.Application.getResources() > - > > Key: YARN-879 > URL: https://issues.apache.org/jira/browse/YARN-879 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-879.patch > > > getResources() will return a list of containers that allocated by RM. > However, it is now return null directly. The worse thing is: if LOG.debug is > enabled, then it will definitely cause NPE exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-865) RM webservices can't query on application Types
[
https://issues.apache.org/jira/browse/YARN-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704780#comment-13704780
]
Hadoop QA commented on YARN-865:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591673/YARN-865.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1451//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1451//console
This message is automatically generated.
> RM webservices can't query on application Types
> ---
>
> Key: YARN-865
> URL: https://issues.apache.org/jira/browse/YARN-865
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: MR-5337.1.patch, YARN-865.1.patch
>
>
> The resource manager web service api to get the list of apps doesn't have a
> query parameter for appTypes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-369) Handle ( or throw a proper error when receiving) status updates from application masters that have not registered
[ https://issues.apache.org/jira/browse/YARN-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704777#comment-13704777 ] Bikas Saha commented on YARN-369: - Thats right. I found other exceptions too that are probably not visible right now to apps. e.g. InvalidResourceRequestException, InvalidResourceBlacklistRequestException etc. Opened YARN-912 to track this. [~mayank_bansal] I have tentatively assigned this to you as a follow up to this comment. Would be great if you could move these and other user facing exceptions in YARN-912. Hopefully, it will be a simple eclipse refactor and we can get this in fast. > Handle ( or throw a proper error when receiving) status updates from > application masters that have not registered > - > > Key: YARN-369 > URL: https://issues.apache.org/jira/browse/YARN-369 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.0.3-alpha, trunk-win >Reporter: Hitesh Shah >Assignee: Mayank Bansal > Fix For: 2.1.0-beta > > Attachments: YARN-369.patch, YARN-369-trunk-1.patch, > YARN-369-trunk-2.patch, YARN-369-trunk-3.patch, YARN-369-trunk-4.patch > > > Currently, an allocate call from an unregistered application is allowed and > the status update for it throws a statemachine error that is silently dropped. > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > STATUS_UPDATE at LAUNCHED >at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) >at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43) >at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445) >at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:588) >at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:99) >at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:471) >at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:452) >at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130) >at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77) >at java.lang.Thread.run(Thread.java:680) > ApplicationMasterService should likely throw an appropriate error for > applications' requests that should not be handled in such cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-865) RM webservices can't query on application Types
[ https://issues.apache.org/jira/browse/YARN-865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704773#comment-13704773 ] Zhijie Shen commented on YARN-865: -- The patch looks good to me, but I have two minor comments: 1. In the test, is it better to add the case of empty type string? 2. the application *Type* of the application, should it be lowercase? > RM webservices can't query on application Types > --- > > Key: YARN-865 > URL: https://issues.apache.org/jira/browse/YARN-865 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: MR-5337.1.patch, YARN-865.1.patch > > > The resource manager web service api to get the list of apps doesn't have a > query parameter for appTypes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-912) Create exceptions package in common/api for yarn and move client facing exceptions to them
Bikas Saha created YARN-912: --- Summary: Create exceptions package in common/api for yarn and move client facing exceptions to them Key: YARN-912 URL: https://issues.apache.org/jira/browse/YARN-912 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Bikas Saha Assignee: Mayank Bansal Exceptions like InvalidResourceBlacklistRequestException, InvalidResourceRequestException, InvalidApplicationMasterRequestException etc are currently inside ResourceManager and not visible to clients. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-368) Fix typo "defiend" should be "defined" in error output
[ https://issues.apache.org/jira/browse/YARN-368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704767#comment-13704767 ] Hudson commented on YARN-368: - Integrated in Hadoop-trunk-Commit #4060 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4060/]) YARN-368. Fixed a typo in error message in Auxiliary services. Contributed by Albert Chu. (Revision 1501852) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501852 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/AuxServices.java > Fix typo "defiend" should be "defined" in error output > -- > > Key: YARN-368 > URL: https://issues.apache.org/jira/browse/YARN-368 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Albert Chu >Assignee: Albert Chu >Priority: Trivial > Fix For: 2.1.1-beta > > Attachments: YARN-368.patch > > > Noticed the following in an error log output while doing some experiements > ./1066018/nodes/hyperion987/log/yarn-achu-nodemanager-hyperion987.out:java.lang.RuntimeException: > No class defiend for uda.shuffle > "defiend" should be "defined" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-295) Resource Manager throws InvalidStateTransitonException: Invalid event: CONTAINER_FINISHED at ALLOCATED for RMAppAttemptImpl
[
https://issues.apache.org/jira/browse/YARN-295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704768#comment-13704768
]
Hudson commented on YARN-295:
-
Integrated in Hadoop-trunk-Commit #4060 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4060/])
YARN-295. Fixed a race condition in ResourceManager RMAppAttempt state
machine. Contributed by Mayank Bansal. (Revision 1501856)
Result = SUCCESS
vinodkv :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501856
Files :
* /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
*
/hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java
> Resource Manager throws InvalidStateTransitonException: Invalid event:
> CONTAINER_FINISHED at ALLOCATED for RMAppAttemptImpl
> ---
>
> Key: YARN-295
> URL: https://issues.apache.org/jira/browse/YARN-295
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.0.2-alpha, 2.0.1-alpha
>Reporter: Devaraj K
>Assignee: Mayank Bansal
> Fix For: 2.1.1-beta
>
> Attachments: YARN-295-trunk-1.patch, YARN-295-trunk-2.patch,
> YARN-295-trunk-3.patch
>
>
> {code:xml}
> 2012-12-28 14:03:56,956 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> CONTAINER_FINISHED at ALLOCATED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:490)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:80)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:433)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:414)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-701) ApplicationTokens should be used irrespective of kerberos
[
https://issues.apache.org/jira/browse/YARN-701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704765#comment-13704765
]
Hadoop QA commented on YARN-701:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591671/YARN-701-20130710.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests:
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1450//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1450//console
This message is automatically generated.
> ApplicationTokens should be used irrespective of kerberos
> -
>
> Key: YARN-701
> URL: https://issues.apache.org/jira/browse/YARN-701
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.1.0-beta
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Attachments: YARN-701-20130520.txt, YARN-701-20130709.3.txt,
> YARN-701-20130710.txt
>
>
> - Single code path for secure and non-secure cases is useful for testing,
> coverage.
> - Having this in non-secure mode will help us avoid accidental bugs in AMs
> DDos'ing and bringing down RM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-295) Resource Manager throws InvalidStateTransitonException: Invalid event: CONTAINER_FINISHED at ALLOCATED for RMAppAttemptImpl
[
https://issues.apache.org/jira/browse/YARN-295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704757#comment-13704757
]
Vinod Kumar Vavilapalli commented on YARN-295:
--
+1, this looks good. Checking it in.
> Resource Manager throws InvalidStateTransitonException: Invalid event:
> CONTAINER_FINISHED at ALLOCATED for RMAppAttemptImpl
> ---
>
> Key: YARN-295
> URL: https://issues.apache.org/jira/browse/YARN-295
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: 2.0.2-alpha, 2.0.1-alpha
>Reporter: Devaraj K
>Assignee: Mayank Bansal
> Attachments: YARN-295-trunk-1.patch, YARN-295-trunk-2.patch,
> YARN-295-trunk-3.patch
>
>
> {code:xml}
> 2012-12-28 14:03:56,956 ERROR
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> CONTAINER_FINISHED at ALLOCATED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:490)
> at
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:80)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:433)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:414)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-865) RM webservices can't query on application Types
[ https://issues.apache.org/jira/browse/YARN-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated YARN-865: --- Attachment: YARN-865.1.patch Recreate the patch based on the latest trunk > RM webservices can't query on application Types > --- > > Key: YARN-865 > URL: https://issues.apache.org/jira/browse/YARN-865 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: MR-5337.1.patch, YARN-865.1.patch > > > The resource manager web service api to get the list of apps doesn't have a > query parameter for appTypes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-368) Fix typo "defiend" should be "defined" in error output
[ https://issues.apache.org/jira/browse/YARN-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli reassigned YARN-368: Assignee: Albert Chu > Fix typo "defiend" should be "defined" in error output > -- > > Key: YARN-368 > URL: https://issues.apache.org/jira/browse/YARN-368 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Albert Chu >Assignee: Albert Chu >Priority: Trivial > Attachments: YARN-368.patch > > > Noticed the following in an error log output while doing some experiements > ./1066018/nodes/hyperion987/log/yarn-achu-nodemanager-hyperion987.out:java.lang.RuntimeException: > No class defiend for uda.shuffle > "defiend" should be "defined" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-369) Handle ( or throw a proper error when receiving) status updates from application masters that have not registered
[ https://issues.apache.org/jira/browse/YARN-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704737#comment-13704737 ] Vinod Kumar Vavilapalli commented on YARN-369: -- Shouldn't InvalidApplicationMasterRequestException be in api/common module so that it is visible to the AMs? > Handle ( or throw a proper error when receiving) status updates from > application masters that have not registered > - > > Key: YARN-369 > URL: https://issues.apache.org/jira/browse/YARN-369 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.0.3-alpha, trunk-win >Reporter: Hitesh Shah >Assignee: Mayank Bansal > Fix For: 2.1.0-beta > > Attachments: YARN-369.patch, YARN-369-trunk-1.patch, > YARN-369-trunk-2.patch, YARN-369-trunk-3.patch, YARN-369-trunk-4.patch > > > Currently, an allocate call from an unregistered application is allowed and > the status update for it throws a statemachine error that is silently dropped. > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > STATUS_UPDATE at LAUNCHED >at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) >at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43) >at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445) >at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:588) >at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:99) >at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:471) >at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:452) >at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130) >at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77) >at java.lang.Thread.run(Thread.java:680) > ApplicationMasterService should likely throw an appropriate error for > applications' requests that should not be handled in such cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-649) Make container logs available over HTTP in plain text
[ https://issues.apache.org/jira/browse/YARN-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704730#comment-13704730 ] Vinod Kumar Vavilapalli commented on YARN-649: -- bq. for the new HTTP service returning logs, a 64K buffer is used to stream the logs, so there things are bounded. Interesting, this is in common? bq. For the existing web ui page that display the logs, that is a different story, the current implementation is loading the complete log in memory, creating the page, and returning it. The later should be fix to do streaming similar to the new HTTP plain/text, but that is another JIRA (just created YARN-911) I checked this before itself, we aren't loading it in memory. I see this "pre._(new String(cbuf, 0, len));" which eventually is doing a println, and depending on the HttpServer buffer size, it will be flushed sometime. Is that correct? bq. Are we good then with the latest patch here? No. I pointed out atleast one possible bug and few nits that still need to be fixed. > Make container logs available over HTTP in plain text > - > > Key: YARN-649 > URL: https://issues.apache.org/jira/browse/YARN-649 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-649-2.patch, YARN-649-3.patch, YARN-649-4.patch, > YARN-649-5.patch, YARN-649.patch, YARN-752-1.patch > > > It would be good to make container logs available over the REST API for > MAPREDUCE-4362 and so that they can be accessed programatically in general. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-701) ApplicationTokens should be used irrespective of kerberos
[ https://issues.apache.org/jira/browse/YARN-701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-701: - Attachment: YARN-701-20130710.txt Same patch updated against latest trunk. > ApplicationTokens should be used irrespective of kerberos > - > > Key: YARN-701 > URL: https://issues.apache.org/jira/browse/YARN-701 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.1.0-beta >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: YARN-701-20130520.txt, YARN-701-20130709.3.txt, > YARN-701-20130710.txt > > > - Single code path for secure and non-secure cases is useful for testing, > coverage. > - Having this in non-secure mode will help us avoid accidental bugs in AMs > DDos'ing and bringing down RM. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-649) Make container logs available over HTTP in plain text
[ https://issues.apache.org/jira/browse/YARN-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704639#comment-13704639 ] Alejandro Abdelnur commented on YARN-649: - [~vinodkv], for the new HTTP service returning logs, a 64K buffer is used to stream the logs, so there things are bounded. For the existing web ui page that display the logs, that is a different story, the current implementation is loading the complete log in memory, creating the page, and returning it. The later should be fix to do streaming similar to the new HTTP plain/text, but that is another JIRA (just created YARN-911), here things are moving around a bit in the web ui log service just to reuse certain logic. Are we good then with the latest patch here? > Make container logs available over HTTP in plain text > - > > Key: YARN-649 > URL: https://issues.apache.org/jira/browse/YARN-649 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-649-2.patch, YARN-649-3.patch, YARN-649-4.patch, > YARN-649-5.patch, YARN-649.patch, YARN-752-1.patch > > > It would be good to make container logs available over the REST API for > MAPREDUCE-4362 and so that they can be accessed programatically in general. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-911) NM web ui log page loads log file fully in memory to create the html
Alejandro Abdelnur created YARN-911: --- Summary: NM web ui log page loads log file fully in memory to create the html Key: YARN-911 URL: https://issues.apache.org/jira/browse/YARN-911 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 2.0.5-alpha Reporter: Alejandro Abdelnur this could bloat the memory of the NM, we should either stream, paginate or cap the log size, it depends what the web ui framework allows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-896) Roll up for long lived YARN
[ https://issues.apache.org/jira/browse/YARN-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704623#comment-13704623 ] Robert Joseph Evans commented on YARN-896: -- Chris, Yes I missed the app master retry issue. Those two with the discussion on them seem to cover what we are looking for. > Roll up for long lived YARN > --- > > Key: YARN-896 > URL: https://issues.apache.org/jira/browse/YARN-896 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Robert Joseph Evans > > YARN is intended to be general purpose, but it is missing some features to be > able to truly support long lived applications and long lived containers. > This ticket is intended to > # discuss what is needed to support long lived processes > # track the resulting JIRA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-879) Fix NPE in Application.getResources()
[ https://issues.apache.org/jira/browse/YARN-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704583#comment-13704583 ] Junping Du commented on YARN-879: - The history may not show this if tests are commented since YARN-1 (I only check TestFifoScheduler which belongs to this case). Anyway, we can fix these tests or remove them completely. It doesn't make sense to me that some test code are commented for a long time as it just confused people like us. :) > Fix NPE in Application.getResources() > - > > Key: YARN-879 > URL: https://issues.apache.org/jira/browse/YARN-879 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0, 2.1.0-beta >Reporter: Junping Du >Assignee: Junping Du > Attachments: YARN-879.patch > > > getResources() will return a list of containers that allocated by RM. > However, it is now return null directly. The worse thing is: if LOG.debug is > enabled, then it will definitely cause NPE exception. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-369) Handle ( or throw a proper error when receiving) status updates from application masters that have not registered
[ https://issues.apache.org/jira/browse/YARN-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704541#comment-13704541 ] Hudson commented on YARN-369: - Integrated in Hadoop-Mapreduce-trunk #1483 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1483/]) YARN-369. Handle ( or throw a proper error when receiving) status updates from application masters that have not registered (Mayank Bansal & Abhishek Kapoor via bikas) (Revision 1501605) Result = FAILURE bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501605 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/InvalidApplicationMasterRequestException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java > Handle ( or throw a proper error when receiving) status updates from > application masters that have not registered > - > > Key: YARN-369 > URL: https://issues.apache.org/jira/browse/YARN-369 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.0.3-alpha, trunk-win >Reporter: Hitesh Shah >Assignee: Mayank Bansal > Fix For: 2.1.0-beta > > Attachments: YARN-369.patch, YARN-369-trunk-1.patch, > YARN-369-trunk-2.patch, YARN-369-trunk-3.patch, YARN-369-trunk-4.patch > > > Currently, an allocate call from an unregistered application is allowed and > the status update for it throws a statemachine error that is silently dropped. > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > STATUS_UPDATE at LAUNCHED >at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) >at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43) >at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445) >at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:588) >at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:99) >at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:471) >at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:452) >at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130) >at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77) >at java.lang.Thread.run(Thread.java:680) > ApplicationMasterService should likely throw an appropriate error for > applications' requests that should not be handled in such cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-727) ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter
[ https://issues.apache.org/jira/browse/YARN-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704545#comment-13704545 ] Hudson commented on YARN-727: - Integrated in Hadoop-Mapreduce-trunk #1483 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1483/]) YARN-727, MAPREDUCE-5325. ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter. Contributed by Xuan Gong. (Revision 1501599) Result = FAILURE hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501599 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetAllApplicationsRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetAllApplicationsResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/applicationclient_protocol.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/YarnCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ApplicationClientProtocolPBClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ApplicationClientProtocolPBServiceImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAllApplicationsRequestPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAllApplicationsResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationACLs.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java > ClientRMProtocol.getAllApplications should accept ApplicationTyp
[jira] [Commented] (YARN-727) ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter
[ https://issues.apache.org/jira/browse/YARN-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704532#comment-13704532 ] Hudson commented on YARN-727: - Integrated in Hadoop-Hdfs-trunk #1456 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1456/]) YARN-727, MAPREDUCE-5325. ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter. Contributed by Xuan Gong. (Revision 1501599) Result = FAILURE hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501599 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetAllApplicationsRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetAllApplicationsResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/applicationclient_protocol.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/YarnCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ApplicationClientProtocolPBClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ApplicationClientProtocolPBServiceImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAllApplicationsRequestPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAllApplicationsResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationACLs.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java > ClientRMProtocol.getAllApplications should accept ApplicationType as a >
[jira] [Commented] (YARN-369) Handle ( or throw a proper error when receiving) status updates from application masters that have not registered
[ https://issues.apache.org/jira/browse/YARN-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704528#comment-13704528 ] Hudson commented on YARN-369: - Integrated in Hadoop-Hdfs-trunk #1456 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1456/]) YARN-369. Handle ( or throw a proper error when receiving) status updates from application masters that have not registered (Mayank Bansal & Abhishek Kapoor via bikas) (Revision 1501605) Result = FAILURE bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501605 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/InvalidApplicationMasterRequestException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java > Handle ( or throw a proper error when receiving) status updates from > application masters that have not registered > - > > Key: YARN-369 > URL: https://issues.apache.org/jira/browse/YARN-369 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.0.3-alpha, trunk-win >Reporter: Hitesh Shah >Assignee: Mayank Bansal > Fix For: 2.1.0-beta > > Attachments: YARN-369.patch, YARN-369-trunk-1.patch, > YARN-369-trunk-2.patch, YARN-369-trunk-3.patch, YARN-369-trunk-4.patch > > > Currently, an allocate call from an unregistered application is allowed and > the status update for it throws a statemachine error that is silently dropped. > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > STATUS_UPDATE at LAUNCHED >at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) >at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43) >at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445) >at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:588) >at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:99) >at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:471) >at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:452) >at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130) >at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77) >at java.lang.Thread.run(Thread.java:680) > ApplicationMasterService should likely throw an appropriate error for > applications' requests that should not be handled in such cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-369) Handle ( or throw a proper error when receiving) status updates from application masters that have not registered
[ https://issues.apache.org/jira/browse/YARN-369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704419#comment-13704419 ] Hudson commented on YARN-369: - Integrated in Hadoop-Yarn-trunk #266 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/266/]) YARN-369. Handle ( or throw a proper error when receiving) status updates from application masters that have not registered (Mayank Bansal & Abhishek Kapoor via bikas) (Revision 1501605) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501605 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/InvalidApplicationMasterRequestException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java > Handle ( or throw a proper error when receiving) status updates from > application masters that have not registered > - > > Key: YARN-369 > URL: https://issues.apache.org/jira/browse/YARN-369 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.0.3-alpha, trunk-win >Reporter: Hitesh Shah >Assignee: Mayank Bansal > Fix For: 2.1.0-beta > > Attachments: YARN-369.patch, YARN-369-trunk-1.patch, > YARN-369-trunk-2.patch, YARN-369-trunk-3.patch, YARN-369-trunk-4.patch > > > Currently, an allocate call from an unregistered application is allowed and > the status update for it throws a statemachine error that is silently dropped. > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > STATUS_UPDATE at LAUNCHED >at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) >at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43) >at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445) >at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:588) >at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:99) >at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:471) >at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:452) >at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130) >at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77) >at java.lang.Thread.run(Thread.java:680) > ApplicationMasterService should likely throw an appropriate error for > applications' requests that should not be handled in such cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-727) ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter
[ https://issues.apache.org/jira/browse/YARN-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704423#comment-13704423 ] Hudson commented on YARN-727: - Integrated in Hadoop-Yarn-trunk #266 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/266/]) YARN-727, MAPREDUCE-5325. ClientRMProtocol.getAllApplications should accept ApplicationType as a parameter. Contributed by Xuan Gong. (Revision 1501599) Result = SUCCESS hitesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1501599 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestClientRedirect.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetAllApplicationsRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetAllApplicationsResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationsResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/applicationclient_protocol.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/YarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/YarnClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/YarnCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestYarnCLI.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/client/ApplicationClientProtocolPBClientImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/impl/pb/service/ApplicationClientProtocolPBServiceImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAllApplicationsRequestPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetAllApplicationsResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsRequestPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/GetApplicationsResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationACLs.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java > ClientRMProtocol.getAllApplications should accept ApplicationType as a > pa
[jira] [Commented] (YARN-649) Make container logs available over HTTP in plain text
[ https://issues.apache.org/jira/browse/YARN-649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704289#comment-13704289 ] Vinod Kumar Vavilapalli commented on YARN-649: -- Also, don't exactly how to and if we can test if the buffers are indeed bounded. If nothing, if you have done some manual test or something, it will be great if you can report that. Tx! > Make container logs available over HTTP in plain text > - > > Key: YARN-649 > URL: https://issues.apache.org/jira/browse/YARN-649 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: YARN-649-2.patch, YARN-649-3.patch, YARN-649-4.patch, > YARN-649-5.patch, YARN-649.patch, YARN-752-1.patch > > > It would be good to make container logs available over the REST API for > MAPREDUCE-4362 and so that they can be accessed programatically in general. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
