[jira] [Updated] (YARN-137) Change the default scheduler to the CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated YARN-137: - Fix Version/s: 0.23.5 Change the default scheduler to the CapacityScheduler - Key: YARN-137 URL: https://issues.apache.org/jira/browse/YARN-137 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.0.0-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 2.0.2-alpha, 0.23.5 Attachments: MR4335_2.txt, MR4335_3.txt, MR4335_4.txt, MR4335.txt, YARN-137_branch23.patch, YARN-137.patch, YARN137.txt There's some bugs in the FifoScheduler atm - doesn't distribute tasks across nodes and some headroom (available resource) issues. That's not the best experience for users trying out the 2.0 branch. The CS with the default configuration of a single queue behaves the same as the FifoScheduler and doesn't have these issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-131) Incorrect ACL properties in capacity scheduler documentation
[
https://issues.apache.org/jira/browse/YARN-131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13466904#comment-13466904
]
Ahmed Radwan commented on YARN-131:
---
Thanks Sid, I'll take a look at those too and update the patch.
Incorrect ACL properties in capacity scheduler documentation
Key: YARN-131
URL: https://issues.apache.org/jira/browse/YARN-131
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler
Affects Versions: 2.0.0-alpha
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
Attachments: YARN-131.patch
The CapacityScheduler apt file incorrectly specifies the property names
controlling acls for application submission and queue administration.
{{yarn.scheduler.capacity.root.queue-path.acl_submit_jobs}}
should be
{{yarn.scheduler.capacity.root.queue-path.acl_submit_applications}}
{{yarn.scheduler.capacity.root.queue-path.acl_administer_jobs}}
should be
{{yarn.scheduler.capacity.root.queue-path.acl_administer_queue}}
Uploading a patch momentarily.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-139) Interrupted Exception within AsyncDispatcher leads to user confusion
Nathan Roberts created YARN-139: --- Summary: Interrupted Exception within AsyncDispatcher leads to user confusion Key: YARN-139 URL: https://issues.apache.org/jira/browse/YARN-139 Project: Hadoop YARN Issue Type: Bug Components: api Affects Versions: 2.0.2-alpha, 0.23.4 Reporter: Nathan Roberts Successful applications tend to get InterruptedExceptions during shutdown. The exception is harmless but it leads to lots of user confusion and therefore could be cleaned up. 2012-09-28 14:50:12,477 WARN [AsyncDispatcher event handler] org.apache.hadoop.yarn.event.AsyncDispatcher: Interrupted Exception while stopping java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1143) at java.lang.Thread.join(Thread.java:1196) at org.apache.hadoop.yarn.event.AsyncDispatcher.stop(AsyncDispatcher.java:105) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99) at org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:437) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:402) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75) at java.lang.Thread.run(Thread.java:619) 2012-09-28 14:50:12,477 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.service.AbstractService: Service:Dispatcher is stopped. 2012-09-28 14:50:12,477 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.service.AbstractService: Service:org.apache.hadoop.mapreduce.v2.app.MRAppMaster is stopped. 2012-09-28 14:50:12,477 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Exiting MR AppMaster..GoodBye -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-137) Change the default scheduler to the CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated YARN-137: - Fix Version/s: (was: 0.23.5) 0.23.4 pulled this into branch-0.23.4 Change the default scheduler to the CapacityScheduler - Key: YARN-137 URL: https://issues.apache.org/jira/browse/YARN-137 Project: Hadoop YARN Issue Type: Improvement Components: scheduler Affects Versions: 2.0.0-alpha Reporter: Siddharth Seth Assignee: Siddharth Seth Fix For: 2.0.2-alpha, 0.23.4 Attachments: MR4335_2.txt, MR4335_3.txt, MR4335_4.txt, MR4335.txt, YARN-137_branch23.patch, YARN-137.patch, YARN137.txt There's some bugs in the FifoScheduler atm - doesn't distribute tasks across nodes and some headroom (available resource) issues. That's not the best experience for users trying out the 2.0 branch. The CS with the default configuration of a single queue behaves the same as the FifoScheduler and doesn't have these issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-138) Improve default config values for YARN
[ https://issues.apache.org/jira/browse/YARN-138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated YARN-138: - Fix Version/s: (was: 0.23.5) 0.23.4 Improve default config values for YARN -- Key: YARN-138 URL: https://issues.apache.org/jira/browse/YARN-138 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager, scheduler Affects Versions: 2.0.0-alpha Reporter: Arun C Murthy Assignee: Harsh J Labels: performance Fix For: 2.0.2-alpha, 0.23.4 Attachments: MAPREDUCE-4316.patch, YARN-138_branch-0.23.patch, YARN138.txt, YARN138.txt, YARN138.txt Currently some of our configs are way off e.g. min-alloc is 128M while max-alloc is 10240. This leads to poor out-of-box performance as noticed by some users: http://s.apache.org/avd -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-103) Add a yarn AM - RM client module
[
https://issues.apache.org/jira/browse/YARN-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467208#comment-13467208
]
Siddharth Seth commented on YARN-103:
-
This is looking good with the synchronization fixes and error handling. The
MR-3902 branch has some similar changes in RMContainerRequester. I'll try
plugging this in there.
A few comments.
- {{makeContainerRequest}} was called {{addContainerRequest}} in
RMContainerRequestor. Why rename it. To me, addContainerRequest makes more
sense since the AMRMClient maintains an aggregate of requests.
- The javadoc for removeContainerRequest is a little misleading. This will not
remove previous requests after allocate has been called for those requests.
removeContainerRequests just decrements the specific request from the main
request table - and sends out whatever has changed in the next allocate call.
- getClusterAvailableResources / getClusterNodeCount - Instead of documenting
that these should be called after an 'allocate' call - should this be enforced,
maybe via an exception ? (Independent of this jira, does it make sense to
include additional information in the Register call - i.e. numClusterNodes,
avaialbleResources / other parts of the allocate payload)
- Similarly, instead of documenting that allocate cannot be called concurrently
- this can be enforced.
- In the unit test, would prefer avoiding MiniMRCluster where possible. This
one should be possible using the already existing {{MockRM}} or by mocking the
RM. (branch MR-3902:TestRMContainerRequestor already does this, and tests a
failure scenario for this patch)
Minor Stuff
- Avoid RecordFactory usage. Replaced by Records.newRecord() or BuilderUtils.*
- The modification to ask in case of a failure could be a simple {{ask.addAll}}
instead of the exists check.
Add a yarn AM - RM client module
Key: YARN-103
URL: https://issues.apache.org/jira/browse/YARN-103
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: YARN-103.1.patch, YARN-103.2.patch, YARN-103.3.patch
Add a basic client wrapper library to the AM RM protocol in order to prevent
proliferation of code being duplicated everywhere. Provide helper functions
to perform reverse mapping of container requests to RM allocation resource
request table format.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-40) Provide support for missing yarn commands
[ https://issues.apache.org/jira/browse/YARN-40?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467302#comment-13467302 ] Vinod Kumar Vavilapalli commented on YARN-40: - Been a while, will rekick Jenkins and commit this. Provide support for missing yarn commands - Key: YARN-40 URL: https://issues.apache.org/jira/browse/YARN-40 Project: Hadoop YARN Issue Type: Bug Components: client Affects Versions: 2.0.0-alpha Reporter: Devaraj K Assignee: Devaraj K Attachments: MAPREDUCE-4155-1.patch, MAPREDUCE-4155.patch, YARN-40-1.patch, YARN-40-20120917.1.txt, YARN-40-20120917.txt, YARN-40-20120924.txt, YARN-40.patch 1. status app-id 2. kill app-id (Already issue present with Id : MAPREDUCE-3793) 3. list-apps [all] 4. nodes-report -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-94) DistributedShell jar should point to Client as the main class by default
[
https://issues.apache.org/jira/browse/YARN-94?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467307#comment-13467307
]
Hitesh Shah commented on YARN-94:
-
@Vinod, could you re-check? The test I ran worked for me. Unit test change
already looks at the exception thrown.
DistributedShell jar should point to Client as the main class by default
Key: YARN-94
URL: https://issues.apache.org/jira/browse/YARN-94
Project: Hadoop YARN
Issue Type: Bug
Components: applications/distributed-shell
Reporter: Vinod Kumar Vavilapalli
Assignee: Hitesh Shah
Attachments: YARN-94.1.patch, YARN-94.2.patch
Today, it says so..
{code}
$ $YARN_HOME/bin/yarn jar
$YARN_HOME/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-$VERSION.jar
RunJar jarFile [mainClass] args...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-40) Provide support for missing yarn commands
[
https://issues.apache.org/jira/browse/YARN-40?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13467311#comment-13467311
]
Hadoop QA commented on YARN-40:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12546438/YARN-40-20120924.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site:
org.apache.hadoop.yarn.client.TestYarnClient
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/65//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/65//console
This message is automatically generated.
Provide support for missing yarn commands
-
Key: YARN-40
URL: https://issues.apache.org/jira/browse/YARN-40
Project: Hadoop YARN
Issue Type: Bug
Components: client
Affects Versions: 2.0.0-alpha
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4155-1.patch, MAPREDUCE-4155.patch,
YARN-40-1.patch, YARN-40-20120917.1.txt, YARN-40-20120917.txt,
YARN-40-20120924.txt, YARN-40.patch
1. status app-id
2. kill app-id (Already issue present with Id : MAPREDUCE-3793)
3. list-apps [all]
4. nodes-report
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-131) Incorrect ACL properties in capacity scheduler documentation
[
https://issues.apache.org/jira/browse/YARN-131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Radwan updated YARN-131:
--
Attachment: YARN-131_rev2.patch
Here is the updated patch correcting the additional file that Siddharth pointed
out.
Incorrect ACL properties in capacity scheduler documentation
Key: YARN-131
URL: https://issues.apache.org/jira/browse/YARN-131
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler
Affects Versions: 2.0.0-alpha
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
Attachments: YARN-131.patch, YARN-131_rev2.patch
The CapacityScheduler apt file incorrectly specifies the property names
controlling acls for application submission and queue administration.
{{yarn.scheduler.capacity.root.queue-path.acl_submit_jobs}}
should be
{{yarn.scheduler.capacity.root.queue-path.acl_submit_applications}}
{{yarn.scheduler.capacity.root.queue-path.acl_administer_jobs}}
should be
{{yarn.scheduler.capacity.root.queue-path.acl_administer_queue}}
Uploading a patch momentarily.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
