[jira] [Commented] (YARN-753) Add individual factory method for api protocol records
[ https://issues.apache.org/jira/browse/YARN-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674105#comment-13674105 ] Vinod Kumar Vavilapalli commented on YARN-753: -- Looked at the patch. Mostly looks good except the following: - GetAllApplicationsResponse's factory isn't setting the appReports field. - PreemptionResourceRequestPBImpl should have equals/hashCode etc. Add individual factory method for api protocol records -- Key: YARN-753 URL: https://issues.apache.org/jira/browse/YARN-753 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: YARN-753.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-761) TestNMClientAsync fails sometimes
[ https://issues.apache.org/jira/browse/YARN-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-761: - Assignee: Zhijie Shen (was: Vinod Kumar Vavilapalli) TestNMClientAsync fails sometimes - Key: YARN-761 URL: https://issues.apache.org/jira/browse/YARN-761 Project: Hadoop YARN Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Assignee: Zhijie Shen Attachments: YARN-761.1.patch See https://builds.apache.org/job/PreCommit-YARN-Build/1101//testReport/. It passed on my machine though. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-761) TestNMClientAsync fails sometimes
[
https://issues.apache.org/jira/browse/YARN-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen updated YARN-761:
-
Attachment: YARN-761.1.patch
It's the race condition problem, but is restricted with the test.
It happens when
{code}
Assert.assertEquals(Completed container is not removed, 0,
asyncClient.containers.size());
{code}
executes before all ContainerEventProcessors completes and containers are
removed.
After the following step:
{code}
while (!((TestCallbackHandler1) asyncClient.callbackHandler)
.isStopFailureCallsExecuted()) {
Thread.sleep(10);
}
{code}
all the callback functions are executed, but the remaining operations in
ContainerEventProcessors#run may still not be executed.
TestNMClientAsync fails sometimes
-
Key: YARN-761
URL: https://issues.apache.org/jira/browse/YARN-761
Project: Hadoop YARN
Issue Type: Bug
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
Attachments: YARN-761.1.patch
See https://builds.apache.org/job/PreCommit-YARN-Build/1101//testReport/.
It passed on my machine though.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-755) Rename AllocateResponse.reboot to AllocateResponse.resync
[ https://issues.apache.org/jira/browse/YARN-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674107#comment-13674107 ] Hudson commented on YARN-755: - Integrated in Hadoop-trunk-Commit #3857 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3857/]) YARN-755. Renamed AllocateResponse.reboot to AllocateResponse.resync. Contributed by Bikas Saha. (Revision 1489295) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489295 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCResponseId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestApplicationTokens.java Rename AllocateResponse.reboot to AllocateResponse.resync - Key: YARN-755 URL: https://issues.apache.org/jira/browse/YARN-755 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.1.0-beta Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 2.1.0-beta Attachments: YARN-755.1.patch, YARN-755.2.patch For work preserving rm restart the am's will be resyncing instead of rebooting. rebooting is an action that currently satisfies the resync requirement. Changing the name now so that it continues to make sense in the real resync case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-569) CapacityScheduler: support for preemption (using a capacity monitor)
[ https://issues.apache.org/jira/browse/YARN-569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated YARN-569: --- Attachment: YARN-569.4.patch Rebase after YARN-635, YARN-735, YARN-748, YARN-749. Fixed findbugs warnings. CapacityScheduler: support for preemption (using a capacity monitor) Key: YARN-569 URL: https://issues.apache.org/jira/browse/YARN-569 Project: Hadoop YARN Issue Type: Sub-task Components: capacityscheduler Reporter: Carlo Curino Assignee: Carlo Curino Attachments: 3queues.pdf, CapScheduler_with_preemption.pdf, preemption.2.patch, YARN-569.1.patch, YARN-569.2.patch, YARN-569.3.patch, YARN-569.4.patch, YARN-569.patch, YARN-569.patch There is a tension between the fast-pace reactive role of the CapacityScheduler, which needs to respond quickly to applications resource requests, and node updates, and the more introspective, time-based considerations needed to observe and correct for capacity balance. To this purpose we opted instead of hacking the delicate mechanisms of the CapacityScheduler directly to add support for preemption by means of a Capacity Monitor, which can be run optionally as a separate service (much like the NMLivelinessMonitor). The capacity monitor (similarly to equivalent functionalities in the fairness scheduler) operates running on intervals (e.g., every 3 seconds), observe the state of the assignment of resources to queues from the capacity scheduler, performs off-line computation to determine if preemption is needed, and how best to edit the current schedule to improve capacity, and generates events that produce four possible actions: # Container de-reservations # Resource-based preemptions # Container-based preemptions # Container killing The actions listed above are progressively more costly, and it is up to the policy to use them as desired to achieve the rebalancing goals. Note that due to the lag in the effect of these actions the policy should operate at the macroscopic level (e.g., preempt tens of containers from a queue) and not trying to tightly and consistently micromanage container allocations. - Preemption policy (ProportionalCapacityPreemptionPolicy): - Preemption policies are by design pluggable, in the following we present an initial policy (ProportionalCapacityPreemptionPolicy) we have been experimenting with. The ProportionalCapacityPreemptionPolicy behaves as follows: # it gathers from the scheduler the state of the queues, in particular, their current capacity, guaranteed capacity and pending requests (*) # if there are pending requests from queues that are under capacity it computes a new ideal balanced state (**) # it computes the set of preemptions needed to repair the current schedule and achieve capacity balance (accounting for natural completion rates, and respecting bounds on the amount of preemption we allow for each round) # it selects which applications to preempt from each over-capacity queue (the last one in the FIFO order) # it remove reservations from the most recently assigned app until the amount of resource to reclaim is obtained, or until no more reservations exits # (if not enough) it issues preemptions for containers from the same applications (reverse chronological order, last assigned container first) again until necessary or until no containers except the AM container are left, # (if not enough) it moves onto unreserve and preempt from the next application. # containers that have been asked to preempt are tracked across executions. If a containers is among the one to be preempted for more than a certain time, the container is moved in a the list of containers to be forcibly killed. Notes: (*) at the moment, in order to avoid double-counting of the requests, we only look at the ANY part of pending resource requests, which means we might not preempt on behalf of AMs that ask only for specific locations but not any. (**) The ideal balance state is one in which each queue has at least its guaranteed capacity, and the spare capacity is distributed among queues (that wants some) as a weighted fair share. Where the weighting is based on the guaranteed capacity of a queue, and the function runs to a fix point. Tunables of the ProportionalCapacityPreemptionPolicy: # observe-only mode (i.e., log the actions it would take, but behave as read-only) # how frequently to run the policy # how long to wait between preemption and kill of a container # which fraction of the containers I would like to obtain should I preempt (has to do with the natural rate at which containers are returned) # deadzone size, i.e., what % of
[jira] [Commented] (YARN-761) TestNMClientAsync fails sometimes
[
https://issues.apache.org/jira/browse/YARN-761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674111#comment-13674111
]
Hadoop QA commented on YARN-761:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586056/YARN-761.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1105//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1105//console
This message is automatically generated.
TestNMClientAsync fails sometimes
-
Key: YARN-761
URL: https://issues.apache.org/jira/browse/YARN-761
Project: Hadoop YARN
Issue Type: Bug
Reporter: Vinod Kumar Vavilapalli
Assignee: Zhijie Shen
Attachments: YARN-761.1.patch
See https://builds.apache.org/job/PreCommit-YARN-Build/1101//testReport/.
It passed on my machine though.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-569) CapacityScheduler: support for preemption (using a capacity monitor)
[
https://issues.apache.org/jira/browse/YARN-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674115#comment-13674115
]
Hadoop QA commented on YARN-569:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586057/YARN-569.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1106//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1106//console
This message is automatically generated.
CapacityScheduler: support for preemption (using a capacity monitor)
Key: YARN-569
URL: https://issues.apache.org/jira/browse/YARN-569
Project: Hadoop YARN
Issue Type: Sub-task
Components: capacityscheduler
Reporter: Carlo Curino
Assignee: Carlo Curino
Attachments: 3queues.pdf, CapScheduler_with_preemption.pdf,
preemption.2.patch, YARN-569.1.patch, YARN-569.2.patch, YARN-569.3.patch,
YARN-569.4.patch, YARN-569.patch, YARN-569.patch
There is a tension between the fast-pace reactive role of the
CapacityScheduler, which needs to respond quickly to
applications resource requests, and node updates, and the more introspective,
time-based considerations
needed to observe and correct for capacity balance. To this purpose we opted
instead of hacking the delicate
mechanisms of the CapacityScheduler directly to add support for preemption by
means of a Capacity Monitor,
which can be run optionally as a separate service (much like the
NMLivelinessMonitor).
The capacity monitor (similarly to equivalent functionalities in the fairness
scheduler) operates running on intervals
(e.g., every 3 seconds), observe the state of the assignment of resources to
queues from the capacity scheduler,
performs off-line computation to determine if preemption is needed, and how
best to edit the current schedule to
improve capacity, and generates events that produce four possible actions:
# Container de-reservations
# Resource-based preemptions
# Container-based preemptions
# Container killing
The actions listed above are progressively more costly, and it is up to the
policy to use them as desired to achieve the rebalancing goals.
Note that due to the lag in the effect of these actions the policy should
operate at the macroscopic level (e.g., preempt tens of containers
from a queue) and not trying to tightly and consistently micromanage
container allocations.
- Preemption policy (ProportionalCapacityPreemptionPolicy):
-
Preemption policies are by design pluggable, in the following we present an
initial policy (ProportionalCapacityPreemptionPolicy) we have been
experimenting with. The ProportionalCapacityPreemptionPolicy behaves as
follows:
# it gathers from the scheduler the state of the queues, in particular, their
current capacity, guaranteed capacity and pending requests (*)
# if there are pending requests from queues that are under capacity it
computes a new ideal balanced state (**)
# it computes the set of preemptions needed to repair the current schedule
and achieve capacity balance (accounting for natural completion rates, and
respecting bounds on the amount of preemption we allow for each round)
# it selects which applications to preempt from each over-capacity queue (the
last one in the FIFO order)
# it remove reservations from the most recently assigned app until the amount
of resource to reclaim is obtained, or until no more reservations exits
# (if not enough) it issues preemptions for containers from the same
applications (reverse chronological order, last assigned container first)
again until necessary or
[jira] [Updated] (YARN-514) Delayed store operations should not result in RM unavailability for app submission
[ https://issues.apache.org/jira/browse/YARN-514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 闫昆 updated YARN-514: Component/s: (was: resourcemanager) Delayed store operations should not result in RM unavailability for app submission -- Key: YARN-514 URL: https://issues.apache.org/jira/browse/YARN-514 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: YARN-514.1.patch, YARN-514.2.patch, YARN-514.3.patch, YARN-514.4.patch, YARN-514.5.patch, YARN-514.6.patch, YARN-514.7.patch, YARN-514.8.patch Currently, app submission is the only store operation performed synchronously because the app must be stored before the request returns with success. This makes the RM susceptible to blocking all client threads on slow store operations, resulting in RM being perceived as unavailable by clients. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-635) Rename YarnRemoteException to YarnException
[
https://issues.apache.org/jira/browse/YARN-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674202#comment-13674202
]
Steve Loughran commented on YARN-635:
-
I've just noticed this and am dealing with the rebasing of the YARN-117
patches. Why doesn't this exception extend {{IOException}}? There's a lot of
code that throws IOEs...
Rename YarnRemoteException to YarnException
---
Key: YARN-635
URL: https://issues.apache.org/jira/browse/YARN-635
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.1.0-beta
Reporter: Xuan Gong
Assignee: Siddharth Seth
Fix For: 2.1.0-beta
Attachments: y635_afterRefactor.txt, y635_trunk_fix.txt,
YARN-635.2.txt, YARN-635.3.txt, YARN-635.3.txt, YARN-635.txt,
YARN-635_yarnOnly_branch2.txt, YARN-635.yarnOnly.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-505) NPE at AsyncDispatcher$GenericEventHandler
[
https://issues.apache.org/jira/browse/YARN-505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674208#comment-13674208
]
Niranjan Singh commented on YARN-505:
-
The NPE is in the LinkedBlockingQueue class of JDK.
In handler.handle(null), the event which is null in this case is put in
LinkedBlockingQueue ,
without checking whether the event is null or not.
eventQueue.put(event);line 188 of org.apache.hadoop.yarn.event.AsyncDispatcher.
One solution could be to check for null when event is put in eventQueue below.
if(event != null){
eventQueue.put(event);
}else{
LOG.warn(Event cannot be null+event);
}
NPE at AsyncDispatcher$GenericEventHandler
--
Key: YARN-505
URL: https://issues.apache.org/jira/browse/YARN-505
Project: Hadoop YARN
Issue Type: Improvement
Components: resourcemanager
Affects Versions: 3.0.0
Reporter: Przemyslaw Pretki
Priority: Minor
Steps to reproduce:
{code}
@Test
public void testAsyncDispatcher()
{
AsyncDispatcher dispatcher = new AsyncDispatcher();
EventHandler handler = dispatcher.getEventHandler();
handler.handle(null);
}
{code}
Moreover, event taken from *BlockingQueue* will never be *null*, so that it
seems that the following condition is not necessary
(AsyncDispatcher.createThread() method):
{code}
if (event != null) {
dispatch(event);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-762) Javadoc params not matching in ContainerManagerImpl.authorizeRequest method
Niranjan Singh created YARN-762: --- Summary: Javadoc params not matching in ContainerManagerImpl.authorizeRequest method Key: YARN-762 URL: https://issues.apache.org/jira/browse/YARN-762 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Reporter: Niranjan Singh Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-762) Javadoc params not matching in ContainerManagerImpl.authorizeRequest method
[ https://issues.apache.org/jira/browse/YARN-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niranjan Singh updated YARN-762: Description: In ContainerManagerImpl.authorizeRequest method the number of parameters passed are four where as in Javadoc the params are only three. (was: In ) Javadoc params not matching in ContainerManagerImpl.authorizeRequest method --- Key: YARN-762 URL: https://issues.apache.org/jira/browse/YARN-762 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Reporter: Niranjan Singh Priority: Minor In ContainerManagerImpl.authorizeRequest method the number of parameters passed are four where as in Javadoc the params are only three. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-756) Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract to api.records
[ https://issues.apache.org/jira/browse/YARN-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674247#comment-13674247 ] Hudson commented on YARN-756: - Integrated in Hadoop-Yarn-trunk #230 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/230/]) YARN-756. Move Preemption* records to yarn.api where they really belong. Contributed by Jian He. (Revision 1489290) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489290 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/PreemptionContainer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/PreemptionContract.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/PreemptionMessage.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/PreemptionResourceRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/StrictPreemptionContract.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/PreemptionContainerPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/PreemptionContractPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/PreemptionMessagePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/StrictPreemptionContractPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/PreemptionContainer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/PreemptionContract.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/PreemptionMessage.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/StrictPreemptionContract.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/PreemptionContainerPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/PreemptionContractPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/PreemptionMessagePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/StrictPreemptionContractPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract to api.records Key: YARN-756 URL: https://issues.apache.org/jira/browse/YARN-756 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: YARN-756.1.patch, YARN-756.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-726) Queue, FinishTime fields broken on RM UI
[ https://issues.apache.org/jira/browse/YARN-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674248#comment-13674248 ] Hudson commented on YARN-726: - Integrated in Hadoop-Yarn-trunk #230 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/230/]) YARN-726. Fix queue finish time fields in web-ui for ResourceManager. Contributed by Mayank Bansal. (Revision 1489234) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489234 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java Queue, FinishTime fields broken on RM UI Key: YARN-726 URL: https://issues.apache.org/jira/browse/YARN-726 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Siddharth Seth Assignee: Mayank Bansal Priority: Critical Fix For: 2.1.0-beta Attachments: YARN-726-trunk-1.patch The queue shows up as Invalid Date Finish Time shows up as a Long value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-635) Rename YarnRemoteException to YarnException
[ https://issues.apache.org/jira/browse/YARN-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674250#comment-13674250 ] Hudson commented on YARN-635: - Integrated in Hadoop-Yarn-trunk #230 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/230/]) YARN-635 and MAPREDUCE-5301. Bug fix as the committed patch ended up being stale. (Revision 1489287) YARN-635. Renamed YarnRemoteException to YarnException. Contributed by Siddharth Seth. MAPREDUCE-5301. Updated MR code to work with YARN-635 changes of renaming YarnRemoteException to YarnException. Contributed by Siddharth Seth (Revision 1489283) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489287 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryContext.java vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489283 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryCopyService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/commit/CommitterEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java *
[jira] [Commented] (YARN-755) Rename AllocateResponse.reboot to AllocateResponse.resync
[ https://issues.apache.org/jira/browse/YARN-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674245#comment-13674245 ] Hudson commented on YARN-755: - Integrated in Hadoop-Yarn-trunk #230 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/230/]) YARN-755. Renamed AllocateResponse.reboot to AllocateResponse.resync. Contributed by Bikas Saha. (Revision 1489295) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489295 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCResponseId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestApplicationTokens.java Rename AllocateResponse.reboot to AllocateResponse.resync - Key: YARN-755 URL: https://issues.apache.org/jira/browse/YARN-755 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.1.0-beta Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 2.1.0-beta Attachments: YARN-755.1.patch, YARN-755.2.patch For work preserving rm restart the am's will be resyncing instead of rebooting. rebooting is an action that currently satisfies the resync requirement. Changing the name now so that it continues to make sense in the real resync case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-398) Enhance CS to allow for white-list of resources
[ https://issues.apache.org/jira/browse/YARN-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674259#comment-13674259 ] Hudson commented on YARN-398: - Integrated in Hadoop-Yarn-trunk #230 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/230/]) YARN-398. Make it possible to specify hard locality constraints in resource requests for CapacityScheduler. Contributed by Arun C. Murthy. (Revision 1489087) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489087 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java Enhance CS to allow for white-list of resources --- Key: YARN-398 URL: https://issues.apache.org/jira/browse/YARN-398 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 2.1.0-beta Attachments: YARN-398.patch, YARN-398.patch Allow white-list and black-list of resources in scheduler api. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-748) Move BuilderUtils from yarn-common to yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674246#comment-13674246 ] Hudson commented on YARN-748: - Integrated in Hadoop-Yarn-trunk #230 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/230/]) YARN-748. Moved BuilderUtils from yarn-common to yarn-server-common for eventual retirement. Contributed by Jian He. MAPREDUCE-5297. Updated MR App since BuilderUtils is no longer public after YARN-748. Contributed by Jian He. (Revision 1489257) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489257 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestTaskHeartbeatHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttemptContainerRequest.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/security/MRDelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRBuilderUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/api/records/TestIds.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestMapReduceTrackingUriPlugin.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryEvents.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHSWebApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java *
[jira] [Commented] (YARN-689) Add multiplier unit to resourcecapabilities
[ https://issues.apache.org/jira/browse/YARN-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674293#comment-13674293 ] Arun C Murthy commented on YARN-689: bq. It seems that there's a concern about impacting CS with this change [~tomwhite] FTR, the same concern (too much fragmentation and resulting poor utilization) applies to the FairScheduler too. Add multiplier unit to resourcecapabilities --- Key: YARN-689 URL: https://issues.apache.org/jira/browse/YARN-689 Project: Hadoop YARN Issue Type: Bug Components: api, scheduler Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-689.patch, YARN-689.patch, YARN-689.patch, YARN-689.patch, YARN-689.patch Currently we overloading the minimum resource value as the actual multiplier used by the scheduler. Today with a minimum memory set to 1GB, requests for 1.5GB are always translated to allocation of 2GB. We should decouple the minimum allocation from the multiplier. The multiplier should also be exposed to the client via the RegisterApplicationMasterResponse -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-750) Allow for black-listing resources in CS
[ https://issues.apache.org/jira/browse/YARN-750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated YARN-750: --- Attachment: YARN-750.patch Allow for black-listing resources in CS --- Key: YARN-750 URL: https://issues.apache.org/jira/browse/YARN-750 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Arun C Murthy Attachments: YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch YARN-392 and YARN-398 enhance scheduler api to allow for white-lists of resources. This jira is a companion to allow for black-listing (in CS). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-326) Add multi-resource scheduling to the fair scheduler
[ https://issues.apache.org/jira/browse/YARN-326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674329#comment-13674329 ] Hudson commented on YARN-326: - Integrated in Hadoop-Hdfs-trunk #1420 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1420/]) YARN-326. Add multi-resource scheduling to the fair scheduler. (sandyr via tucu) (Revision 1489070) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489070 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/ResourceType.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/ResourceWeights.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerNode.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/FairSharePolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FakeSchedulable.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestSchedulingPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/TestDominantResourceFairnessPolicy.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm Add multi-resource scheduling to the fair scheduler --- Key: YARN-326 URL: https://issues.apache.org/jira/browse/YARN-326 Project: Hadoop YARN Issue Type: New Feature
[jira] [Commented] (YARN-726) Queue, FinishTime fields broken on RM UI
[ https://issues.apache.org/jira/browse/YARN-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674339#comment-13674339 ] Hudson commented on YARN-726: - Integrated in Hadoop-Hdfs-trunk #1420 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1420/]) YARN-726. Fix queue finish time fields in web-ui for ResourceManager. Contributed by Mayank Bansal. (Revision 1489234) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489234 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java Queue, FinishTime fields broken on RM UI Key: YARN-726 URL: https://issues.apache.org/jira/browse/YARN-726 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Siddharth Seth Assignee: Mayank Bansal Priority: Critical Fix For: 2.1.0-beta Attachments: YARN-726-trunk-1.patch The queue shows up as Invalid Date Finish Time shows up as a Long value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-748) Move BuilderUtils from yarn-common to yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674337#comment-13674337 ] Hudson commented on YARN-748: - Integrated in Hadoop-Hdfs-trunk #1420 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1420/]) YARN-748. Moved BuilderUtils from yarn-common to yarn-server-common for eventual retirement. Contributed by Jian He. MAPREDUCE-5297. Updated MR App since BuilderUtils is no longer public after YARN-748. Contributed by Jian He. (Revision 1489257) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489257 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestTaskHeartbeatHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttemptContainerRequest.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/security/MRDelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRBuilderUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/api/records/TestIds.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestMapReduceTrackingUriPlugin.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryEvents.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHSWebApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java *
[jira] [Commented] (YARN-755) Rename AllocateResponse.reboot to AllocateResponse.resync
[ https://issues.apache.org/jira/browse/YARN-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674336#comment-13674336 ] Hudson commented on YARN-755: - Integrated in Hadoop-Hdfs-trunk #1420 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1420/]) YARN-755. Renamed AllocateResponse.reboot to AllocateResponse.resync. Contributed by Bikas Saha. (Revision 1489295) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489295 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCResponseId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestApplicationTokens.java Rename AllocateResponse.reboot to AllocateResponse.resync - Key: YARN-755 URL: https://issues.apache.org/jira/browse/YARN-755 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.1.0-beta Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 2.1.0-beta Attachments: YARN-755.1.patch, YARN-755.2.patch For work preserving rm restart the am's will be resyncing instead of rebooting. rebooting is an action that currently satisfies the resync requirement. Changing the name now so that it continues to make sense in the real resync case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-757) TestRMRestart failing/stuck on trunk
[ https://issues.apache.org/jira/browse/YARN-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674343#comment-13674343 ] Hudson commented on YARN-757: - Integrated in Hadoop-Hdfs-trunk #1420 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1420/]) YARN-757. Changed TestRMRestart to use the default scheduler to avoid test failures. Contributed by Bikas Saha. (Revision 1489268) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489268 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java TestRMRestart failing/stuck on trunk Key: YARN-757 URL: https://issues.apache.org/jira/browse/YARN-757 Project: Hadoop YARN Issue Type: Bug Reporter: Bikas Saha Assignee: Bikas Saha Priority: Blocker Fix For: 2.1.0-beta Attachments: YARN-757.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-756) Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract to api.records
[ https://issues.apache.org/jira/browse/YARN-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674338#comment-13674338 ] Hudson commented on YARN-756: - Integrated in Hadoop-Hdfs-trunk #1420 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1420/]) YARN-756. Move Preemption* records to yarn.api where they really belong. Contributed by Jian He. (Revision 1489290) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489290 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/PreemptionContainer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/PreemptionContract.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/PreemptionMessage.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/PreemptionResourceRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/StrictPreemptionContract.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/PreemptionContainerPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/PreemptionContractPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/PreemptionMessagePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/StrictPreemptionContractPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/PreemptionContainer.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/PreemptionContract.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/PreemptionMessage.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/StrictPreemptionContract.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/PreemptionContainerPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/PreemptionContractPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/PreemptionMessagePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/StrictPreemptionContractPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract to api.records Key: YARN-756 URL: https://issues.apache.org/jira/browse/YARN-756 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: YARN-756.1.patch, YARN-756.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-398) Enhance CS to allow for white-list of resources
[ https://issues.apache.org/jira/browse/YARN-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674350#comment-13674350 ] Hudson commented on YARN-398: - Integrated in Hadoop-Hdfs-trunk #1420 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1420/]) YARN-398. Make it possible to specify hard locality constraints in resource requests for CapacityScheduler. Contributed by Arun C. Murthy. (Revision 1489087) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489087 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java Enhance CS to allow for white-list of resources --- Key: YARN-398 URL: https://issues.apache.org/jira/browse/YARN-398 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 2.1.0-beta Attachments: YARN-398.patch, YARN-398.patch Allow white-list and black-list of resources in scheduler api. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-750) Allow for black-listing resources in CS
[
https://issues.apache.org/jira/browse/YARN-750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674359#comment-13674359
]
Hadoop QA commented on YARN-750:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586096/YARN-750.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 4
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler
org.apache.hadoop.yarn.server.resourcemanager.applicationmasterservice.TestApplicationMasterService
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates
org.apache.hadoop.yarn.server.resourcemanager.TestRM
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.TestRMAppAttemptTransitions
org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1107//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1107//console
This message is automatically generated.
Allow for black-listing resources in CS
---
Key: YARN-750
URL: https://issues.apache.org/jira/browse/YARN-750
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Attachments: YARN-750.patch, YARN-750.patch, YARN-750.patch,
YARN-750.patch, YARN-750.patch
YARN-392 and YARN-398 enhance scheduler api to allow for white-lists of
resources.
This jira is a companion to allow for black-listing (in CS).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-742) Log aggregation causes a lot of redundant setPermission calls
[ https://issues.apache.org/jira/browse/YARN-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674393#comment-13674393 ] Kihwal Lee commented on YARN-742: - +1 the patch looks good. Log aggregation causes a lot of redundant setPermission calls - Key: YARN-742 URL: https://issues.apache.org/jira/browse/YARN-742 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Kihwal Lee Assignee: Jason Lowe Attachments: YARN-742.patch In one of our clusters, namenode RPC is spending 45% of its time on serving setPermission calls. Further investigation has revealed that most calls are redundantly made on /mapred/logs/user/logs. Also mkdirs calls are made before this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-169) Update log4j.appender.EventCounter to use org.apache.hadoop.log.metrics.EventCounter
[
https://issues.apache.org/jira/browse/YARN-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674396#comment-13674396
]
Ravi Prakash commented on YARN-169:
---
Hi! Could this please be merged into branch-0.23 as well?
Also 2 files are now dead code, should we remove them?
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics/jvm/EventCounter.java
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/log/EventCounter.java
Update log4j.appender.EventCounter to use
org.apache.hadoop.log.metrics.EventCounter
Key: YARN-169
URL: https://issues.apache.org/jira/browse/YARN-169
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Anthony Rojas
Assignee: Anthony Rojas
Priority: Minor
Fix For: 2.0.3-alpha
Attachments: YARN-169.patch
We should update the log4j.appender.EventCounter in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties
to use *org.apache.hadoop.log.metrics.EventCounter* rather than
*org.apache.hadoop.metrics.jvm.EventCounter* to avoid triggering the
following warning:
{code}WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated.
Please use org.apache.hadoop.log.metrics.EventCounter in all the
log4j.properties files{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-726) Queue, FinishTime fields broken on RM UI
[ https://issues.apache.org/jira/browse/YARN-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674407#comment-13674407 ] Hudson commented on YARN-726: - Integrated in Hadoop-Mapreduce-trunk #1446 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1446/]) YARN-726. Fix queue finish time fields in web-ui for ResourceManager. Contributed by Mayank Bansal. (Revision 1489234) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489234 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java Queue, FinishTime fields broken on RM UI Key: YARN-726 URL: https://issues.apache.org/jira/browse/YARN-726 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Siddharth Seth Assignee: Mayank Bansal Priority: Critical Fix For: 2.1.0-beta Attachments: YARN-726-trunk-1.patch The queue shows up as Invalid Date Finish Time shows up as a Long value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-530) Define Service model strictly, implement AbstractService for robust subclassing, migrate yarn-common services
[
https://issues.apache.org/jira/browse/YARN-530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-530:
Attachment: YARN-530-018.patch
vinod -thanks for trying to defeat findbugs. Here is your patch rebased for the
YARN-635 exception rename changes
Define Service model strictly, implement AbstractService for robust
subclassing, migrate yarn-common services
-
Key: YARN-530
URL: https://issues.apache.org/jira/browse/YARN-530
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.0.4-alpha
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: YARN-117changes.pdf, YARN-530-005.patch,
YARN-530-008.patch, YARN-530-009.patch, YARN-530-010.patch,
YARN-530-011.patch, YARN-530-012.patch, YARN-530-013.patch,
YARN-530-014.patch, YARN-530-015.patch, YARN-530-016.patch,
YARN-530-017.patch, YARN-530-018.patch, YARN-530-2.patch, YARN-530-3.patch,
YARN-530.4.patch, YARN-530.patch
# Extend the YARN {{Service}} interface as discussed in YARN-117
# Implement the changes in {{AbstractService}} and {{FilterService}}.
# Migrate all services in yarn-common to the more robust service model, test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-748) Move BuilderUtils from yarn-common to yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674405#comment-13674405 ] Hudson commented on YARN-748: - Integrated in Hadoop-Mapreduce-trunk #1446 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1446/]) YARN-748. Moved BuilderUtils from yarn-common to yarn-server-common for eventual retirement. Contributed by Jian He. MAPREDUCE-5297. Updated MR App since BuilderUtils is no longer public after YARN-748. Contributed by Jian He. (Revision 1489257) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489257 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestTaskHeartbeatHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttempt.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskAttemptContainerRequest.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/local/TestLocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/security/MRDelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRBuilderUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/api/records/TestIds.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestMapReduceTrackingUriPlugin.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryEvents.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHSWebApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/NotRunningJob.java *
[jira] [Commented] (YARN-635) Rename YarnRemoteException to YarnException
[ https://issues.apache.org/jira/browse/YARN-635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674409#comment-13674409 ] Hudson commented on YARN-635: - Integrated in Hadoop-Mapreduce-trunk #1446 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1446/]) YARN-635 and MAPREDUCE-5301. Bug fix as the committed patch ended up being stale. (Revision 1489287) YARN-635. Renamed YarnRemoteException to YarnException. Contributed by Siddharth Seth. MAPREDUCE-5301. Updated MR code to work with YARN-635 changes of renaming YarnRemoteException to YarnException. Contributed by Siddharth Seth (Revision 1489283) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489287 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryContext.java vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489283 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryCopyService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/commit/CommitterEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java *
[jira] [Commented] (YARN-398) Enhance CS to allow for white-list of resources
[ https://issues.apache.org/jira/browse/YARN-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674418#comment-13674418 ] Hudson commented on YARN-398: - Integrated in Hadoop-Mapreduce-trunk #1446 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1446/]) YARN-398. Make it possible to specify hard locality constraints in resource requests for CapacityScheduler. Contributed by Arun C. Murthy. (Revision 1489087) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489087 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java Enhance CS to allow for white-list of resources --- Key: YARN-398 URL: https://issues.apache.org/jira/browse/YARN-398 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 2.1.0-beta Attachments: YARN-398.patch, YARN-398.patch Allow white-list and black-list of resources in scheduler api. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-755) Rename AllocateResponse.reboot to AllocateResponse.resync
[ https://issues.apache.org/jira/browse/YARN-755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674404#comment-13674404 ] Hudson commented on YARN-755: - Integrated in Hadoop-Mapreduce-trunk #1446 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1446/]) YARN-755. Renamed AllocateResponse.reboot to AllocateResponse.resync. Contributed by Bikas Saha. (Revision 1489295) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489295 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/AllocateResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/AMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestAMRMClientAsync.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/utils/BuilderUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRMRPCResponseId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestApplicationTokens.java Rename AllocateResponse.reboot to AllocateResponse.resync - Key: YARN-755 URL: https://issues.apache.org/jira/browse/YARN-755 Project: Hadoop YARN Issue Type: Sub-task Affects Versions: 2.1.0-beta Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 2.1.0-beta Attachments: YARN-755.1.patch, YARN-755.2.patch For work preserving rm restart the am's will be resyncing instead of rebooting. rebooting is an action that currently satisfies the resync requirement. Changing the name now so that it continues to make sense in the real resync case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-117) Enhance YARN service model
[
https://issues.apache.org/jira/browse/YARN-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-117:
Attachment: YARN-117-018.patch
patch applying vinod's changes to YARN-530 and migration to the YARN-635
exception rename changes; in sync w/ YARN-530-018.patch
Enhance YARN service model
--
Key: YARN-117
URL: https://issues.apache.org/jira/browse/YARN-117
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.0.4-alpha
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: MAPREDUCE-5298-016.patch, YARN-117-007.patch,
YARN-117-008.patch, YARN-117-009.patch, YARN-117-010.patch,
YARN-117-011.patch, YARN-117-012.patch, YARN-117-013.patch,
YARN-117-014.patch, YARN-117-015.patch, YARN-117-016.patch,
YARN-117-018.patch, YARN-117-2.patch, YARN-117-3.patch, YARN-117.4.patch,
YARN-117.5.patch, YARN-117.6.patch, YARN-117.patch
Having played the YARN service model, there are some issues
that I've identified based on past work and initial use.
This JIRA issue is an overall one to cover the issues, with solutions pushed
out to separate JIRAs.
h2. state model prevents stopped state being entered if you could not
successfully start the service.
In the current lifecycle you cannot stop a service unless it was successfully
started, but
* {{init()}} may acquire resources that need to be explicitly released
* if the {{start()}} operation fails partway through, the {{stop()}}
operation may be needed to release resources.
*Fix:* make {{stop()}} a valid state transition from all states and require
the implementations to be able to stop safely without requiring all fields to
be non null.
Before anyone points out that the {{stop()}} operations assume that all
fields are valid; and if called before a {{start()}} they will NPE;
MAPREDUCE-3431 shows that this problem arises today, MAPREDUCE-3502 is a fix
for this. It is independent of the rest of the issues in this doc but it will
aid making {{stop()}} execute from all states other than stopped.
MAPREDUCE-3502 is too big a patch and needs to be broken down for easier
review and take up; this can be done with issues linked to this one.
h2. AbstractService doesn't prevent duplicate state change requests.
The {{ensureState()}} checks to verify whether or not a state transition is
allowed from the current state are performed in the base {{AbstractService}}
class -yet subclasses tend to call this *after* their own {{init()}},
{{start()}} {{stop()}} operations. This means that these operations can be
performed out of order, and even if the outcome of the call is an exception,
all actions performed by the subclasses will have taken place. MAPREDUCE-3877
demonstrates this.
This is a tricky one to address. In HADOOP-3128 I used a base class instead
of an interface and made the {{init()}}, {{start()}} {{stop()}} methods
{{final}}. These methods would do the checks, and then invoke protected inner
methods, {{innerStart()}}, {{innerStop()}}, etc. It should be possible to
retrofit the same behaviour to everything that extends {{AbstractService}}
-something that must be done before the class is considered stable (because
once the lifecycle methods are declared final, all subclasses that are out of
the source tree will need fixing by the respective developers.
h2. AbstractService state change doesn't defend against race conditions.
There's no concurrency locks on the state transitions. Whatever fix for wrong
state calls is added should correct this to prevent re-entrancy, such as
{{stop()}} being called from two threads.
h2. Static methods to choreograph of lifecycle operations
Helper methods to move things through lifecycles. init-start is common,
stop-if-service!=null another. Some static methods can execute these, and
even call {{stop()}} if {{init()}} raises an exception. These could go into a
class {{ServiceOps}} in the same package. These can be used by those services
that wrap other services, and help manage more robust shutdowns.
h2. state transition failures are something that registered service listeners
may wish to be informed of.
When a state transition fails a {{RuntimeException}} can be thrown -and the
service listeners are not informed as the notification point isn't reached.
They may wish to know this, especially for management and diagnostics.
*Fix:* extend {{ServiceStateChangeListener}} with a callback such as
{{stateChangeFailed(Service service,Service.State targeted-state,
RuntimeException e)}} that is invoked from the (final) state change methods
in the {{AbstractService}} class (once they delegate to their inner
{{innerStart()}}, {{innerStop()}} methods; make a no-op on the existing
implementations of the interface.
[jira] [Commented] (YARN-530) Define Service model strictly, implement AbstractService for robust subclassing, migrate yarn-common services
[
https://issues.apache.org/jira/browse/YARN-530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674435#comment-13674435
]
Hadoop QA commented on YARN-530:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586103/YARN-530-018.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1108//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/1108//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1108//console
This message is automatically generated.
Define Service model strictly, implement AbstractService for robust
subclassing, migrate yarn-common services
-
Key: YARN-530
URL: https://issues.apache.org/jira/browse/YARN-530
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.0.4-alpha
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: YARN-117changes.pdf, YARN-530-005.patch,
YARN-530-008.patch, YARN-530-009.patch, YARN-530-010.patch,
YARN-530-011.patch, YARN-530-012.patch, YARN-530-013.patch,
YARN-530-014.patch, YARN-530-015.patch, YARN-530-016.patch,
YARN-530-017.patch, YARN-530-018.patch, YARN-530-2.patch, YARN-530-3.patch,
YARN-530.4.patch, YARN-530.patch
# Extend the YARN {{Service}} interface as discussed in YARN-117
# Implement the changes in {{AbstractService}} and {{FilterService}}.
# Migrate all services in yarn-common to the more robust service model, test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-762) Javadoc params not matching in ContainerManagerImpl.authorizeRequest method
[ https://issues.apache.org/jira/browse/YARN-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niranjan Singh updated YARN-762: Attachment: YARN-762 Rectified Javadoc. Javadoc params not matching in ContainerManagerImpl.authorizeRequest method --- Key: YARN-762 URL: https://issues.apache.org/jira/browse/YARN-762 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Reporter: Niranjan Singh Priority: Minor Attachments: YARN-762 In ContainerManagerImpl.authorizeRequest method the number of parameters passed are four where as in Javadoc the params are only three. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-762) Javadoc params not matching in ContainerManagerImpl.authorizeRequest method
[ https://issues.apache.org/jira/browse/YARN-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niranjan Singh updated YARN-762: Attachment: (was: YARN-762) Javadoc params not matching in ContainerManagerImpl.authorizeRequest method --- Key: YARN-762 URL: https://issues.apache.org/jira/browse/YARN-762 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Reporter: Niranjan Singh Priority: Minor Attachments: YARN-762.patch In ContainerManagerImpl.authorizeRequest method the number of parameters passed are four where as in Javadoc the params are only three. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-762) Javadoc params not matching in ContainerManagerImpl.authorizeRequest method
[ https://issues.apache.org/jira/browse/YARN-762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niranjan Singh updated YARN-762: Attachment: YARN-762.patch Missed file extension .patch. Added again. Javadoc params not matching in ContainerManagerImpl.authorizeRequest method --- Key: YARN-762 URL: https://issues.apache.org/jira/browse/YARN-762 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Reporter: Niranjan Singh Priority: Minor Attachments: YARN-762.patch In ContainerManagerImpl.authorizeRequest method the number of parameters passed are four where as in Javadoc the params are only three. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-750) Allow for black-listing resources in CS
[ https://issues.apache.org/jira/browse/YARN-750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated YARN-750: --- Attachment: YARN-750.patch Fixed javadoc warning (broke during rebase after YARN-635) and NPE in tests. Allow for black-listing resources in CS --- Key: YARN-750 URL: https://issues.apache.org/jira/browse/YARN-750 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Arun C Murthy Attachments: YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch YARN-392 and YARN-398 enhance scheduler api to allow for white-lists of resources. This jira is a companion to allow for black-listing (in CS). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-750) Allow for black-listing resources in CS
[
https://issues.apache.org/jira/browse/YARN-750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674461#comment-13674461
]
Thomas Graves commented on YARN-750:
javadocs warnings are complaining about + * {@link ResourceRequest#ANY} in
InvalidBlacklistRequestException.java
.
Allow for black-listing resources in CS
---
Key: YARN-750
URL: https://issues.apache.org/jira/browse/YARN-750
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Attachments: YARN-750.patch, YARN-750.patch, YARN-750.patch,
YARN-750.patch, YARN-750.patch, YARN-750.patch
YARN-392 and YARN-398 enhance scheduler api to allow for white-lists of
resources.
This jira is a companion to allow for black-listing (in CS).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (YARN-750) Allow for black-listing resources in CS
[
https://issues.apache.org/jira/browse/YARN-750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674461#comment-13674461
]
Thomas Graves edited comment on YARN-750 at 6/4/13 2:59 PM:
javadocs warnings are complaining about + * {@link ResourceRequest#ANY} in
InvalidBlacklistRequestException.java
.
Update: Ignore looks like Arun updated patch as I was commenting.
was (Author: tgraves):
javadocs warnings are complaining about + * {@link ResourceRequest#ANY} in
InvalidBlacklistRequestException.java
.
Allow for black-listing resources in CS
---
Key: YARN-750
URL: https://issues.apache.org/jira/browse/YARN-750
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Attachments: YARN-750.patch, YARN-750.patch, YARN-750.patch,
YARN-750.patch, YARN-750.patch, YARN-750.patch
YARN-392 and YARN-398 enhance scheduler api to allow for white-lists of
resources.
This jira is a companion to allow for black-listing (in CS).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-750) Allow for black-listing resources in CS
[
https://issues.apache.org/jira/browse/YARN-750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674481#comment-13674481
]
Hadoop QA commented on YARN-750:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586113/YARN-750.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 13 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1109//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1109//console
This message is automatically generated.
Allow for black-listing resources in CS
---
Key: YARN-750
URL: https://issues.apache.org/jira/browse/YARN-750
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Attachments: YARN-750.patch, YARN-750.patch, YARN-750.patch,
YARN-750.patch, YARN-750.patch, YARN-750.patch
YARN-392 and YARN-398 enhance scheduler api to allow for white-lists of
resources.
This jira is a companion to allow for black-listing (in CS).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-750) Allow for black-listing resources in CS
[ https://issues.apache.org/jira/browse/YARN-750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674527#comment-13674527 ] Thomas Graves commented on YARN-750: Can we make BlacklistRequestPBImpl immutable since we are changing that in other places (YARN-735)? Allow for black-listing resources in CS --- Key: YARN-750 URL: https://issues.apache.org/jira/browse/YARN-750 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Arun C Murthy Attachments: YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch YARN-392 and YARN-398 enhance scheduler api to allow for white-lists of resources. This jira is a companion to allow for black-listing (in CS). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-762) Javadoc params not matching in ContainerManagerImpl.authorizeRequest method
[ https://issues.apache.org/jira/browse/YARN-762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674575#comment-13674575 ] Omkar Vinit Joshi commented on YARN-762: [~niranjanmaisnam] thanks niranjan... probably we will have to update it even more after YARN-613 Javadoc params not matching in ContainerManagerImpl.authorizeRequest method --- Key: YARN-762 URL: https://issues.apache.org/jira/browse/YARN-762 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Reporter: Niranjan Singh Priority: Minor Attachments: YARN-762.patch In ContainerManagerImpl.authorizeRequest method the number of parameters passed are four where as in Javadoc the params are only three. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-750) Allow for black-listing resources in CS
[
https://issues.apache.org/jira/browse/YARN-750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13674582#comment-13674582
]
Bikas Saha commented on YARN-750:
-
+1. Looks good overall. Need a jira for FIFO scheduler changes.
Minor comments.
Empty block of code?
{code}
+
+ if (request.getBlacklistRequest() != null) {
+
+ }
// sanity check
{code}
Probably would be better if javadoc treated blacklisting * as one case of an
invalid blacklist exception. unlikely that we will be creating multiple such
exceptions.
{code}
+ * The exception is thrown when an application tries to blacklist
+ * {@link ResourceRequest#ANY}.
+ */
+public class InvalidBlacklistRequestException extends YarnRuntimeException {
{code}
TestApplicationTokens - change probably isnt needed
{code}
allocateRequest.setApplicationAttemptId(applicationAttemptId);
+ allocateRequest.setBlacklistRequest(
+ Records.newRecord(BlacklistRequest.class));
{code}
Allow for black-listing resources in CS
---
Key: YARN-750
URL: https://issues.apache.org/jira/browse/YARN-750
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Attachments: YARN-750.patch, YARN-750.patch, YARN-750.patch,
YARN-750.patch, YARN-750.patch, YARN-750.patch
YARN-392 and YARN-398 enhance scheduler api to allow for white-lists of
resources.
This jira is a companion to allow for black-listing (in CS).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (YARN-756) Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract to api.records
[ https://issues.apache.org/jira/browse/YARN-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He reopened YARN-756: -- *PBImp should go to package impl.pb Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract to api.records Key: YARN-756 URL: https://issues.apache.org/jira/browse/YARN-756 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: YARN-756.1.patch, YARN-756.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-756) Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract to api.records
[ https://issues.apache.org/jira/browse/YARN-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-756: - Attachment: YARN-756.3.patch simple eclipse from package impl to impl.pb Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract to api.records Key: YARN-756 URL: https://issues.apache.org/jira/browse/YARN-756 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: YARN-756.1.patch, YARN-756.2.patch, YARN-756.3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-750) Allow for black-listing resources in CS
[ https://issues.apache.org/jira/browse/YARN-750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated YARN-750: --- Attachment: YARN-750.patch Allow for black-listing resources in CS --- Key: YARN-750 URL: https://issues.apache.org/jira/browse/YARN-750 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Arun C Murthy Attachments: YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch YARN-392 and YARN-398 enhance scheduler api to allow for white-lists of resources. This jira is a companion to allow for black-listing (in CS). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-689) Add multiplier unit to resourcecapabilities
[ https://issues.apache.org/jira/browse/YARN-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675121#comment-13675121 ] Timothy St. Clair commented on YARN-689: Just to interject again, but whenever a scheduler sub-divides resources, there will eventually need to reside some separate deamon/tool/policy to re-balance/defrag your cluster. I haven't really seen any JIRA's around it, so I'm also curious how that is intended to be dealt with. Add multiplier unit to resourcecapabilities --- Key: YARN-689 URL: https://issues.apache.org/jira/browse/YARN-689 Project: Hadoop YARN Issue Type: Bug Components: api, scheduler Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-689.patch, YARN-689.patch, YARN-689.patch, YARN-689.patch, YARN-689.patch Currently we overloading the minimum resource value as the actual multiplier used by the scheduler. Today with a minimum memory set to 1GB, requests for 1.5GB are always translated to allocation of 2GB. We should decouple the minimum allocation from the multiplier. The multiplier should also be exposed to the client via the RegisterApplicationMasterResponse -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-742) Log aggregation causes a lot of redundant setPermission calls
[ https://issues.apache.org/jira/browse/YARN-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675126#comment-13675126 ] Kihwal Lee commented on YARN-742: - There were other changes made in trunk so the patch doesn't apply any more. Please update the patch. Log aggregation causes a lot of redundant setPermission calls - Key: YARN-742 URL: https://issues.apache.org/jira/browse/YARN-742 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Kihwal Lee Assignee: Jason Lowe Attachments: YARN-742.patch In one of our clusters, namenode RPC is spending 45% of its time on serving setPermission calls. Further investigation has revealed that most calls are redundantly made on /mapred/logs/user/logs. Also mkdirs calls are made before this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-514) Delayed store operations should not result in RM unavailability for app submission
[ https://issues.apache.org/jira/browse/YARN-514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-514: - Component/s: resourcemanager Delayed store operations should not result in RM unavailability for app submission -- Key: YARN-514 URL: https://issues.apache.org/jira/browse/YARN-514 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Bikas Saha Assignee: Zhijie Shen Fix For: 2.1.0-beta Attachments: YARN-514.1.patch, YARN-514.2.patch, YARN-514.3.patch, YARN-514.4.patch, YARN-514.5.patch, YARN-514.6.patch, YARN-514.7.patch, YARN-514.8.patch Currently, app submission is the only store operation performed synchronously because the app must be stored before the request returns with success. This makes the RM susceptible to blocking all client threads on slow store operations, resulting in RM being perceived as unavailable by clients. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-756) Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract to api.records
[ https://issues.apache.org/jira/browse/YARN-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-756: - Attachment: YARN-756.5.patch Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract to api.records Key: YARN-756 URL: https://issues.apache.org/jira/browse/YARN-756 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: YARN-756.1.patch, YARN-756.2.patch, YARN-756.3.patch, YARN-756.5.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-756) Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract/PreemptionResourceRequest to api.records
[ https://issues.apache.org/jira/browse/YARN-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-756: - Summary: Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract/PreemptionResourceRequest to api.records (was: Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract to api.records) Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract/PreemptionResourceRequest to api.records -- Key: YARN-756 URL: https://issues.apache.org/jira/browse/YARN-756 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: YARN-756.1.patch, YARN-756.2.patch, YARN-756.3.patch, YARN-756.5.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-756) Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract/PreemptionResourceRequest to api.records
[ https://issues.apache.org/jira/browse/YARN-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675182#comment-13675182 ] Jian He commented on YARN-756: -- The newest patch also moved PreemptionResourceRequest to api.record, and move the preemption records to yarn_protos.proto Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract/PreemptionResourceRequest to api.records -- Key: YARN-756 URL: https://issues.apache.org/jira/browse/YARN-756 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: YARN-756.1.patch, YARN-756.2.patch, YARN-756.3.patch, YARN-756.5.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-169) Update log4j.appender.EventCounter to use org.apache.hadoop.log.metrics.EventCounter
[
https://issues.apache.org/jira/browse/YARN-169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated YARN-169:
-
Fix Version/s: 0.23.9
Update log4j.appender.EventCounter to use
org.apache.hadoop.log.metrics.EventCounter
Key: YARN-169
URL: https://issues.apache.org/jira/browse/YARN-169
Project: Hadoop YARN
Issue Type: Improvement
Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Anthony Rojas
Assignee: Anthony Rojas
Priority: Minor
Fix For: 2.0.3-alpha, 0.23.9
Attachments: YARN-169.patch
We should update the log4j.appender.EventCounter in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/container-log4j.properties
to use *org.apache.hadoop.log.metrics.EventCounter* rather than
*org.apache.hadoop.metrics.jvm.EventCounter* to avoid triggering the
following warning:
{code}WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated.
Please use org.apache.hadoop.log.metrics.EventCounter in all the
log4j.properties files{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-742) Log aggregation causes a lot of redundant setPermission calls
[
https://issues.apache.org/jira/browse/YARN-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675196#comment-13675196
]
Hadoop QA commented on YARN-742:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586157/YARN-742-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1112//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1112//console
This message is automatically generated.
Log aggregation causes a lot of redundant setPermission calls
-
Key: YARN-742
URL: https://issues.apache.org/jira/browse/YARN-742
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 0.23.7, 2.0.4-alpha
Reporter: Kihwal Lee
Assignee: Jason Lowe
Attachments: YARN-742-1.patch, YARN-742.patch
In one of our clusters, namenode RPC is spending 45% of its time on serving
setPermission calls. Further investigation has revealed that most calls are
redundantly made on /mapred/logs/user/logs. Also mkdirs calls are made
before this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-750) Allow for black-listing resources in CS
[
https://issues.apache.org/jira/browse/YARN-750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675208#comment-13675208
]
Hadoop QA commented on YARN-750:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586152/YARN-750.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build///testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build///console
This message is automatically generated.
Allow for black-listing resources in CS
---
Key: YARN-750
URL: https://issues.apache.org/jira/browse/YARN-750
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Attachments: YARN-750.patch, YARN-750.patch, YARN-750.patch,
YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch
YARN-392 and YARN-398 enhance scheduler api to allow for white-lists of
resources.
This jira is a companion to allow for black-listing (in CS).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-753) Add individual factory method for api protocol records
[ https://issues.apache.org/jira/browse/YARN-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-753: - Attachment: YARN-753.2.patch Add individual factory method for api protocol records -- Key: YARN-753 URL: https://issues.apache.org/jira/browse/YARN-753 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: YARN-753.1.patch, YARN-753.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-756) Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract/PreemptionResourceRequest to api.records
[
https://issues.apache.org/jira/browse/YARN-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675232#comment-13675232
]
Hadoop QA commented on YARN-756:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586162/YARN-756.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1113//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1113//console
This message is automatically generated.
Move
PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract/PreemptionResourceRequest
to api.records
--
Key: YARN-756
URL: https://issues.apache.org/jira/browse/YARN-756
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Jian He
Assignee: Jian He
Fix For: 2.1.0-beta
Attachments: YARN-756.1.patch, YARN-756.2.patch, YARN-756.3.patch,
YARN-756.5.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-742) Log aggregation causes a lot of redundant setPermission calls
[ https://issues.apache.org/jira/browse/YARN-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675230#comment-13675230 ] Hudson commented on YARN-742: - Integrated in Hadoop-trunk-Commit #3858 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3858/]) YARN-742. Log aggregation causes a lot of redundant setPermission calls. Contributed by Jason Lowe. (Revision 1489596) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489596 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/LogAggregationService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java Log aggregation causes a lot of redundant setPermission calls - Key: YARN-742 URL: https://issues.apache.org/jira/browse/YARN-742 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Kihwal Lee Assignee: Jason Lowe Attachments: YARN-742-1.patch, YARN-742.patch In one of our clusters, namenode RPC is spending 45% of its time on serving setPermission calls. Further investigation has revealed that most calls are redundantly made on /mapred/logs/user/logs. Also mkdirs calls are made before this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-742) Log aggregation causes a lot of redundant setPermission calls
[ https://issues.apache.org/jira/browse/YARN-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675262#comment-13675262 ] Kihwal Lee commented on YARN-742: - Jason, the test failed to merge in branch-0.23. Log aggregation causes a lot of redundant setPermission calls - Key: YARN-742 URL: https://issues.apache.org/jira/browse/YARN-742 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Kihwal Lee Assignee: Jason Lowe Attachments: YARN-742-1.patch, YARN-742.patch In one of our clusters, namenode RPC is spending 45% of its time on serving setPermission calls. Further investigation has revealed that most calls are redundantly made on /mapred/logs/user/logs. Also mkdirs calls are made before this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-689) Add multiplier unit to resourcecapabilities
[ https://issues.apache.org/jira/browse/YARN-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675281#comment-13675281 ] Alejandro Abdelnur commented on YARN-689: - [~acmurthy]. Can you explain why if minimum and increment are the same (as it is today), and their value is set to a low number (i.e. 128m) the 'fragmentation' problem does not arise? Also, why do you see 'fragmentation' as an issue? We are not talking about a fragmented memory that we won't be able to allocate because it is in multiple chunks, if there is capacity it can be used, there is no memory fragmentation. Add multiplier unit to resourcecapabilities --- Key: YARN-689 URL: https://issues.apache.org/jira/browse/YARN-689 Project: Hadoop YARN Issue Type: Bug Components: api, scheduler Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: YARN-689.patch, YARN-689.patch, YARN-689.patch, YARN-689.patch, YARN-689.patch Currently we overloading the minimum resource value as the actual multiplier used by the scheduler. Today with a minimum memory set to 1GB, requests for 1.5GB are always translated to allocation of 2GB. We should decouple the minimum allocation from the multiplier. The multiplier should also be exposed to the client via the RegisterApplicationMasterResponse -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-756) Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract/PreemptionResourceRequest to api.records
[ https://issues.apache.org/jira/browse/YARN-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675282#comment-13675282 ] Hudson commented on YARN-756: - Integrated in Hadoop-trunk-Commit #3859 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3859/]) YARN-756. Remnants from moving Preemption* records to yarn.api where they really belong. Contributed by Jian He. (Revision 1489614) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489614 Files : * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/PreemptionResourceRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/AllocateResponsePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/impl/pb/PreemptionResourceRequestPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/PreemptionContract.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/PreemptionResourceRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/PreemptionContainerPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/PreemptionContractPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/PreemptionMessagePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/StrictPreemptionContractPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/PreemptionContainerPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/PreemptionContractPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/PreemptionMessagePBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/PreemptionResourceRequestPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/StrictPreemptionContractPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/proto/yarn_service_protos.proto * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java Move PreemptionContainer/PremptionContract/PreemptionMessage/StrictPreemptionContract/PreemptionResourceRequest to api.records -- Key: YARN-756 URL: https://issues.apache.org/jira/browse/YARN-756 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: YARN-756.1.patch, YARN-756.2.patch, YARN-756.3.patch, YARN-756.5.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-753) Add individual factory method for api protocol records
[
https://issues.apache.org/jira/browse/YARN-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675297#comment-13675297
]
Hadoop QA commented on YARN-753:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586170/YARN-753.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1114//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1114//console
This message is automatically generated.
Add individual factory method for api protocol records
--
Key: YARN-753
URL: https://issues.apache.org/jira/browse/YARN-753
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-753.1.patch, YARN-753.2.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-724) Move ProtoBase from api.records to api.records.impl.pb
[ https://issues.apache.org/jira/browse/YARN-724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-724: - Attachment: YARN-724.1.patch Move ProtoBase from api.records to api.records.impl.pb -- Key: YARN-724 URL: https://issues.apache.org/jira/browse/YARN-724 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: YARN-724.1.patch Simply move ProtoBase to records.impl.pb -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-245) Node Manager gives InvalidStateTransitonException for FINISH_APPLICATION at FINISHED
[
https://issues.apache.org/jira/browse/YARN-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayank Bansal updated YARN-245:
---
Attachment: YARN-245-trunk-1.patch
Attaching patch.
Thanks,
Mayank
Node Manager gives InvalidStateTransitonException for FINISH_APPLICATION at
FINISHED
Key: YARN-245
URL: https://issues.apache.org/jira/browse/YARN-245
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.0.2-alpha, 2.0.1-alpha
Reporter: Devaraj K
Assignee: Mayank Bansal
Attachments: YARN-245-trunk-1.patch
{code:xml}
2012-11-25 12:56:11,795 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
FINISH_APPLICATION at FINISHED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:398)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:58)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:520)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:512)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
2012-11-25 12:56:11,796 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1353818859056_0004 transitioned from FINISHED to null
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-245) Node Manager gives InvalidStateTransitonException for FINISH_APPLICATION at FINISHED
[
https://issues.apache.org/jira/browse/YARN-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayank Bansal reassigned YARN-245:
--
Assignee: Mayank Bansal (was: Devaraj K)
Node Manager gives InvalidStateTransitonException for FINISH_APPLICATION at
FINISHED
Key: YARN-245
URL: https://issues.apache.org/jira/browse/YARN-245
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.0.2-alpha, 2.0.1-alpha
Reporter: Devaraj K
Assignee: Mayank Bansal
Attachments: YARN-245-trunk-1.patch
{code:xml}
2012-11-25 12:56:11,795 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
FINISH_APPLICATION at FINISHED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:398)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:58)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:520)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:512)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
2012-11-25 12:56:11,796 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1353818859056_0004 transitioned from FINISHED to null
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-295) Resource Manager throws InvalidStateTransitonException: Invalid event: CONTAINER_FINISHED at ALLOCATED for RMAppAttemptImpl
[
https://issues.apache.org/jira/browse/YARN-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayank Bansal reassigned YARN-295:
--
Assignee: Mayank Bansal
Resource Manager throws InvalidStateTransitonException: Invalid event:
CONTAINER_FINISHED at ALLOCATED for RMAppAttemptImpl
---
Key: YARN-295
URL: https://issues.apache.org/jira/browse/YARN-295
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Affects Versions: 2.0.2-alpha, 2.0.1-alpha
Reporter: Devaraj K
Assignee: Mayank Bansal
{code:xml}
2012-12-28 14:03:56,956 ERROR
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
CONTAINER_FINISHED at ALLOCATED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:490)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:80)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:433)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:414)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-295) Resource Manager throws InvalidStateTransitonException: Invalid event: CONTAINER_FINISHED at ALLOCATED for RMAppAttemptImpl
[
https://issues.apache.org/jira/browse/YARN-295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675302#comment-13675302
]
Mayank Bansal commented on YARN-295:
Working on it
Resource Manager throws InvalidStateTransitonException: Invalid event:
CONTAINER_FINISHED at ALLOCATED for RMAppAttemptImpl
---
Key: YARN-295
URL: https://issues.apache.org/jira/browse/YARN-295
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Affects Versions: 2.0.2-alpha, 2.0.1-alpha
Reporter: Devaraj K
Assignee: Mayank Bansal
{code:xml}
2012-12-28 14:03:56,956 ERROR
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
CONTAINER_FINISHED at ALLOCATED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:490)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:80)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:433)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:414)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-748) Move BuilderUtils from yarn-common to yarn-server-common
[ https://issues.apache.org/jira/browse/YARN-748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-748: - Issue Type: Sub-task (was: Bug) Parent: YARN-386 Move BuilderUtils from yarn-common to yarn-server-common Key: YARN-748 URL: https://issues.apache.org/jira/browse/YARN-748 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: YARN-748.1.patch, YARN-748.2.patch, YARN-748.3.patch, YARN-748.4.patch, YARN-748.5.patch, YARN-748.includeMR-branch-2.patch, YARN-748.includeMR.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-753) Add individual factory method for api protocol records
[ https://issues.apache.org/jira/browse/YARN-753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-753: - Issue Type: Sub-task (was: Bug) Parent: YARN-386 Add individual factory method for api protocol records -- Key: YARN-753 URL: https://issues.apache.org/jira/browse/YARN-753 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: YARN-753.1.patch, YARN-753.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-735) Make ApplicationAttemptID, ContainerID, NodeID immutable
[ https://issues.apache.org/jira/browse/YARN-735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-735: - Issue Type: Sub-task (was: Bug) Parent: YARN-386 Make ApplicationAttemptID, ContainerID, NodeID immutable Key: YARN-735 URL: https://issues.apache.org/jira/browse/YARN-735 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: YARN-735.1.patch, YARN-735.2.patch, YARN-735.2.patch, YARN-735.3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-737) ContainerManagerImpl can directly throw NMNotYetReadyException and InvalidContainerException after YARN-142
[ https://issues.apache.org/jira/browse/YARN-737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-737: - Issue Type: Sub-task (was: Bug) Parent: YARN-386 ContainerManagerImpl can directly throw NMNotYetReadyException and InvalidContainerException after YARN-142 Key: YARN-737 URL: https://issues.apache.org/jira/browse/YARN-737 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Attachments: YARN-737.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-753) Add individual factory method for api protocol records
[ https://issues.apache.org/jira/browse/YARN-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675326#comment-13675326 ] Hudson commented on YARN-753: - Integrated in Hadoop-trunk-Commit #3861 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3861/]) YARN-753. Added individual factory methods for all api protocol records and converted the records to be abstract classes. Contributed by Jian He. (Revision 1489644) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489644 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/CancelDelegationTokenRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/CancelDelegationTokenResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/FinishApplicationMasterRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/FinishApplicationMasterResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetAllApplicationsRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetAllApplicationsResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationReportRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetApplicationReportResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetClusterMetricsRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetClusterMetricsResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetClusterNodesRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetClusterNodesResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetContainerStatusRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetContainerStatusResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetDelegationTokenRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetDelegationTokenResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetNewApplicationRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetNewApplicationResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetQueueInfoRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetQueueInfoResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetQueueUserAclsInfoRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/GetQueueUserAclsInfoResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/KillApplicationRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/KillApplicationResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RefreshAdminAclsRequest.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RefreshAdminAclsResponse.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/protocolrecords/RefreshNodesRequest.java *
[jira] [Commented] (YARN-245) Node Manager gives InvalidStateTransitonException for FINISH_APPLICATION at FINISHED
[
https://issues.apache.org/jira/browse/YARN-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675332#comment-13675332
]
Hadoop QA commented on YARN-245:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586186/YARN-245-trunk-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1115//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1115//console
This message is automatically generated.
Node Manager gives InvalidStateTransitonException for FINISH_APPLICATION at
FINISHED
Key: YARN-245
URL: https://issues.apache.org/jira/browse/YARN-245
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.0.2-alpha, 2.0.1-alpha
Reporter: Devaraj K
Assignee: Mayank Bansal
Attachments: YARN-245-trunk-1.patch
{code:xml}
2012-11-25 12:56:11,795 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
FINISH_APPLICATION at FINISHED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:398)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.handle(ApplicationImpl.java:58)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:520)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ApplicationEventDispatcher.handle(ContainerManagerImpl.java:512)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
2012-11-25 12:56:11,796 INFO
org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application:
Application application_1353818859056_0004 transitioned from FINISHED to null
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-742) Log aggregation causes a lot of redundant setPermission calls
[ https://issues.apache.org/jira/browse/YARN-742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated YARN-742: Attachment: YARN-742-1.branch-0.23.patch Thanks for the review, Kihwal, and apologies with the patch issues. Here's a patch for branch-0.23 Log aggregation causes a lot of redundant setPermission calls - Key: YARN-742 URL: https://issues.apache.org/jira/browse/YARN-742 Project: Hadoop YARN Issue Type: Bug Components: nodemanager Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Kihwal Lee Assignee: Jason Lowe Attachments: YARN-742-1.branch-0.23.patch, YARN-742-1.patch, YARN-742.patch In one of our clusters, namenode RPC is spending 45% of its time on serving setPermission calls. Further investigation has revealed that most calls are redundantly made on /mapred/logs/user/logs. Also mkdirs calls are made before this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-742) Log aggregation causes a lot of redundant setPermission calls
[
https://issues.apache.org/jira/browse/YARN-742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675337#comment-13675337
]
Hadoop QA commented on YARN-742:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586204/YARN-742-1.branch-0.23.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1117//console
This message is automatically generated.
Log aggregation causes a lot of redundant setPermission calls
-
Key: YARN-742
URL: https://issues.apache.org/jira/browse/YARN-742
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 0.23.7, 2.0.4-alpha
Reporter: Kihwal Lee
Assignee: Jason Lowe
Attachments: YARN-742-1.branch-0.23.patch, YARN-742-1.patch,
YARN-742.patch
In one of our clusters, namenode RPC is spending 45% of its time on serving
setPermission calls. Further investigation has revealed that most calls are
redundantly made on /mapred/logs/user/logs. Also mkdirs calls are made
before this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-724) Move ProtoBase from api.records to api.records.impl.pb
[
https://issues.apache.org/jira/browse/YARN-724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675352#comment-13675352
]
Hadoop QA commented on YARN-724:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586185/YARN-724.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1116//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1116//console
This message is automatically generated.
Move ProtoBase from api.records to api.records.impl.pb
--
Key: YARN-724
URL: https://issues.apache.org/jira/browse/YARN-724
Project: Hadoop YARN
Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
Attachments: YARN-724.1.patch
Simply move ProtoBase to records.impl.pb
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-724) Move ProtoBase from api.records to api.records.impl.pb
[ https://issues.apache.org/jira/browse/YARN-724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675360#comment-13675360 ] Vinod Kumar Vavilapalli commented on YARN-724: -- +1 for the trivial move. Checking this in. Move ProtoBase from api.records to api.records.impl.pb -- Key: YARN-724 URL: https://issues.apache.org/jira/browse/YARN-724 Project: Hadoop YARN Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: YARN-724.1.patch Simply move ProtoBase to records.impl.pb -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-724) Move ProtoBase from api.records to api.records.impl.pb
[ https://issues.apache.org/jira/browse/YARN-724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-724: - Issue Type: Sub-task (was: Bug) Parent: YARN-386 Move ProtoBase from api.records to api.records.impl.pb -- Key: YARN-724 URL: https://issues.apache.org/jira/browse/YARN-724 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jian He Assignee: Jian He Fix For: 2.1.0-beta Attachments: YARN-724.1.patch Simply move ProtoBase to records.impl.pb -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-724) Move ProtoBase from api.records to api.records.impl.pb
[ https://issues.apache.org/jira/browse/YARN-724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675386#comment-13675386 ] Hudson commented on YARN-724: - Integrated in Hadoop-trunk-Commit #3862 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3862/]) YARN-724. Moved ProtoBase from api.records to api.records.impl.pb. Contributed by Jian He. MAPREDUCE-5303. Changed MR app after moving ProtoBase to package impl.pb via YARN-724. Contributed by Jian He. (Revision 1489658) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1489658 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/CancelDelegationTokenRequestPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/CancelDelegationTokenResponsePBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/FailTaskAttemptRequestPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/FailTaskAttemptResponsePBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/GetCountersRequestPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/GetCountersResponsePBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/GetDelegationTokenRequestPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/GetDelegationTokenResponsePBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/GetDiagnosticsRequestPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/GetDiagnosticsResponsePBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/GetJobReportRequestPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/GetJobReportResponsePBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/GetTaskAttemptCompletionEventsRequestPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/GetTaskAttemptCompletionEventsResponsePBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/GetTaskAttemptReportRequestPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/GetTaskAttemptReportResponsePBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/GetTaskReportRequestPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/GetTaskReportResponsePBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/api/protocolrecords/impl/pb/GetTaskReportsRequestPBImpl.java *
[jira] [Updated] (YARN-530) Define Service model strictly, implement AbstractService for robust subclassing, migrate yarn-common services
[
https://issues.apache.org/jira/browse/YARN-530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-530:
Attachment: YARN-117-019.patch
# make sure addService, removeService and service iteration are all thread safe
-eliminates risk of concurrency problems if the service list is changed during
init/start/stop of children. As these methods are protected, a subclass would
have to do this in another thread -or make the methods public
# composite service policy is switched to stop all services that are STARTED or
INITED
Define Service model strictly, implement AbstractService for robust
subclassing, migrate yarn-common services
-
Key: YARN-530
URL: https://issues.apache.org/jira/browse/YARN-530
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.0.4-alpha
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: YARN-117-019.patch, YARN-117changes.pdf,
YARN-530-005.patch, YARN-530-008.patch, YARN-530-009.patch,
YARN-530-010.patch, YARN-530-011.patch, YARN-530-012.patch,
YARN-530-013.patch, YARN-530-014.patch, YARN-530-015.patch,
YARN-530-016.patch, YARN-530-017.patch, YARN-530-018.patch, YARN-530-2.patch,
YARN-530-3.patch, YARN-530.4.patch, YARN-530.patch
# Extend the YARN {{Service}} interface as discussed in YARN-117
# Implement the changes in {{AbstractService}} and {{FilterService}}.
# Migrate all services in yarn-common to the more robust service model, test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-117) Enhance YARN service model
[
https://issues.apache.org/jira/browse/YARN-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-117:
Attachment: (was: MAPREDUCE-5298-016.patch)
Enhance YARN service model
--
Key: YARN-117
URL: https://issues.apache.org/jira/browse/YARN-117
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.0.4-alpha
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: YARN-117-007.patch, YARN-117-008.patch,
YARN-117-009.patch, YARN-117-010.patch, YARN-117-011.patch,
YARN-117-012.patch, YARN-117-013.patch, YARN-117-014.patch,
YARN-117-015.patch, YARN-117-016.patch, YARN-117-018.patch,
YARN-117-019.patch, YARN-117-2.patch, YARN-117-3.patch, YARN-117.4.patch,
YARN-117.5.patch, YARN-117.6.patch, YARN-117.patch
Having played the YARN service model, there are some issues
that I've identified based on past work and initial use.
This JIRA issue is an overall one to cover the issues, with solutions pushed
out to separate JIRAs.
h2. state model prevents stopped state being entered if you could not
successfully start the service.
In the current lifecycle you cannot stop a service unless it was successfully
started, but
* {{init()}} may acquire resources that need to be explicitly released
* if the {{start()}} operation fails partway through, the {{stop()}}
operation may be needed to release resources.
*Fix:* make {{stop()}} a valid state transition from all states and require
the implementations to be able to stop safely without requiring all fields to
be non null.
Before anyone points out that the {{stop()}} operations assume that all
fields are valid; and if called before a {{start()}} they will NPE;
MAPREDUCE-3431 shows that this problem arises today, MAPREDUCE-3502 is a fix
for this. It is independent of the rest of the issues in this doc but it will
aid making {{stop()}} execute from all states other than stopped.
MAPREDUCE-3502 is too big a patch and needs to be broken down for easier
review and take up; this can be done with issues linked to this one.
h2. AbstractService doesn't prevent duplicate state change requests.
The {{ensureState()}} checks to verify whether or not a state transition is
allowed from the current state are performed in the base {{AbstractService}}
class -yet subclasses tend to call this *after* their own {{init()}},
{{start()}} {{stop()}} operations. This means that these operations can be
performed out of order, and even if the outcome of the call is an exception,
all actions performed by the subclasses will have taken place. MAPREDUCE-3877
demonstrates this.
This is a tricky one to address. In HADOOP-3128 I used a base class instead
of an interface and made the {{init()}}, {{start()}} {{stop()}} methods
{{final}}. These methods would do the checks, and then invoke protected inner
methods, {{innerStart()}}, {{innerStop()}}, etc. It should be possible to
retrofit the same behaviour to everything that extends {{AbstractService}}
-something that must be done before the class is considered stable (because
once the lifecycle methods are declared final, all subclasses that are out of
the source tree will need fixing by the respective developers.
h2. AbstractService state change doesn't defend against race conditions.
There's no concurrency locks on the state transitions. Whatever fix for wrong
state calls is added should correct this to prevent re-entrancy, such as
{{stop()}} being called from two threads.
h2. Static methods to choreograph of lifecycle operations
Helper methods to move things through lifecycles. init-start is common,
stop-if-service!=null another. Some static methods can execute these, and
even call {{stop()}} if {{init()}} raises an exception. These could go into a
class {{ServiceOps}} in the same package. These can be used by those services
that wrap other services, and help manage more robust shutdowns.
h2. state transition failures are something that registered service listeners
may wish to be informed of.
When a state transition fails a {{RuntimeException}} can be thrown -and the
service listeners are not informed as the notification point isn't reached.
They may wish to know this, especially for management and diagnostics.
*Fix:* extend {{ServiceStateChangeListener}} with a callback such as
{{stateChangeFailed(Service service,Service.State targeted-state,
RuntimeException e)}} that is invoked from the (final) state change methods
in the {{AbstractService}} class (once they delegate to their inner
{{innerStart()}}, {{innerStop()}} methods; make a no-op on the existing
implementations of the interface.
h2. Service listener failures not handled
Is this an error an error or not? Log and ignore may not be what is desired.
*Proposed:*
[jira] [Updated] (YARN-530) Define Service model strictly, implement AbstractService for robust subclassing, migrate yarn-common services
[
https://issues.apache.org/jira/browse/YARN-530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-530:
Attachment: YARN-530-019.patch
# make sure addService, removeService and service iteration are all thread safe
-eliminates risk of concurrency problems if the service list is changed during
init/start/stop of children. As these methods are protected, a subclass would
have to do this in another thread -or make the methods public
# composite service policy is switched to stop all services that are STARTED or
INITED
Define Service model strictly, implement AbstractService for robust
subclassing, migrate yarn-common services
-
Key: YARN-530
URL: https://issues.apache.org/jira/browse/YARN-530
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.0.4-alpha
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: YARN-117-019.patch, YARN-117changes.pdf,
YARN-530-005.patch, YARN-530-008.patch, YARN-530-009.patch,
YARN-530-010.patch, YARN-530-011.patch, YARN-530-012.patch,
YARN-530-013.patch, YARN-530-014.patch, YARN-530-015.patch,
YARN-530-016.patch, YARN-530-017.patch, YARN-530-018.patch,
YARN-530-019.patch, YARN-530-2.patch, YARN-530-3.patch, YARN-530.4.patch,
YARN-530.patch
# Extend the YARN {{Service}} interface as discussed in YARN-117
# Implement the changes in {{AbstractService}} and {{FilterService}}.
# Migrate all services in yarn-common to the more robust service model, test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-117) Enhance YARN service model
[
https://issues.apache.org/jira/browse/YARN-117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated YARN-117:
Attachment: YARN-117-019.patch
# make sure addService, removeService and service iteration are all thread safe
-eliminates risk of concurrency problems if the service list is changed during
init/start/stop of children. As these methods are protected, a subclass would
have to do this in another thread -or make the methods public
# composite service policy is switched to stop all services that are STARTED or
INITED
Enhance YARN service model
--
Key: YARN-117
URL: https://issues.apache.org/jira/browse/YARN-117
Project: Hadoop YARN
Issue Type: Improvement
Affects Versions: 2.0.4-alpha
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: YARN-117-007.patch, YARN-117-008.patch,
YARN-117-009.patch, YARN-117-010.patch, YARN-117-011.patch,
YARN-117-012.patch, YARN-117-013.patch, YARN-117-014.patch,
YARN-117-015.patch, YARN-117-016.patch, YARN-117-018.patch,
YARN-117-019.patch, YARN-117-2.patch, YARN-117-3.patch, YARN-117.4.patch,
YARN-117.5.patch, YARN-117.6.patch, YARN-117.patch
Having played the YARN service model, there are some issues
that I've identified based on past work and initial use.
This JIRA issue is an overall one to cover the issues, with solutions pushed
out to separate JIRAs.
h2. state model prevents stopped state being entered if you could not
successfully start the service.
In the current lifecycle you cannot stop a service unless it was successfully
started, but
* {{init()}} may acquire resources that need to be explicitly released
* if the {{start()}} operation fails partway through, the {{stop()}}
operation may be needed to release resources.
*Fix:* make {{stop()}} a valid state transition from all states and require
the implementations to be able to stop safely without requiring all fields to
be non null.
Before anyone points out that the {{stop()}} operations assume that all
fields are valid; and if called before a {{start()}} they will NPE;
MAPREDUCE-3431 shows that this problem arises today, MAPREDUCE-3502 is a fix
for this. It is independent of the rest of the issues in this doc but it will
aid making {{stop()}} execute from all states other than stopped.
MAPREDUCE-3502 is too big a patch and needs to be broken down for easier
review and take up; this can be done with issues linked to this one.
h2. AbstractService doesn't prevent duplicate state change requests.
The {{ensureState()}} checks to verify whether or not a state transition is
allowed from the current state are performed in the base {{AbstractService}}
class -yet subclasses tend to call this *after* their own {{init()}},
{{start()}} {{stop()}} operations. This means that these operations can be
performed out of order, and even if the outcome of the call is an exception,
all actions performed by the subclasses will have taken place. MAPREDUCE-3877
demonstrates this.
This is a tricky one to address. In HADOOP-3128 I used a base class instead
of an interface and made the {{init()}}, {{start()}} {{stop()}} methods
{{final}}. These methods would do the checks, and then invoke protected inner
methods, {{innerStart()}}, {{innerStop()}}, etc. It should be possible to
retrofit the same behaviour to everything that extends {{AbstractService}}
-something that must be done before the class is considered stable (because
once the lifecycle methods are declared final, all subclasses that are out of
the source tree will need fixing by the respective developers.
h2. AbstractService state change doesn't defend against race conditions.
There's no concurrency locks on the state transitions. Whatever fix for wrong
state calls is added should correct this to prevent re-entrancy, such as
{{stop()}} being called from two threads.
h2. Static methods to choreograph of lifecycle operations
Helper methods to move things through lifecycles. init-start is common,
stop-if-service!=null another. Some static methods can execute these, and
even call {{stop()}} if {{init()}} raises an exception. These could go into a
class {{ServiceOps}} in the same package. These can be used by those services
that wrap other services, and help manage more robust shutdowns.
h2. state transition failures are something that registered service listeners
may wish to be informed of.
When a state transition fails a {{RuntimeException}} can be thrown -and the
service listeners are not informed as the notification point isn't reached.
They may wish to know this, especially for management and diagnostics.
*Fix:* extend {{ServiceStateChangeListener}} with a callback such as
{{stateChangeFailed(Service service,Service.State targeted-state,
[jira] [Assigned] (YARN-296) Resource Manager throws InvalidStateTransitonException: Invalid event: APP_ACCEPTED at RUNNING for RMAppImpl
[
https://issues.apache.org/jira/browse/YARN-296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayank Bansal reassigned YARN-296:
--
Assignee: Mayank Bansal
Resource Manager throws InvalidStateTransitonException: Invalid event:
APP_ACCEPTED at RUNNING for RMAppImpl
Key: YARN-296
URL: https://issues.apache.org/jira/browse/YARN-296
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Affects Versions: 2.0.2-alpha, 2.0.1-alpha
Reporter: Devaraj K
Assignee: Mayank Bansal
{code:xml}
2012-12-28 11:14:47,671 ERROR
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Can't handle
this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
APP_ACCEPTED at RUNNING
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:528)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:72)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:405)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:389)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-296) Resource Manager throws InvalidStateTransitonException: Invalid event: APP_ACCEPTED at RUNNING for RMAppImpl
[
https://issues.apache.org/jira/browse/YARN-296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675441#comment-13675441
]
Mayank Bansal commented on YARN-296:
[~devaraj.k]
Hi Devaraj,
I tried to reproduce the issue and looked at the code and I didnt see any
condition where this can happen.
Can you please give some more info to reproduce the issue?
Thanks,
Mayank
Resource Manager throws InvalidStateTransitonException: Invalid event:
APP_ACCEPTED at RUNNING for RMAppImpl
Key: YARN-296
URL: https://issues.apache.org/jira/browse/YARN-296
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Affects Versions: 2.0.2-alpha, 2.0.1-alpha
Reporter: Devaraj K
{code:xml}
2012-12-28 11:14:47,671 ERROR
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Can't handle
this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
APP_ACCEPTED at RUNNING
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:528)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl.handle(RMAppImpl.java:72)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:405)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationEventDispatcher.handle(ResourceManager.java:389)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-295) Resource Manager throws InvalidStateTransitonException: Invalid event: CONTAINER_FINISHED at ALLOCATED for RMAppAttemptImpl
[
https://issues.apache.org/jira/browse/YARN-295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675472#comment-13675472
]
Vinod Kumar Vavilapalli commented on YARN-295:
--
I think this can happen in one corner case: If a startContainer fails in
AMLauncher and NM comes back and reports failure before the in-jvm event from
AMLauncher goes to RMAppAttempt, then this transition can result.
Devaraj, do you have more information? RM logs etc?
Resource Manager throws InvalidStateTransitonException: Invalid event:
CONTAINER_FINISHED at ALLOCATED for RMAppAttemptImpl
---
Key: YARN-295
URL: https://issues.apache.org/jira/browse/YARN-295
Project: Hadoop YARN
Issue Type: Sub-task
Components: resourcemanager
Affects Versions: 2.0.2-alpha, 2.0.1-alpha
Reporter: Devaraj K
Assignee: Mayank Bansal
{code:xml}
2012-12-28 14:03:56,956 ERROR
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
Can't handle this event at current state
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
CONTAINER_FINISHED at ALLOCATED
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:490)
at
org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:80)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:433)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:414)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-530) Define Service model strictly, implement AbstractService for robust subclassing, migrate yarn-common services
[
https://issues.apache.org/jira/browse/YARN-530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675489#comment-13675489
]
Hadoop QA commented on YARN-530:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586216/YARN-530-019.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 39 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 6 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-unmanaged-am-launcher
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy:
org.apache.hadoop.conf.TestConfiguration
org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryParsing
org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEvents
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1118//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/1118//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/1118//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1118//console
This message is automatically generated.
Define Service model strictly, implement AbstractService for robust
subclassing, migrate yarn-common services
-
Key: YARN-530
URL: https://issues.apache.org/jira/browse/YARN-530
Project: Hadoop YARN
Issue Type: Sub-task
Affects Versions: 2.0.4-alpha
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: YARN-117-019.patch, YARN-117changes.pdf,
YARN-530-005.patch, YARN-530-008.patch, YARN-530-009.patch,
YARN-530-010.patch, YARN-530-011.patch, YARN-530-012.patch,
YARN-530-013.patch, YARN-530-014.patch, YARN-530-015.patch,
YARN-530-016.patch, YARN-530-017.patch, YARN-530-018.patch,
YARN-530-019.patch, YARN-530-2.patch, YARN-530-3.patch, YARN-530.4.patch,
YARN-530.patch
# Extend the YARN {{Service}} interface as discussed in YARN-117
# Implement the changes in {{AbstractService}} and {{FilterService}}.
# Migrate all services in yarn-common to the more robust service model, test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-750) Allow for black-listing resources in CS
[ https://issues.apache.org/jira/browse/YARN-750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated YARN-750: --- Attachment: YARN-750.patch Thanks for the review Vinod. Here is a patch with your comments incorporated. Allow for black-listing resources in CS --- Key: YARN-750 URL: https://issues.apache.org/jira/browse/YARN-750 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Arun C Murthy Attachments: YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch YARN-392 and YARN-398 enhance scheduler api to allow for white-lists of resources. This jira is a companion to allow for black-listing (in CS). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-686) Flatten NodeReport
[ https://issues.apache.org/jira/browse/YARN-686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675509#comment-13675509 ] Arun C Murthy commented on YARN-686: [~sandyr] Are you working on this? Flatten NodeReport -- Key: YARN-686 URL: https://issues.apache.org/jira/browse/YARN-686 Project: Hadoop YARN Issue Type: Sub-task Components: api Affects Versions: 2.0.4-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza The NodeReport returned by getClusterNodes or given to AMs in heartbeat responses includes both a NodeState (enum) and a NodeHealthStatus (object). As UNHEALTHY is already NodeState, a separate NodeHealthStatus doesn't seem necessary. I propose eliminating NodeHealthStatus#getIsNodeHealthy and moving its two other methods, getHealthReport and getLastHealthReportTime, into NodeReport. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (YARN-763) AMRMClientAsync should stop heartbeating after receiving shutdown from RM
Bikas Saha created YARN-763: --- Summary: AMRMClientAsync should stop heartbeating after receiving shutdown from RM Key: YARN-763 URL: https://issues.apache.org/jira/browse/YARN-763 Project: Hadoop YARN Issue Type: Bug Reporter: Bikas Saha -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-759) Create Command enum in AllocateResponse
[ https://issues.apache.org/jira/browse/YARN-759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated YARN-759: Assignee: Bikas Saha Create Command enum in AllocateResponse --- Key: YARN-759 URL: https://issues.apache.org/jira/browse/YARN-759 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Bikas Saha Attachments: YARN-759.1.patch Use command enums for shutdown/resync instead of booleans. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-759) Create Command enum in AllocateResponse
[ https://issues.apache.org/jira/browse/YARN-759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated YARN-759: Attachment: YARN-759.1.patch Also renames the callback in AMRMClientAsync appropriately. Create Command enum in AllocateResponse --- Key: YARN-759 URL: https://issues.apache.org/jira/browse/YARN-759 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Attachments: YARN-759.1.patch Use command enums for shutdown/resync instead of booleans. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-750) Allow for black-listing resources in CS
[
https://issues.apache.org/jira/browse/YARN-750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675523#comment-13675523
]
Hadoop QA commented on YARN-750:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586237/YARN-750.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 13 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests:
org.apache.hadoop.yarn.client.TestNMClientAsync
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1119//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1119//console
This message is automatically generated.
Allow for black-listing resources in CS
---
Key: YARN-750
URL: https://issues.apache.org/jira/browse/YARN-750
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Attachments: YARN-750.patch, YARN-750.patch, YARN-750.patch,
YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch, YARN-750.patch
YARN-392 and YARN-398 enhance scheduler api to allow for white-lists of
resources.
This jira is a companion to allow for black-listing (in CS).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-759) Create Command enum in AllocateResponse
[
https://issues.apache.org/jira/browse/YARN-759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675531#comment-13675531
]
Hadoop QA commented on YARN-759:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12586242/YARN-759.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/1120//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1120//console
This message is automatically generated.
Create Command enum in AllocateResponse
---
Key: YARN-759
URL: https://issues.apache.org/jira/browse/YARN-759
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: YARN-759.1.patch
Use command enums for shutdown/resync instead of booleans.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-276) Capacity Scheduler can hang when submit many jobs concurrently
[ https://issues.apache.org/jira/browse/YARN-276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675574#comment-13675574 ] Thomas Graves commented on YARN-276: I need to spend more time looking through the new logic, here are a few more comments for now. remove the comment from overAMUsedPercent about max active application since its no longer present. FicaSchedulerApp getAMResource, change amRequedResource - amRequiredResource the max active applications per user used to use the absolute queue capacity instead of the absolute max queue capacity. It was changed to use the absolute capacity because it uses the userlimitfactor in the calculation, which should be applied to the capacity and not max capacity (see MAPREDUCE-3897 for more details). We should change the overAMUsedPercentPerUser similarly to use absolute capacity, not absolute max capacity. This can be filed as a separate jira since it was pre-existing but a bad app that requests 0 for the memory could cause divide by zero exception. Capacity Scheduler can hang when submit many jobs concurrently -- Key: YARN-276 URL: https://issues.apache.org/jira/browse/YARN-276 Project: Hadoop YARN Issue Type: Bug Components: capacityscheduler Affects Versions: 3.0.0, 2.0.1-alpha Reporter: nemon lou Assignee: nemon lou Labels: incompatible Attachments: YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch, YARN-276.patch Original Estimate: 24h Remaining Estimate: 24h In hadoop2.0.1,When i submit many jobs concurrently at the same time,Capacity scheduler can hang with most resources taken up by AM and don't have enough resources for tasks.And then all applications hang there. The cause is that yarn.scheduler.capacity.maximum-am-resource-percent not check directly.Instead ,this property only used for maxActiveApplications. And maxActiveApplications is computed by minimumAllocation (not by Am actually used). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
