[
https://issues.apache.org/jira/browse/YARN-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019730#comment-14019730
]
Wangda Tan commented on YARN-1408:
----------------------------------
Hi [~sunilg],
bq. I feel as per your design, we are keeping a much information than needed.
Rather we can just keep only requried info for a ResourceRequest creation.
I meant we need keep a complete Resource Request, which includes Resource
Request itself, rack/ANY level Resource Request, in case of the entire
ResourceRequest removed when count reaches zero. I think we don't have
difference here. :), right?
bq. I feel we can store enough information to create a new ResourceRequest at
RMContainer itself. In FiCaSchedulerApp#allocate, a replica or partial
information of ResourceRequest can be stored in RMContainer itself.
+1 for this, since the resource requests is only a part of RMContainer, we
don't need store it in app level
bq. CapacityScheduler is extending PreemptableResourceScheduler interface and
it has killContainer implementation. So preempt event handling can be done at
killContainer.
IMHO, it's better to put recover ResourceRequest logic in
RMContainerImpl.FinishedTransition(), we can check if original state is
Allocated and event is KILL. The benefit of this choice is we don't need
separately modify FairScheduler and CapacityScheduler. Make sense?
bq. api AppSchedulingInfo#updateResourceRequests can be used to update new
ResourceRequest.
Yes, but we cannot reuse this API without modifying its definition. The
AppSchedulingInfo#updateResourceRequests will replace original ResourceRequests
to new ResourceRequests, and our requirement is only increase the original
ResourceRequest. We can either add a flag to this API indicate it's a
recovering behavior or create a new API for it. And please note that, we need
update QueueMetrics as well.
Thanks,
Wangda
> Preemption caused Invalid State Event: ACQUIRED at KILLED and caused a task
> timeout for 30mins
> ----------------------------------------------------------------------------------------------
>
> Key: YARN-1408
> URL: https://issues.apache.org/jira/browse/YARN-1408
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Affects Versions: 2.2.0
> Reporter: Sunil G
> Assignee: Sunil G
> Attachments: Yarn-1408.1.patch, Yarn-1408.2.patch, Yarn-1408.3.patch,
> Yarn-1408.4.patch, Yarn-1408.patch
>
>
> Capacity preemption is enabled as follows.
> * yarn.resourcemanager.scheduler.monitor.enable= true ,
> *
> yarn.resourcemanager.scheduler.monitor.policies=org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy
> Queue = a,b
> Capacity of Queue A = 80%
> Capacity of Queue B = 20%
> Step 1: Assign a big jobA on queue a which uses full cluster capacity
> Step 2: Submitted a jobB to queue b which would use less than 20% of cluster
> capacity
> JobA task which uses queue b capcity is been preempted and killed.
> This caused below problem:
> 1. New Container has got allocated for jobA in Queue A as per node update
> from an NM.
> 2. This container has been preempted immediately as per preemption.
> Here ACQUIRED at KILLED Invalid State exception came when the next AM
> heartbeat reached RM.
> ERROR
> org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl:
> Can't handle this event at current state
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> ACQUIRED at KILLED
> This also caused the Task to go for a timeout for 30minutes as this Container
> was already killed by preemption.
> attempt_1380289782418_0003_m_000000_0 Timed out after 1800 secs
--
This message was sent by Atlassian JIRA
(v6.2#6252)
