[
https://issues.apache.org/jira/browse/SPARK-9396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14647258#comment-14647258
]
prakhar jauhari commented on SPARK-9396:
----------------------------------------
Can you please assign this issue to me, I am adding a PR.
> Spark yarn allocator does not call "removeContainerRequest" for allocated
> Container requests, resulting in bloated ask[] toYarn RM.
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-9396
> URL: https://issues.apache.org/jira/browse/SPARK-9396
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.2.1
> Environment: Spark-1.2.1 on hadoop-yarn-2.4.0 cluster. All servers in
> cluster running Linux version 2.6.32.
> Reporter: prakhar jauhari
>
> Note : Attached logs contain logs that i added (spark yarn allocator side and
> Yarn client side) for debugging purpose.
> !!!!! My spark job is configured for 2 executors, on killing 1 executor the
> ask is of 3 !!!!!!!
> On killing a executor - resource request logs :
> *************Killed container: ask for 3 containers, instead for 1***********
> 15/07/15 10:49:01 INFO yarn.YarnAllocationHandler: Will allocate 1 executor
> containers, each with 2432 MB memory including 384 MB overhead
> 15/07/15 10:49:01 INFO yarn.YarnAllocationHandler: numExecutors: 1
> 15/07/15 10:49:01 INFO yarn.YarnAllocationHandler: host preferences is empty
> 15/07/15 10:49:01 INFO yarn.YarnAllocationHandler: Container request (host:
> Any, priority: 1, capability: <memory:2432, vCores:4>
> 15/07/15 10:49:01 INFO impl.AMRMClientImpl: prakhar : AMRMClientImpl :
> allocate: this.ask = [{Priority: 1, Capability: <memory:2432, vCores:4>, #
> Containers: 3, Location: *, Relax Locality: true}]
> 15/07/15 10:49:01 INFO impl.AMRMClientImpl: prakhar : AMRMClientImpl :
> allocate: allocateRequest = ask { priority{ priority: 1 } resource_name: "*"
> capability { memory: 2432 virtual_cores: 4 } num_containers: 3
> relax_locality: true } blacklist_request { } response_id: 354 progress: 0.1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
