[
https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15600121#comment-15600121
]
Naganarasimha G R commented on MAPREDUCE-6773:
----------------------------------------------
hi @devaraj k
Sorry from my side as i took a while to reply back on this jira. I think we
need to cancel the patch and rebase it based on the work done in
MAPREDUCE-6772.
And coming to some major discussion points :
# Generally i like the approach of modifying the existing hierarchy i.e
{{RMCommunicator <- RMContainerRequestor <- RMContainerAllocator}} to
{{RMCommunicator <- RMContainerAllocator}} and having a new hierarchy
{{ContainerReqestor <- RMContainerRequestor <- RMContainerReUseRequestor}}
which specifically deals with the handling of structures required for the
requesting, but any other better name as it not exactly requesting the
container but just storing the stats required for it ?
# IIUC, Though superficially {{container.reuse.max-maptaskcontainers}} looks
usefull, again was wondering how the job submitter/admin is going to configure
this max # of concurrent MAP(/reduce) container which can be reused ? atleast
the previous one(container.reuse.max-maptasks) based on the historic values,
admin can determine how much time one map/reduce task is going to take time and
accordingly ask the job submitter to configure but this concurrent max reuse
(MAP/REDUCE) tasks is just a blind configuration and would be difficult to
determine the right value. Thoughts ? (hope my understanding of this config is
correct !).
# may be we can split this jira into 2 parts like creating the interface for
ContainerReqestor and either in new jira or in MAPREDUCE-6786, add the code
related to reuse so that its complete. as these jiras doesnt look to be
testable pieces of jiras.
nits :
# has white space errors when applying the patch
# if we plan to use MR_AM_CONTAINER_REUSE_MAX_MAPTASKCONTAINERS (also reduce
one), then we need to rename the prefix from {{yarn.app.mapreduce.am}} to
{{yarn.app.mapreduce}} & *mapred-default.xml* needs to be updated
# {{JOB_CONTAINER_COMPLETED}}, is it related to this patch ? should it be in
MAPREDUCE-6784? seems like not used anywhere in the current patch.
# {{ContainerRequestorEvent}} has almost similar naming to that of
{{ContainerRequestEvent}}, further was wondering whether we can rename it to
ContainerReUseRequestEvent, as its only used by RMContainerReUseRequestor.
> Implement RM Container Reuse Requestor to handle the reuse containers for
> resource requests
> -------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6773
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
> Reporter: Devaraj K
> Assignee: Devaraj K
> Attachments: MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch,
> MAPREDUCE-6773-v2.patch
>
>
> Add RM Container Reuse Requestor which handles the reuse containers against
> the Job reource requests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
