[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6785: - Resolution: Fixed Fix Version/s: MR-6749 Status: Resolved (was: Patch Available) Thanks [~Naganarasimha] for the patch. Committed to MR-6749 branch. > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Naganarasimha G R > Fix For: MR-6749 > > Attachments: MAPREDUCE-6785-MR-6749.003.patch, > MAPREDUCE-6785-MR-6749.004.patch, MAPREDUCE-6785-MR-6749.005.patch, > MAPREDUCE-6785-MR-6749.006.patch, MAPREDUCE-6785-MR-6749.007.patch, > MAPREDUCE-6785-v0.patch, MAPREDUCE-6785-v1.patch, MAPREDUCE-6785-v2.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6785: - Hadoop Flags: Reviewed +1, looks good to me, committing it. > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Naganarasimha G R > Attachments: MAPREDUCE-6785-MR-6749.003.patch, > MAPREDUCE-6785-MR-6749.004.patch, MAPREDUCE-6785-MR-6749.005.patch, > MAPREDUCE-6785-MR-6749.006.patch, MAPREDUCE-6785-MR-6749.007.patch, > MAPREDUCE-6785-v0.patch, MAPREDUCE-6785-v1.patch, MAPREDUCE-6785-v2.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6785: - Status: Patch Available (was: Open) > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Naganarasimha G R > Attachments: MAPREDUCE-6785-MR-6749.003.patch, > MAPREDUCE-6785-MR-6749.004.patch, MAPREDUCE-6785-MR-6749.005.patch, > MAPREDUCE-6785-MR-6749.006.patch, MAPREDUCE-6785-MR-6749.007.patch, > MAPREDUCE-6785-v0.patch, MAPREDUCE-6785-v1.patch, MAPREDUCE-6785-v2.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6785: - Assignee: Naganarasimha G R (was: Devaraj K) Status: Open (was: Patch Available) Thanks [~Naganarasimha] for the patch. Patch looks fine to me except the checkstyle warnings. I think we can handle these, Can you update the patch with checkstyle fixes which are showing in the report? > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Naganarasimha G R > Attachments: MAPREDUCE-6785-MR-6749.003.patch, > MAPREDUCE-6785-MR-6749.004.patch, MAPREDUCE-6785-MR-6749.005.patch, > MAPREDUCE-6785-MR-6749.006.patch, MAPREDUCE-6785-v0.patch, > MAPREDUCE-6785-v1.patch, MAPREDUCE-6785-v2.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6785: - Status: Open (was: Patch Available) > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6785-MR-6749.003.patch, > MAPREDUCE-6785-MR-6749.004.patch, MAPREDUCE-6785-v0.patch, > MAPREDUCE-6785-v1.patch, MAPREDUCE-6785-v2.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[
https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895105#comment-15895105
]
Devaraj K commented on MAPREDUCE-6785:
--
Thanks [~Naganarasimha] for the patch and apologize for the delay in review.
Patch looks ok to me except the Hadoop QA reported errors and some minor
comments about the coding style.
1. TaskAttemptContainerAssignedEvent - Can we move move the setters and getters
after the constructor?
2. RMContainerReuseRequestor - Can we move the HostInfo inner class to the end
of the class?
3. TaskAttemptImpl.java - Can we keep the same format as previous for these
lines,
{code:xml}
- taskAttempt.taskAttemptListener.registerPendingTask(
- taskAttempt.remoteTask, taskAttempt.jvmID);
+ taskAttempt.taskAttemptListener
+ .registerPendingTask(taskAttempt.remoteTask, taskAttempt.jvmID);
{code}
> ContainerLauncherImpl support for reusing the containers
>
>
> Key: MAPREDUCE-6785
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-6785-MR-6749.003.patch,
> MAPREDUCE-6785-MR-6749.004.patch, MAPREDUCE-6785-v0.patch,
> MAPREDUCE-6785-v1.patch, MAPREDUCE-6785-v2.patch
>
>
> Add support to Container Launcher for reuse of the containers.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6833) Display only corresponding Task logs for each task attempt in AM and JHS Web UI
Devaraj K created MAPREDUCE-6833: Summary: Display only corresponding Task logs for each task attempt in AM and JHS Web UI Key: MAPREDUCE-6833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6833 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Devaraj K Assignee: Devaraj K When the container gets reused for multiple tasks, Logs get generated in the same container log file. At present Task Attempt log is linked to Container log, In UI for each attempt it shows the whole container log file. This task is to handle the display of the corresponding Task logs for each task attempt in AM and JHS Web UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15797194#comment-15797194 ] Devaraj K commented on MAPREDUCE-6785: -- Thanks [~Naganarasimha] for the review, I have addressed the comments in the patch. bq. Also one query, In cases where Speculative attempt is running, killing of a attempts should kill the container ? Yes, We will be reusing the container only when the task in the container is successful, container will not be considered for reuse if the task fails or gets killed. > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6785-MR-6749.003.patch, > MAPREDUCE-6785-v0.patch, MAPREDUCE-6785-v1.patch, MAPREDUCE-6785-v2.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6785: - Target Version/s: MR-6749 Status: Patch Available (was: Open) > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6785-MR-6749.003.patch, > MAPREDUCE-6785-v0.patch, MAPREDUCE-6785-v1.patch, MAPREDUCE-6785-v2.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6785: - Attachment: MAPREDUCE-6785-MR-6749.003.patch > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6785-MR-6749.003.patch, > MAPREDUCE-6785-v0.patch, MAPREDUCE-6785-v1.patch, MAPREDUCE-6785-v2.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6785: - Status: Open (was: Patch Available) > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6785-v0.patch, MAPREDUCE-6785-v1.patch, > MAPREDUCE-6785-v2.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6785: - Status: Patch Available (was: Open) Updated the patch with findbug fix. > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6785-v0.patch, MAPREDUCE-6785-v1.patch, > MAPREDUCE-6785-v2.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6785: - Attachment: MAPREDUCE-6785-v2.patch > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6785-v0.patch, MAPREDUCE-6785-v1.patch, > MAPREDUCE-6785-v2.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6785: - Status: Open (was: Patch Available) > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6785-v0.patch, MAPREDUCE-6785-v1.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6785: - Status: Patch Available (was: Open) Updated the patch for this task. > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6785-v0.patch, MAPREDUCE-6785-v1.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6785: - Attachment: MAPREDUCE-6785-v1.patch > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6785-v0.patch, MAPREDUCE-6785-v1.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6786) TaskAttemptImpl state changes for containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K reassigned MAPREDUCE-6786: Assignee: Naganarasimha G R (was: Devaraj K) > TaskAttemptImpl state changes for containers reuse > -- > > Key: MAPREDUCE-6786 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6786 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Naganarasimha G R > Attachments: MAPREDUCE-6786-v0.patch > > > Update TaskAttemptImpl to support the reuse of containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6784) JobImpl state changes for containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15697324#comment-15697324 ] Devaraj K commented on MAPREDUCE-6784: -- Sure [~Naganarasimha], Please take it. > JobImpl state changes for containers reuse > -- > > Key: MAPREDUCE-6784 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6784 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6784-v0.patch > > > Add JobImpl state changes for supporting reusing of containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[
https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15693884#comment-15693884
]
Devaraj K commented on MAPREDUCE-6773:
--
bq. if its uber job then no need to do this registration RMContainerAllocator
@ ln 252.
If it is a uber job then the flow will not come into RMContainerAllocator,
instead it uses LocalContainerAllocator.
{code:title=MRAppMaster.java|borderStyle=solid}
if (job.isUber()) {
MRApps.setupDistributedCacheLocal(getConfig());
this.containerAllocator = new LocalContainerAllocator(
this.clientService, this.context, nmHost, nmPort, nmHttpPort
, containerID);
} else {
this.containerAllocator = new RMContainerAllocator(
this.clientService, this.context, preemptionPolicy, dispatcher);
}
{code}
> Implement RM Container Reuse Requestor to handle the reuse containers for
> resource requests
> ---
>
> Key: MAPREDUCE-6773
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: applicationmaster, mrv2
>Affects Versions: MR-6749
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-6773-MR-6749.003.patch,
> MAPREDUCE-6773-MR-6749.004.patch, MAPREDUCE-6773-MR-6749.005.patch,
> MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch, MAPREDUCE-6773-v2.patch
>
>
> Add RM Container Reuse Requestor which handles the reuse containers against
> the Job reource requests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Status: Patch Available (was: Open) Thanks [~Naganarasimha] for the patch review. I have updated the patch with the 1 & 2 comments fix, 3 & 4 comments will be handled as part of the MAPREDUCE-6784 jira for sending and handling the event when the container is no more reusing. Please check the latest patch and add your comments if any. > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Affects Versions: MR-6749 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-MR-6749.003.patch, > MAPREDUCE-6773-MR-6749.004.patch, MAPREDUCE-6773-MR-6749.005.patch, > MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch, MAPREDUCE-6773-v2.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Attachment: MAPREDUCE-6773-MR-6749.005.patch > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Affects Versions: MR-6749 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-MR-6749.003.patch, > MAPREDUCE-6773-MR-6749.004.patch, MAPREDUCE-6773-MR-6749.005.patch, > MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch, MAPREDUCE-6773-v2.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Status: Open (was: Patch Available) > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Affects Versions: MR-6749 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-MR-6749.003.patch, > MAPREDUCE-6773-MR-6749.004.patch, MAPREDUCE-6773-v0.patch, > MAPREDUCE-6773-v1.patch, MAPREDUCE-6773-v2.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Status: Patch Available (was: Open) I have updated the patch, [~Naganarasimha], can you have a look into this? Thanks > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Affects Versions: MR-6749 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-MR-6749.003.patch, > MAPREDUCE-6773-MR-6749.004.patch, MAPREDUCE-6773-v0.patch, > MAPREDUCE-6773-v1.patch, MAPREDUCE-6773-v2.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Attachment: MAPREDUCE-6773-MR-6749.004.patch > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Affects Versions: MR-6749 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-MR-6749.003.patch, > MAPREDUCE-6773-MR-6749.004.patch, MAPREDUCE-6773-v0.patch, > MAPREDUCE-6773-v1.patch, MAPREDUCE-6773-v2.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6809) Create ContainerRequestor interface and refactor RMContainerRequestor to use it
[ https://issues.apache.org/jira/browse/MAPREDUCE-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6809: - Status: Patch Available (was: Open) Updated the patch with the review comments fixes. > Create ContainerRequestor interface and refactor RMContainerRequestor to use > it > --- > > Key: MAPREDUCE-6809 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6809 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6809-MR-6749.001.patch, > MAPREDUCE-6809-MR-6749.002.patch > > > As per the discussion in MAPREDUCE-6773, create a ContainerRequestor > interface and refactor RMContainerRequestor to use this interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6809) Create ContainerRequestor interface and refactor RMContainerRequestor to use it
[ https://issues.apache.org/jira/browse/MAPREDUCE-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6809: - Attachment: MAPREDUCE-6809-MR-6749.002.patch > Create ContainerRequestor interface and refactor RMContainerRequestor to use > it > --- > > Key: MAPREDUCE-6809 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6809 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6809-MR-6749.001.patch, > MAPREDUCE-6809-MR-6749.002.patch > > > As per the discussion in MAPREDUCE-6773, create a ContainerRequestor > interface and refactor RMContainerRequestor to use this interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6809) Create ContainerRequestor interface and refactor RMContainerRequestor to use it
[ https://issues.apache.org/jira/browse/MAPREDUCE-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6809: - Target Version/s: MR-6749 Status: Open (was: Patch Available) Thanks [~Naganarasimha] for the comments, cancelling patch to address them. > Create ContainerRequestor interface and refactor RMContainerRequestor to use > it > --- > > Key: MAPREDUCE-6809 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6809 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6809-MR-6749.001.patch > > > As per the discussion in MAPREDUCE-6773, create a ContainerRequestor > interface and refactor RMContainerRequestor to use this interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6809) Create ContainerRequestor interface and refactor RMContainerRequestor to use it
[ https://issues.apache.org/jira/browse/MAPREDUCE-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6809: - Status: Patch Available (was: Open) > Create ContainerRequestor interface and refactor RMContainerRequestor to use > it > --- > > Key: MAPREDUCE-6809 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6809 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6809-MR-6749.001.patch > > > As per the discussion in MAPREDUCE-6773, create a ContainerRequestor > interface and refactor RMContainerRequestor to use this interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6809) Create ContainerRequestor interface and refactor RMContainerRequestor to use it
[ https://issues.apache.org/jira/browse/MAPREDUCE-6809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6809: - Attachment: MAPREDUCE-6809-MR-6749.001.patch Attaching patch for this. > Create ContainerRequestor interface and refactor RMContainerRequestor to use > it > --- > > Key: MAPREDUCE-6809 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6809 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6809-MR-6749.001.patch > > > As per the discussion in MAPREDUCE-6773, create a ContainerRequestor > interface and refactor RMContainerRequestor to use this interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Status: Open (was: Patch Available) Thanks [~Naganarasimha] for the comments. bq. lets keep this jira to only extract the new interface out (ContainerRequestor) and pull out the changes related to the reuse in another jira (or merge with MAPREDUCE-6786 as we will be having some logical conclusion and related things will be covered). I have created MAPREDUCE-6809 for the interface refactoring changes, and will have the RM Container Reuse Requestor implementation in this jira as title indicates. bq. in our offline discussion you had mentioned that RMContainerRequestor also uses ContainerRequestorEvent bq. Checked it i think we can remove it just tried it locally seems to work. Here we need to have a container available event handling mechanism when the container becomes available for reuse, I am thinking this to rename as ContainerAvailableEvent. > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Affects Versions: MR-6749 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-MR-6749.003.patch, > MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch, MAPREDUCE-6773-v2.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6809) Create ContainerRequestor interface and refactor RMContainerRequestor to use it
Devaraj K created MAPREDUCE-6809: Summary: Create ContainerRequestor interface and refactor RMContainerRequestor to use it Key: MAPREDUCE-6809 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6809 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Devaraj K Assignee: Devaraj K As per the discussion in MAPREDUCE-6773, create a ContainerRequestor interface and refactor RMContainerRequestor to use this interface. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Affects Version/s: MR-6749 Status: Patch Available (was: Open) I have updated the patch with changes, and will update with these once we have conclusion about the above and below points. bq. if we plan to use MR_AM_CONTAINER_REUSE_MAX_MAPTASKCONTAINERS (also reduce one), then we need to rename the prefix from yarn.app.mapreduce.am to yarn.app.mapreduce & mapred-default.xml needs to be updated We have already renamed these configs as part of MAPREDUCE-6772 and we are just referring the constants from MRJobConfig.java. Is there anything else you are referring here? bq. JOB_CONTAINER_COMPLETED, is it related to this patch ? should it be in MAPREDUCE-6784? seems like not used anywhere in the current patch. I removed this as part of this patch, will add it with MAPREDUCE-6784 or other jira. bq. ContainerRequestorEvent has almost similar naming to that of ContainerRequestEvent, further was wondering whether we can rename it to ContainerReUseRequestEvent, as its only used by RMContainerReUseRequestor. Currently we are referring ContainerRequestorEvent from RMContainerRequestor and RMContainerReuseRequestor. I think the name RMContainerReUseRequestor would not match properly for RMContainerRequestor. Any suggestions? > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Affects Versions: MR-6749 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-MR-6749.003.patch, > MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch, MAPREDUCE-6773-v2.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Attachment: MAPREDUCE-6773-MR-6749.003.patch > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-MR-6749.003.patch, > MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch, MAPREDUCE-6773-v2.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15613847#comment-15613847 ] Devaraj K commented on MAPREDUCE-6773: -- Thanks [~Naganarasimha] for the comments and sorry for the delay. I am working on this to update the patch, before providing patch I would like to confirm these. bq. RMContainerReUseRequestor which specifically deals with the handling of structures required for the requesting, but any other better name as it not exactly requesting the container but just storing the stats required for it ? Do you have any suggestion for this name or anything in your mind which you think better? bq. IIUC, Though superficially container.reuse.max-maptaskcontainers looks usefull, again was wondering how the job submitter/admin is going to configure this max # of concurrent MAP(/reduce) container which can be reused ? atleast the previous one(container.reuse.max-maptasks) based on the historic values, admin can determine how much time one map/reduce task is going to take time and accordingly ask the job submitter to configure but this concurrent max reuse (MAP/REDUCE) tasks is just a blind configuration and would be difficult to determine the right value. Thoughts ? (hope my understanding of this config is correct !). Here we are providing this configuration to leave the headroom for other applications/jobs instead of using all the resources for reuse containers of one Job(if there is a need of resources for the Job). The User/admin would be having idea of what workloads going to run, they can decide to configure value for this based on the coming jobs while running the current Job. bq. may be we can split this jira into 2 parts like creating the interface for ContainerReqestor and either in new jira or in MAPREDUCE-6786, add the code related to reuse so that its complete. as these jiras doesnt look to be testable pieces of jiras. You mean, we need to move the ContainerReqestor interface definition to another jira or the instantiation of the ContainerReqestor? bq. has white space errors when applying the patch I am not getting this white space errors in my env, and also above QA report doesn't show this, which way you are trying to apply the patch? > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch, > MAPREDUCE-6773-v2.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6772) Add MR Job Configurations for Containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15603691#comment-15603691 ] Devaraj K commented on MAPREDUCE-6772: -- Thanks [~Naganarasimha] for the review. > Add MR Job Configurations for Containers reuse > -- > > Key: MAPREDUCE-6772 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6772 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Fix For: MR-6749 > > Attachments: MAPREDUCE-6772-MR-6749.004.patch, > MAPREDUCE-6772-v0.patch, MAPREDUCE-6772-v1.patch, > MR-6749-MAPREDUCE-6772.003.patch > > > This task adds configurations required for MR AM Container reuse feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6772) Add MR Job Configurations for Containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6772: - Status: Patch Available (was: Open) Thanks [~Naganarasimha] for the comments. I have uploaded the patch with the updated mapred-default.xml configs and patch naming convention. > Add MR Job Configurations for Containers reuse > -- > > Key: MAPREDUCE-6772 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6772 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6772-MR-6749.004.patch, > MAPREDUCE-6772-v0.patch, MAPREDUCE-6772-v1.patch, > MR-6749-MAPREDUCE-6772.003.patch > > > This task adds configurations required for MR AM Container reuse feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6772) Add MR Job Configurations for Containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6772: - Attachment: MAPREDUCE-6772-MR-6749.004.patch > Add MR Job Configurations for Containers reuse > -- > > Key: MAPREDUCE-6772 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6772 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6772-MR-6749.004.patch, > MAPREDUCE-6772-v0.patch, MAPREDUCE-6772-v1.patch, > MR-6749-MAPREDUCE-6772.003.patch > > > This task adds configurations required for MR AM Container reuse feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6772) Add MR Job Configurations for Containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6772: - Status: Open (was: Patch Available) > Add MR Job Configurations for Containers reuse > -- > > Key: MAPREDUCE-6772 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6772 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6772-v0.patch, MAPREDUCE-6772-v1.patch, > MR-6749-MAPREDUCE-6772.003.patch > > > This task adds configurations required for MR AM Container reuse feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6772) Add MR Job Configurations for Containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6772: - Status: Patch Available (was: Open) > Add MR Job Configurations for Containers reuse > -- > > Key: MAPREDUCE-6772 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6772 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6772-v0.patch, MAPREDUCE-6772-v1.patch, > MR-6749-MAPREDUCE-6772.003.patch > > > This task adds configurations required for MR AM Container reuse feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6772) Add MR Job Configurations for Containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6772: - Attachment: MR-6749-MAPREDUCE-6772.003.patch Thanks [~Naganarasimha] for the review. I have updated the patch with the comment, and also I updated other config names to make it consistent with it. > Add MR Job Configurations for Containers reuse > -- > > Key: MAPREDUCE-6772 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6772 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6772-v0.patch, MAPREDUCE-6772-v1.patch, > MR-6749-MAPREDUCE-6772.003.patch > > > This task adds configurations required for MR AM Container reuse feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6772) Add MR Job Configurations for Containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6772: - Status: Open (was: Patch Available) > Add MR Job Configurations for Containers reuse > -- > > Key: MAPREDUCE-6772 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6772 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6772-v0.patch, MAPREDUCE-6772-v1.patch > > > This task adds configurations required for MR AM Container reuse feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583132#comment-15583132 ] Devaraj K commented on MAPREDUCE-6749: -- I have created a branch MR-6749 for this. > MR AM should reuse containers for Map/Reduce Tasks > -- > > Key: MAPREDUCE-6749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf > > > It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers > for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574159#comment-15574159 ] Devaraj K commented on MAPREDUCE-6749: -- Thanks [~Naganarasimha] for the responses and discussion. bq. hope we can create a new branch and get the things in there so that its easier for others to have a look before it gets into the trunk or main stream branches Sure, I will create a branch for this. bq. If there was a way for admin to enforce it it would be usefull. If its just client level configuration it just adds into already long list of configurations and users will not be clear what to configure for it. And besides would it be better to have just how many tasks can reuse a given container and try to avoid for Map and Reduce seperately ? I am thinking that we can provide comprehensive way to control this feature, I can convince that it will be an another configuration for the user to configure it. We can discuss this in MAPREDUCE-6772/MAPREDUCE-6773. bq. Btw it could be also good to introduce a metric for number of Map Tasks or Reduce tasks which has reused the containers Good thought, we can have metrics for this. bq. This was the problem which we generally faced and difficult for the customers to understand that entire log is not for the task attempt, so was wondering to have any better approach to this. We can think of displaying the part of the container log(logs generated for a task attempt) in JHS Web UI instead of the whole container logs by discarding the other task attempt logs. Do you think this sounds ok or any better way? > MR AM should reuse containers for Map/Reduce Tasks > -- > > Key: MAPREDUCE-6749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf > > > It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers > for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15552367#comment-15552367 ] Devaraj K commented on MAPREDUCE-6773: -- The failed test and javac warnings are not related to this patch. > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch, > MAPREDUCE-6773-v2.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Attachment: MAPREDUCE-6773-v2.patch Updated with checkstyle and javac warning fix. > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch, > MAPREDUCE-6773-v2.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550874#comment-15550874
]
Devaraj K commented on MAPREDUCE-6749:
--
Thanks [~Naganarasimha] for looking into this and looking forward to work on
this with you.
{quote}
Is it required to have number of total Maps(/reducers) to be run using
container reuse? i felt we can just keep whether to reuse the containers are
not and one assumption that map containers will be reused only for map and
reduce only for reduce
{quote}
I think the limit configuration for no of map/reduce reuse containers would
allow other applications to start running without waiting for the Job to be
finished when reuse is enabled. If there is a big Job running which could
occupy the entire cluster, and then any high priority application gets
submitted this limit for maps/reduce container would probably give a room for
high priority application to start running without preempting the containers of
the previous Job. By default there is no limit for number of containers to be
reused and if any user/Job wanted to have this constraint they can configure
it.
{quote}
did not understand the need of RMContainerReuseRequestor.java, may be i will go
through MAPREDUCE-6773 but was expecting more like just TaskAttemptIimpl gets
TA_DONE event was expecting it do have a new ReuseContainerTransition where in
if required it will reassign a new task to it else will continue to
CLEANUP_CONTAINER_TRANSITION. may be we can discuss more on this in
MAPREDUCE-6773 ?
{quote}
RMContainerReuseRequestor has the functionality for checking the container
reuse constraints and adding to reuse containers if it is eligible. This class
abstracts the reuse functionality from regular RMContainerRequestor and
RMContainerReuseRequestor will be used only when the RMContainer reuse is
enabled. If you want to have a try this feature, you can apply MAPREDUCE-6773,
MAPREDUCE-6781, MAPREDUCE-6784, MAPREDUCE-6785, MAPREDUCE-6786 and then try
this feature. I would be happy to get your feedback or suggestions if any.
{quote}
If we have some conclusion on issue 2, then we can think of whether only
TaskAttemptImpl state is sufficient or we need to modify others too.
{quote}
Sure, We can discuss it further if required.
{quote} Logs generated by the tasks in a container will be appended in the same
container log file we need to ensure we are able to display trhough JHS and
other places where ever required to display
{quote}
TaskAttempt is linked to the container and the corresponding container log
would be displayed for TaskAttempt through the JHS. Here we should note that
the whole container log which is displaying for TaskAttempt is not applicable
to the TaskAttempt and the log can be identified easily which part applicable
to it.
> MR AM should reuse containers for Map/Reduce Tasks
> --
>
> Key: MAPREDUCE-6749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster, mrv2
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf
>
>
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers
> for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Attachment: MAPREDUCE-6773-v1.patch Rebased the stale patch. > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6786) TaskAttemptImpl state changes for containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6786: - Attachment: MAPREDUCE-6786-v0.patch Attaching the patch for this. > TaskAttemptImpl state changes for containers reuse > -- > > Key: MAPREDUCE-6786 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6786 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6786-v0.patch > > > Update TaskAttemptImpl to support the reuse of containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6785: - Attachment: MAPREDUCE-6785-v0.patch Attaching the first version of the patch for this. > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6785-v0.patch > > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6784) JobImpl state changes for containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6784: - Attachment: MAPREDUCE-6784-v0.patch Attaching the patch for this. > JobImpl state changes for containers reuse > -- > > Key: MAPREDUCE-6784 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6784 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6784-v0.patch > > > Add JobImpl state changes for supporting reusing of containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6786) TaskAttemptImpl state changes for containers reuse
Devaraj K created MAPREDUCE-6786: Summary: TaskAttemptImpl state changes for containers reuse Key: MAPREDUCE-6786 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6786 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Devaraj K Assignee: Devaraj K Update TaskAttemptImpl to support the reuse of containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
Devaraj K created MAPREDUCE-6785: Summary: ContainerLauncherImpl support for reusing the containers Key: MAPREDUCE-6785 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Devaraj K Assignee: Devaraj K -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6785) ContainerLauncherImpl support for reusing the containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6785: - Description: Add support to Container Launcher for reuse of the containers. > ContainerLauncherImpl support for reusing the containers > > > Key: MAPREDUCE-6785 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6785 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > > Add support to Container Launcher for reuse of the containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6784) JobImpl state changes for containers reuse
Devaraj K created MAPREDUCE-6784: Summary: JobImpl state changes for containers reuse Key: MAPREDUCE-6784 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6784 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Devaraj K Assignee: Devaraj K Add JobImpl state changes for supporting reusing of containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6781) YarnChild should wait for another task when reuse is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6781: - Attachment: MAPREDUCE-6781-v0.patch > YarnChild should wait for another task when reuse is enabled > > > Key: MAPREDUCE-6781 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6781 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6781-v0.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6781) YarnChild should wait for another task when reuse is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-6781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K reassigned MAPREDUCE-6781: Assignee: Devaraj K > YarnChild should wait for another task when reuse is enabled > > > Key: MAPREDUCE-6781 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6781 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6781) YarnChild should wait for another task when reuse is enabled
Devaraj K created MAPREDUCE-6781: Summary: YarnChild should wait for another task when reuse is enabled Key: MAPREDUCE-6781 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6781 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Devaraj K -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Attachment: (was: MAPREDUCE-6773-v0.patch) > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Attachment: MAPREDUCE-6773-v0.patch > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-v0.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Status: Patch Available (was: Open) > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-v0.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6772) Add MR Job Configurations for Containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6772: - Status: Patch Available (was: Open) > Add MR Job Configurations for Containers reuse > -- > > Key: MAPREDUCE-6772 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6772 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6772-v0.patch, MAPREDUCE-6772-v1.patch > > > This task adds configurations required for MR AM Container reuse feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6772) Add MR Job Configurations for Containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6772: - Attachment: MAPREDUCE-6772-v1.patch Attaching patch with check-style fixes. > Add MR Job Configurations for Containers reuse > -- > > Key: MAPREDUCE-6772 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6772 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6772-v0.patch, MAPREDUCE-6772-v1.patch > > > This task adds configurations required for MR AM Container reuse feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6772) Add MR Job Configurations for Containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6772: - Status: Open (was: Patch Available) > Add MR Job Configurations for Containers reuse > -- > > Key: MAPREDUCE-6772 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6772 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6772-v0.patch > > > This task adds configurations required for MR AM Container reuse feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6772) Add MR Job Configurations for Containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6772: - Status: Patch Available (was: Open) > Add MR Job Configurations for Containers reuse > -- > > Key: MAPREDUCE-6772 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6772 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6772-v0.patch > > > This task adds configurations required for MR AM Container reuse feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Attachment: MAPREDUCE-6773-v0.patch Attaching the first version of the patch. > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-v0.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K reassigned MAPREDUCE-6773: Assignee: Devaraj K > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
Devaraj K created MAPREDUCE-6773: Summary: Implement RM Container Reuse Requestor to handle the reuse containers for resource requests Key: MAPREDUCE-6773 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Devaraj K Add RM Container Reuse Requestor which handles the reuse containers against the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6772) Add MR Job Configurations for Containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6772: - Attachment: MAPREDUCE-6772-v0.patch Attaching the first version of the patch. > Add MR Job Configurations for Containers reuse > -- > > Key: MAPREDUCE-6772 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6772 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6772-v0.patch > > > This task adds configurations required for MR AM Container reuse feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6772) Add MR Job Configurations for Containers reuse
Devaraj K created MAPREDUCE-6772: Summary: Add MR Job Configurations for Containers reuse Key: MAPREDUCE-6772 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6772 Project: Hadoop Map/Reduce Issue Type: Sub-task Reporter: Devaraj K Assignee: Devaraj K This task adds configurations required for MR AM Container reuse feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6749: - Attachment: MAPREDUCE-6749-Container Reuse-v0.pdf Please find the details document, your comments/suggestions are welcome. > MR AM should reuse containers for Map/Reduce Tasks > -- > > Key: MAPREDUCE-6749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf > > > It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers > for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
Devaraj K created MAPREDUCE-6749: Summary: MR AM should reuse containers for Map/Reduce Tasks Key: MAPREDUCE-6749 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mrv2 Reporter: Devaraj K Assignee: Devaraj K It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-3902) MR AM should reuse containers for map tasks, there-by allowing fine-grained control on num-maps for users without need for CombineFileInputFormat etc.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15412086#comment-15412086 ] Devaraj K commented on MAPREDUCE-3902: -- It has been a pending item for quite some time, I would like to take this feature forward. I will create a cloned JIRA with the details doc for this to work on and also to avoid the collision with the existing branch MR-3902. > MR AM should reuse containers for map tasks, there-by allowing fine-grained > control on num-maps for users without need for CombineFileInputFormat etc. > -- > > Key: MAPREDUCE-3902 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3902 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Arun C Murthy >Assignee: Kannan Rajah > Attachments: AMContainerRefactorNotes.pdf, AM_ContainerRefactor.pdf, > MAPREDUCE-3902.2.patch, MAPREDUCE-3902.patch > > > The MR AM is now in a great position to reuse containers across (map) tasks. > This is something similar to JVM re-use we had in 0.20.x, but in a > significantly better manner: > # Consider data-locality when re-using containers > # Consider the new shuffle - ensure that reduces fetch output of the whole > container at once (i.e. all maps) : MAPREDUCE-4525 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-5422) [Umbrella] Fix invalid state transitions in MRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K resolved MAPREDUCE-5422. -- Resolution: Fixed I am closing this umbrella jira as all the sub tasks are resolved. > [Umbrella] Fix invalid state transitions in MRAppMaster > --- > > Key: MAPREDUCE-5422 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5422 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: mr-am >Affects Versions: 2.0.5-alpha >Reporter: Devaraj K >Assignee: Devaraj K > > There are mutiple invalid state transitions for the state machines present in > MRAppMaster. All these can be handled as part of this umbrell JIRA. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5400) MRAppMaster throws InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED for JobImpl
[ https://issues.apache.org/jira/browse/MAPREDUCE-5400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-5400: - Resolution: Cannot Reproduce Status: Resolved (was: Patch Available) I don't think it is still an issue since MR has undergone many changes after creating the issue. I am closing it, Please reopen it if you see the issue again. > MRAppMaster throws InvalidStateTransitonException: Invalid event: > JOB_TASK_COMPLETED at SUCCEEDED for JobImpl > - > > Key: MAPREDUCE-5400 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5400 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster >Affects Versions: 2.0.5-alpha >Reporter: J.Andreina >Assignee: Devaraj K >Priority: Minor > Attachments: MAPREDUCE-5400-1.patch, MAPREDUCE-5400.patch > > > Step 1: Install cluster with HDFS , MR > Step 2: Execute a job > Step 3: Issue a kill task attempt for which the task has got completed. > Rex@HOST-10-18-91-55:~/NodeAgentTmpDir/installations/hadoop-2.0.5.tar/hadoop-2.0.5/bin> > ./mapred job -kill-task attempt_1373875322959_0032_m_00_0 > No GC_PROFILE is given. Defaults to medium. > 13/07/15 14:46:32 INFO service.AbstractService: > Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited. > 13/07/15 14:46:32 INFO proxy.ResourceManagerProxies: HA Proxy Creation with > xface : interface org.apache.hadoop.yarn.api.ClientRMProtocol > 13/07/15 14:46:33 INFO service.AbstractService: > Service:org.apache.hadoop.yarn.client.YarnClientImpl is started. > Killed task attempt_1373875322959_0032_m_00_0 > Observation: > === > 1. task state has been transitioned from SUCCEEDED to SCHEDULED > 2. For a Succeeded attempt , when client issues Kill , then the client is > notified as killed for a succeeded attempt. > 3. Launched second task_attempt which is succeeded and then killed later on > client request. > 4. Even after the job state transitioned from SUCCEEDED to ERROR , on UI the > state is succeeded > Issue : > = > 1. Client has been notified that the atttempt is killed , but acutually the > attempt is succeeded and the same is displayed in JHS UI. > 2. At App master InvalidStateTransitonException is thrown . > 3. At client side and JHS job has exited with state Finished/succeeded ,At RM > side the state is Finished/Failed. > AM Logs: > > 2013-07-15 14:46:25,461 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: > attempt_1373875322959_0032_m_00_0 TaskAttempt Transitioned from RUNNING > to SUCCEEDED > 2013-07-15 14:46:25,468 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with > attempt attempt_1373875322959_0032_m_00_0 > 2013-07-15 14:46:25,470 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: > task_1373875322959_0032_m_00 Task Transitioned from RUNNING to SUCCEEDED > 2013-07-15 14:46:33,810 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: > task_1373875322959_0032_m_00 Task Transitioned from SUCCEEDED to SCHEDULED > 2013-07-15 14:46:37,344 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with > attempt attempt_1373875322959_0032_m_00_1 > 2013-07-15 14:46:37,344 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: > task_1373875322959_0032_m_00 Task Transitioned from RUNNING to SUCCEEDED > 2013-07-15 14:46:37,345 ERROR [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Can't handle this event > at current state > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > JOB_TASK_COMPLETED at SUCCEEDED > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:866) > at > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:128) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1095) > at > org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:1091) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130) > at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77) >
[jira] [Resolved] (MAPREDUCE-4754) Job is marked as FAILED and also throwing the TransitonException instead of KILLED when issues a KILL command
[
https://issues.apache.org/jira/browse/MAPREDUCE-4754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K resolved MAPREDUCE-4754.
--
Resolution: Cannot Reproduce
I don't think it is still an issue since MR has undergone many changes after
creating the issue. I am closing it, Please reopen it if you see the issue
again.
> Job is marked as FAILED and also throwing the TransitonException instead of
> KILLED when issues a KILL command
> -
>
> Key: MAPREDUCE-4754
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4754
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 2.0.1-alpha, 2.0.2-alpha
>Reporter: Nishan Shetty
>
> {code}
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> JOB_TASK_COMPLETED at KILLED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:301)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:695)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:119)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:893)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:889)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:126)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:75)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6617) flushTimer in JobHistoryEventHandler should purge canceled flushTimerTask
[ https://issues.apache.org/jira/browse/MAPREDUCE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6617: - Status: Open (was: Patch Available) Cancelling patch for further discussion. > flushTimer in JobHistoryEventHandler should purge canceled flushTimerTask > - > > Key: MAPREDUCE-6617 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6617 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.0.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta > Attachments: MAPREDUCE-6617.0.patch > > > In JobHistoryEventHandler, flushTask is not purged after it's canceled so GC > never sweep flushTask. > It can cause memory leak. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6612) Compressing map output error
[ https://issues.apache.org/jira/browse/MAPREDUCE-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15114729#comment-15114729 ] Devaraj K commented on MAPREDUCE-6612: -- As per the stack trace, it looks like you are running the Job in local mode. Do you see the issue when you run the same job in YARN mode(i.e. mapreduce.framework.name=yarn) as well? > Compressing map output error > > > Key: MAPREDUCE-6612 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6612 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: wangjiayou > > I used gzip compressionor to compress map output,but there is error > occurrence.my hadoop version 2.7.1 > Configuration conf = getConf(); > conf.setBoolean(Job.MAP_OUTPUT_COMPRESS, true); > conf.setClass(Job.MAP_OUTPUT_COMPRESS_CODEC, GzipCodec.class, > CompressionCodec.class); > error info: > 2016-01-21 15:02:23 [org.apache.hadoop.mapred.LocalJobRunner]-[INFO] reduce > > copy > 2016-01-21 15:02:23 [org.apache.hadoop.mapred.LocalJobRunner]-[WARN] > job_local1329832043_0001 > java.lang.Exception: > org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in > shuffle in localfetcher#1 > at > org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529) > Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: > error in shuffle in localfetcher#1 > at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376) > at > org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.io.IOException: not a gzip file > at > org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.processBasicHeader(BuiltInGzipDecompressor.java:496) > at > org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.executeHeaderState(BuiltInGzipDecompressor.java:257) > at > org.apache.hadoop.io.compress.zlib.BuiltInGzipDecompressor.decompress(BuiltInGzipDecompressor.java:186) > at > org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:91) > at > org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85) > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:199) > at > org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.shuffle(InMemoryMapOutput.java:97) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:157) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:102) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:85) > 2016-01-21 15:02:23 [org.apache.hadoop.mapred.LocalJobRunner]-[INFO] reduce > > copy > 2016-01-21 15:02:23 [org.apache.hadoop.mapred.LocalJobRunner]-[INFO] reduce > > copy > 2016-01-21 15:02:24 [org.apache.hadoop.mapred.LocalJobRunner]-[INFO] reduce > > copy -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4788) Job are marking as FAILED even if there are no failed tasks in it
[ https://issues.apache.org/jira/browse/MAPREDUCE-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15061879#comment-15061879 ] Devaraj K commented on MAPREDUCE-4788: -- I faced this issue long time back during the version 2.0.4-alpha. I don't think it is still an issue with the changes happened over the time. bq. Is the job history not correct on this state as well? Yes, the job history was also showing the job state as FAILED and the task state as KILLED but we could see the attempt for the task as succeeded. I don't have logs for this issue since it happened some time ago. [~brahmareddy], Can you give details or share the log for the issue you are facing if you have anything? > Job are marking as FAILED even if there are no failed tasks in it > - > > Key: MAPREDUCE-4788 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4788 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.6.0 >Reporter: Devaraj K > Attachments: MAPREDUCE-4788.patch > > > Sometimes Jobs are marking as FAILED and some the tasks are marking as KILLED > in it. > In MRAppMaster, JobFinishEvent is triggering and waiting for the 5000 millis. > If any tasks final state is unknown by this time those tasks are marking as > KILLED and Job state is marking as FAILED. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-4788) Job are marking as FAILED even if there are no failed tasks in it
[ https://issues.apache.org/jira/browse/MAPREDUCE-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K reassigned MAPREDUCE-4788: Assignee: (was: Devaraj K) > Job are marking as FAILED even if there are no failed tasks in it > - > > Key: MAPREDUCE-4788 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4788 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.6.0 >Reporter: Devaraj K > > Sometimes Jobs are marking as FAILED and some the tasks are marking as KILLED > in it. > In MRAppMaster, JobFinishEvent is triggering and waiting for the 5000 millis. > If any tasks final state is unknown by this time those tasks are marking as > KILLED and Job state is marking as FAILED. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4788) Job are marking as FAILED even if there are no failed tasks in it
[ https://issues.apache.org/jira/browse/MAPREDUCE-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059611#comment-15059611 ] Devaraj K commented on MAPREDUCE-4788: -- [~brahmareddy], If you would like to provide a patch for this, Please take it. Thanks. > Job are marking as FAILED even if there are no failed tasks in it > - > > Key: MAPREDUCE-4788 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4788 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.6.0 >Reporter: Devaraj K >Assignee: Devaraj K > > Sometimes Jobs are marking as FAILED and some the tasks are marking as KILLED > in it. > In MRAppMaster, JobFinishEvent is triggering and waiting for the 5000 millis. > If any tasks final state is unknown by this time those tasks are marking as > KILLED and Job state is marking as FAILED. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6513) MR job got hanged forever when one NM unstable for some time
[ https://issues.apache.org/jira/browse/MAPREDUCE-6513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14965034#comment-14965034 ] Devaraj K commented on MAPREDUCE-6513: -- I agree with [~chen317], failed maps have highest priority(PRIORITY_FAST_FAIL_MAP) than the reducers(PRIORITY_REDUCE), MR AM should get a container for failed map than reducer here if resources are available for map. [~Jobo]/[~varun_saxena], What is the map memory requesting for this Job? And do you have chance to share the complete log of MR App Master? > MR job got hanged forever when one NM unstable for some time > > > Key: MAPREDUCE-6513 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6513 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, resourcemanager >Affects Versions: 2.7.0 >Reporter: Bob >Assignee: Varun Saxena >Priority: Critical > > when job is in-progress which is having more tasks,one node became unstable > due to some OS issue.After the node became unstable, the map on this node > status changed to KILLED state. > Currently maps which were running on unstable node are rescheduled, and all > are in scheduled state and wait for RM assign container.Seen ask requests for > map till Node is good (all those failed), there are no ask request after > this. But AM keeps on preempting the reducers (it's recycling). > Finally reducers are waiting for complete mappers and mappers did n't get > container.. > My Question Is: > > why map requests did not sent AM ,once after node recovery.? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-6426:
-
Hadoop Flags: Reviewed
+1, committing it shortly.
TestShuffleHandler fails in trunk
-
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
Attachments: MAPREDUCE-6426.000.patch, MAPREDUCE-6426.001.patch
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6426) TestShuffleHandler#testGetMapOutputInfo is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-6426:
-
Affects Version/s: (was: 2.8.0)
2.7.2
Summary: TestShuffleHandler#testGetMapOutputInfo is failing
(was: TestShuffleHandler fails in trunk)
TestShuffleHandler#testGetMapOutputInfo is failing
--
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.7.2
Reporter: Devaraj K
Assignee: zhihai xu
Attachments: MAPREDUCE-6426.000.patch, MAPREDUCE-6426.001.patch
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6426) TestShuffleHandler#testGetMapOutputInfo is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-6426:
-
Resolution: Fixed
Fix Version/s: 2.7.2
Status: Resolved (was: Patch Available)
Thanks [~zxu] for the patch.
Committed to trunk, branch-2 and branch-2.7.
TestShuffleHandler#testGetMapOutputInfo is failing
--
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.7.2
Reporter: Devaraj K
Assignee: zhihai xu
Fix For: 2.7.2
Attachments: MAPREDUCE-6426.000.patch, MAPREDUCE-6426.001.patch
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618165#comment-14618165
]
Devaraj K commented on MAPREDUCE-6426:
--
Thanks [~zxu] for addressing comments and updating patch.
It looks good to me, I will commit it tomorrow if there are no other
comments/objections.
TestShuffleHandler fails in trunk
-
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
Attachments: MAPREDUCE-6426.000.patch, MAPREDUCE-6426.001.patch
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14617953#comment-14617953
]
Devaraj K commented on MAPREDUCE-6426:
--
Thanks [~zxu] for the analysis and patch for the issue.
{code:xml}
Assert.assertEquals(failures.size(), 0);
{code}
Can you exchange the args here and also add a message for this assertion as
part of the patch?
TestShuffleHandler fails in trunk
-
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
Attachments: MAPREDUCE-6426.000.patch
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
Devaraj K created MAPREDUCE-6426:
Summary: TestShuffleHandler fails in trunk
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616812#comment-14616812
]
Devaraj K commented on MAPREDUCE-6426:
--
It is passing locally and failing in the Jenkins build.
[~zxu], I am assigning to you, Can you please have a look into this test
failure?
TestShuffleHandler fails in trunk
-
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6425) ShuffleHandler passes wrong base parameter to getMapOutputInfo if mapId is not in the cache.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-6425:
-
Hadoop Flags: Reviewed
Thanks [~zxu] for the updated patch.
+1, latest patch looks good to me, will commit it shortly.
ShuffleHandler passes wrong base parameter to getMapOutputInfo if mapId is
not in the cache.
--
Key: MAPREDUCE-6425
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6425
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6425.000.patch, MAPREDUCE-6425.001.patch
ShuffleHandler passes wrong {{base}} parameter to {{getMapOutputInfo}} if
mapId is not in the cache.
{{getMapOutputInfo}} expected the {{base}} parameter is
{{getBaseLocation(jobId, user) + mapId}}
When it is called inside populateHeaders, the {{base}} parameter is set
correctly
{code}
String base = outputBaseStr + mapId;
MapOutputInfo outputInfo = getMapOutputInfo(base, mapId, reduce,
user);
{code}
When it is called outside populateHeaders, the {{base}} parameter is set
wrongly to outputBasePathStr after number of mapId cached exceeds
{{mapOutputMetaInfoCacheSize}}.
{code}
String outputBasePathStr = getBaseLocation(jobId, user);
MapOutputInfo info = mapOutputInfoMap.get(mapId);
if (info == null) {
info = getMapOutputInfo(outputBasePathStr, mapId, reduceId, user);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6425) ShuffleHandler passes wrong base parameter to getMapOutputInfo if mapId is not in the cache.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-6425:
-
Resolution: Fixed
Fix Version/s: 2.7.2
Status: Resolved (was: Patch Available)
Thanks [~zxu].
Committed to trunk, branch-2 and branch-2.7.
ShuffleHandler passes wrong base parameter to getMapOutputInfo if mapId is
not in the cache.
--
Key: MAPREDUCE-6425
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6425
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Reporter: zhihai xu
Assignee: zhihai xu
Fix For: 2.7.2
Attachments: MAPREDUCE-6425.000.patch, MAPREDUCE-6425.001.patch
ShuffleHandler passes wrong {{base}} parameter to {{getMapOutputInfo}} if
mapId is not in the cache.
{{getMapOutputInfo}} expected the {{base}} parameter is
{{getBaseLocation(jobId, user) + mapId}}
When it is called inside populateHeaders, the {{base}} parameter is set
correctly
{code}
String base = outputBaseStr + mapId;
MapOutputInfo outputInfo = getMapOutputInfo(base, mapId, reduce,
user);
{code}
When it is called outside populateHeaders, the {{base}} parameter is set
wrongly to outputBasePathStr after number of mapId cached exceeds
{{mapOutputMetaInfoCacheSize}}.
{code}
String outputBasePathStr = getBaseLocation(jobId, user);
MapOutputInfo info = mapOutputInfoMap.get(mapId);
if (info == null) {
info = getMapOutputInfo(outputBasePathStr, mapId, reduceId, user);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6425) ShuffleHandler passes wrong base parameter to getMapOutputInfo if mapId is not in the cache.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14613198#comment-14613198
]
Devaraj K commented on MAPREDUCE-6425:
--
Nice catch [~zxu], appreciate your efforts.
Can you add a functional test to demonstrate the issue and fails without the
code changes?
ShuffleHandler passes wrong base parameter to getMapOutputInfo if mapId is
not in the cache.
--
Key: MAPREDUCE-6425
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6425
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6425.000.patch
ShuffleHandler passes wrong {{base}} parameter to {{getMapOutputInfo}} if
mapId is not in the cache.
{{getMapOutputInfo}} expected the {{base}} parameter is
{{getBaseLocation(jobId, user) + mapId}}
When it is called inside populateHeaders, the {{base}} parameter is set
correctly
{code}
String base = outputBaseStr + mapId;
MapOutputInfo outputInfo = getMapOutputInfo(base, mapId, reduce,
user);
{code}
When it is called outside populateHeaders, the {{base}} parameter is set
wrongly to outputBasePathStr after number of mapId cached exceeds
{{mapOutputMetaInfoCacheSize}}.
{code}
String outputBasePathStr = getBaseLocation(jobId, user);
MapOutputInfo info = mapOutputInfoMap.get(mapId);
if (info == null) {
info = getMapOutputInfo(outputBasePathStr, mapId, reduceId, user);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6425) ShuffleHandler passes wrong base parameter to getMapOutputInfo if mapId is not in the cache.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-6425:
-
Target Version/s: 2.7.2
ShuffleHandler passes wrong base parameter to getMapOutputInfo if mapId is
not in the cache.
--
Key: MAPREDUCE-6425
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6425
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6425.000.patch
ShuffleHandler passes wrong {{base}} parameter to {{getMapOutputInfo}} if
mapId is not in the cache.
{{getMapOutputInfo}} expected the {{base}} parameter is
{{getBaseLocation(jobId, user) + mapId}}
When it is called inside populateHeaders, the {{base}} parameter is set
correctly
{code}
String base = outputBaseStr + mapId;
MapOutputInfo outputInfo = getMapOutputInfo(base, mapId, reduce,
user);
{code}
When it is called outside populateHeaders, the {{base}} parameter is set
wrongly to outputBasePathStr after number of mapId cached exceeds
{{mapOutputMetaInfoCacheSize}}.
{code}
String outputBasePathStr = getBaseLocation(jobId, user);
MapOutputInfo info = mapOutputInfoMap.get(mapId);
if (info == null) {
info = getMapOutputInfo(outputBasePathStr, mapId, reduceId, user);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2631) Potential resource leaks in BinaryProtocol$TeeOutputStream.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-2631:
-
Status: Open (was: Patch Available)
Cancelling patch to fix the test failure.
Potential resource leaks in BinaryProtocol$TeeOutputStream.java
---
Key: MAPREDUCE-2631
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2631
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Ravi Teja Ch N V
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-2631.patch, 0002-MAPREDUCE-2631.patch,
MAPREDUCE-2631.02.patch, MAPREDUCE-2631.1.patch, MAPREDUCE-2631.2.patch,
MAPREDUCE-2631.3.patch, MAPREDUCE-2631.patch
{code:title=BinaryProtocol$TeeOutputStream.java|borderStyle=solid}
public void close() throws IOException {
flush();
file.close();
out.close();
}
{code}
In the above code, if the file.close() throws any exception out will not be
closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-2631) Potential resource leaks in BinaryProtocol$TeeOutputStream.java
[
https://issues.apache.org/jira/browse/MAPREDUCE-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-2631:
-
Labels: (was: BB2015-05-TBR)
Component/s: (was: jobtracker)
Potential resource leaks in BinaryProtocol$TeeOutputStream.java
---
Key: MAPREDUCE-2631
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2631
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Ravi Teja Ch N V
Assignee: Sunil G
Attachments: 0001-MAPREDUCE-2631.patch, 0002-MAPREDUCE-2631.patch,
MAPREDUCE-2631.02.patch, MAPREDUCE-2631.1.patch, MAPREDUCE-2631.2.patch,
MAPREDUCE-2631.3.patch, MAPREDUCE-2631.patch
{code:title=BinaryProtocol$TeeOutputStream.java|borderStyle=solid}
public void close() throws IOException {
flush();
file.close();
out.close();
}
{code}
In the above code, if the file.close() throws any exception out will not be
closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3047) FileOutputCommitter throws wrong type of exception when calling abortTask() to handle a directory without permission
[ https://issues.apache.org/jira/browse/MAPREDUCE-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3047: - Target Version/s: 2.8.0 Status: Open (was: Patch Available) FileOutputCommitter throws wrong type of exception when calling abortTask() to handle a directory without permission Key: MAPREDUCE-3047 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3047 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: JiangKai Assignee: Brahma Reddy Battula Priority: Minor Attachments: MAPREDUCE-3047-003.patch, MAPREDUCE-3047-004.patch, MAPREDUCE-3047-1.patch, MAPREDUCE-3047-2.patch, MAPREDUCE-3047.patch When FileOutputCommitter calls abortTask() to create a temp directory, if the user has no permission to access the directory, or a file with the same name has existed, of course it will fail, however the system will output the error information into the log file instead of throwing an exception.As a result, when the temp directory is needed later, since the temp directory hasn't been created yet, system will throw an exception to tell user that the temp directory doesn't exist.In my opinion, the exception is not exact and the error infomation will confuse users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3047) FileOutputCommitter throws wrong type of exception when calling abortTask() to handle a directory without permission
[ https://issues.apache.org/jira/browse/MAPREDUCE-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14605470#comment-14605470 ] Devaraj K commented on MAPREDUCE-3047: -- Thanks [~brahmareddy] for rebasing the patch. Can you also add a test case for the change as part of the patch? FileOutputCommitter throws wrong type of exception when calling abortTask() to handle a directory without permission Key: MAPREDUCE-3047 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3047 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: JiangKai Assignee: Brahma Reddy Battula Priority: Minor Attachments: MAPREDUCE-3047-003.patch, MAPREDUCE-3047-004.patch, MAPREDUCE-3047-1.patch, MAPREDUCE-3047-2.patch, MAPREDUCE-3047.patch When FileOutputCommitter calls abortTask() to create a temp directory, if the user has no permission to access the directory, or a file with the same name has existed, of course it will fail, however the system will output the error information into the log file instead of throwing an exception.As a result, when the temp directory is needed later, since the temp directory hasn't been created yet, system will throw an exception to tell user that the temp directory doesn't exist.In my opinion, the exception is not exact and the error infomation will confuse users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6419) JobHistoryServer doesn't sort properly based on Job ID when Job id's exceed 9999
Devaraj K created MAPREDUCE-6419: Summary: JobHistoryServer doesn't sort properly based on Job ID when Job id's exceed Key: MAPREDUCE-6419 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6419 Project: Hadoop Map/Reduce Issue Type: Bug Components: webapps Affects Versions: 2.7.0 Reporter: Devaraj K When Job id's exceed , JobHistoryServer is not sorting properly based on the Job ID. It is mixing the jobs having with other jobs considering only the first four digits of the job id. The same problem could exist for Job Map tasks and Reduce tasks table as well. It is similar to the issue YARN-3840 exists for YARN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6419) JobHistoryServer doesn't sort properly based on Job ID when Job id's exceed 9999
[ https://issues.apache.org/jira/browse/MAPREDUCE-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6419: - Assignee: Mohammad Shahid Khan JobHistoryServer doesn't sort properly based on Job ID when Job id's exceed Key: MAPREDUCE-6419 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6419 Project: Hadoop Map/Reduce Issue Type: Bug Components: webapps Affects Versions: 2.7.0 Reporter: Devaraj K Assignee: Mohammad Shahid Khan When Job id's exceed , JobHistoryServer is not sorting properly based on the Job ID. It is mixing the jobs having with other jobs considering only the first four digits of the job id. The same problem could exist for Job Map tasks and Reduce tasks table as well. It is similar to the issue YARN-3840 exists for YARN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6419) JobHistoryServer doesn't sort properly based on Job ID when Job id's exceed 9999
[ https://issues.apache.org/jira/browse/MAPREDUCE-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14605451#comment-14605451 ] Devaraj K commented on MAPREDUCE-6419: -- [~mohdshahidkhan], I am assigning this issue to you as you have already started working on the same for YARN. Please reassign to me if you can't handle this. JobHistoryServer doesn't sort properly based on Job ID when Job id's exceed Key: MAPREDUCE-6419 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6419 Project: Hadoop Map/Reduce Issue Type: Bug Components: webapps Affects Versions: 2.7.0 Reporter: Devaraj K Assignee: Mohammad Shahid Khan When Job id's exceed , JobHistoryServer is not sorting properly based on the Job ID. It is mixing the jobs having with other jobs considering only the first four digits of the job id. The same problem could exist for Job Map tasks and Reduce tasks table as well. It is similar to the issue YARN-3840 exists for YARN. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3047) FileOutputCommitter throws wrong type of exception when calling abortTask() to handle a directory without permission
[ https://issues.apache.org/jira/browse/MAPREDUCE-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3047: - Labels: (was: BB2015-05-TBR) FileOutputCommitter throws wrong type of exception when calling abortTask() to handle a directory without permission Key: MAPREDUCE-3047 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3047 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: JiangKai Priority: Minor Attachments: MAPREDUCE-3047-003.patch, MAPREDUCE-3047-004.patch, MAPREDUCE-3047-1.patch, MAPREDUCE-3047-2.patch, MAPREDUCE-3047.patch When FileOutputCommitter calls abortTask() to create a temp directory, if the user has no permission to access the directory, or a file with the same name has existed, of course it will fail, however the system will output the error information into the log file instead of throwing an exception.As a result, when the temp directory is needed later, since the temp directory hasn't been created yet, system will throw an exception to tell user that the temp directory doesn't exist.In my opinion, the exception is not exact and the error infomation will confuse users. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
