[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks

2021-03-29 Thread Brahma Reddy Battula (Jira)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17311099#comment-17311099
 ] 

Brahma Reddy Battula commented on MAPREDUCE-6749:
-

[~devaraj] there are conflicts while rebasing against the trunk as this branch 
was old. Hence i am thinking create a new branch for this,please let me know 
your thoughts on this..? 

 

> MR AM should reuse containers for Map/Reduce Tasks
> --
>
> Key: MAPREDUCE-6749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster, mrv2
>Reporter: Devaraj Kavali
>Assignee: Devaraj Kavali
>Priority: Major
> Attachments: Container Reuse Performance Report.pdf, 
> MAPREDUCE-6749-Container Reuse-v0.pdf
>
>
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers 
> for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks

2021-02-22 Thread Devaraj Kavali (Jira)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288841#comment-17288841
 ] 

Devaraj Kavali commented on MAPREDUCE-6749:
---

Thanks [~brahmareddy] and [~BilwaST] for driving this and for the test report. 

 

[~brahmareddy] Please feel free to review the patches and merge to the branch. 
I will also have a look at those.

> MR AM should reuse containers for Map/Reduce Tasks
> --
>
> Key: MAPREDUCE-6749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster, mrv2
>Reporter: Devaraj Kavali
>Assignee: Devaraj Kavali
>Priority: Major
> Attachments: Container Reuse Performance Report.pdf, 
> MAPREDUCE-6749-Container Reuse-v0.pdf
>
>
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers 
> for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks

2021-02-22 Thread Brahma Reddy Battula (Jira)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288837#comment-17288837
 ] 

Brahma Reddy Battula commented on MAPREDUCE-6749:
-

[~BilwaST] thanks for attaching the performance report. Looks performance is 
better when we enable this feaute.. It can be more benefit when more number of 
container's are there.. you can give how much performabce gain with this also.

 

[~devaraj] can we merge pending jira's and merge this feature..THinking to have 
for 3.3.1 relase based on stablity.

> MR AM should reuse containers for Map/Reduce Tasks
> --
>
> Key: MAPREDUCE-6749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster, mrv2
>Reporter: Devaraj Kavali
>Assignee: Devaraj Kavali
>Priority: Major
> Attachments: Container Reuse Performance Report.pdf, 
> MAPREDUCE-6749-Container Reuse-v0.pdf
>
>
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers 
> for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks

2021-02-17 Thread Bilwa S T (Jira)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285715#comment-17285715
 ] 

Bilwa S T commented on MAPREDUCE-6749:
--

Attached test report for this feature. 

> MR AM should reuse containers for Map/Reduce Tasks
> --
>
> Key: MAPREDUCE-6749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster, mrv2
>Reporter: Devaraj Kavali
>Assignee: Devaraj Kavali
>Priority: Major
> Attachments: Container Reuse Performance Report.pdf, 
> MAPREDUCE-6749-Container Reuse-v0.pdf
>
>
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers 
> for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks

2020-09-22 Thread Devaraj Kavali (Jira)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200190#comment-17200190
 ] 

Devaraj Kavali commented on MAPREDUCE-6749:
---

Thanks [~brahmareddy] for checking on this. There are some more tasks need to 
be done for this feature, I am occupied with some other things, please free to 
take it up these if you are interested.

> MR AM should reuse containers for Map/Reduce Tasks
> --
>
> Key: MAPREDUCE-6749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster, mrv2
>Reporter: Devaraj Kavali
>Assignee: Devaraj Kavali
>Priority: Major
> Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf
>
>
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers 
> for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks

2020-09-22 Thread Brahma Reddy Battula (Jira)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200059#comment-17200059
 ] 

Brahma Reddy Battula commented on MAPREDUCE-6749:
-

[~devaraj] looks no progress on this, we would like know the status and work on 
this.

> MR AM should reuse containers for Map/Reduce Tasks
> --
>
> Key: MAPREDUCE-6749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster, mrv2
>Reporter: Devaraj Kavali
>Assignee: Devaraj Kavali
>Priority: Major
> Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf
>
>
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers 
> for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks

2016-10-18 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15585527#comment-15585527
 ] 

Naganarasimha G R commented on MAPREDUCE-6749:
--

Thanks for creating the branch [~devaraj.k] !

> MR AM should reuse containers for Map/Reduce Tasks
> --
>
> Key: MAPREDUCE-6749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster, mrv2
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf
>
>
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers 
> for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks

2016-10-17 Thread Devaraj K (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583132#comment-15583132
 ] 

Devaraj K commented on MAPREDUCE-6749:
--

I have created a branch MR-6749 for this.

> MR AM should reuse containers for Map/Reduce Tasks
> --
>
> Key: MAPREDUCE-6749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster, mrv2
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf
>
>
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers 
> for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks

2016-10-13 Thread Devaraj K (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574159#comment-15574159
 ] 

Devaraj K commented on MAPREDUCE-6749:
--

Thanks [~Naganarasimha] for the responses and discussion.
bq. hope we can create a new branch and get the things in there so that its 
easier for others to have a look before it gets into the trunk or main stream 
branches
Sure, I will create a branch for this.

bq. If there was a way for admin to enforce it it would be usefull. If its just 
client level configuration it just adds into already long list of 
configurations and users will not be clear what to configure for it. And 
besides would it be better to have just how many tasks can reuse a given 
container and try to avoid for Map and Reduce seperately ?
I am thinking that we can provide comprehensive way to control this feature, I 
can convince that it will be an another configuration for the user to configure 
it. We can discuss this in MAPREDUCE-6772/MAPREDUCE-6773.

bq. Btw it could be also good to introduce a metric for number of Map Tasks or 
Reduce tasks which has reused the containers
Good thought, we can have metrics for this.


bq. This was the problem which we generally faced and difficult for the 
customers to understand that entire log is not for the task attempt, so was 
wondering to have any better approach to this.
We can think of displaying the part of the container log(logs generated for a 
task attempt) in JHS Web UI instead of the whole container logs by discarding 
the other task attempt logs. Do you think this sounds ok or any better way?


> MR AM should reuse containers for Map/Reduce Tasks
> --
>
> Key: MAPREDUCE-6749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster, mrv2
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf
>
>
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers 
> for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks

2016-10-13 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573131#comment-15573131
 ] 

Naganarasimha G R commented on MAPREDUCE-6749:
--

Hi [~devaraj.k],
Thanks for the detailed explanation As it would involve considerable 
modifications and changes in core code, hope we can create a new branch and get 
the things in there so that its easier for others to have a look before it gets 
into the trunk or main stream branches

bq. I think the limit configuration for no of map/reduce reuse containers would 
allow other applications to start running without waiting for the Job to be 
finished when reuse is enabled. If there is a big Job running which could 
occupy the entire cluster, and then any high priority application gets 
submitted this limit for maps/reduce container would probably give a room for 
high priority application to start running without preempting the containers of 
the previous Job. By default there is no limit for number of containers to be 
reused and if any user/Job wanted to have this constraint they can configure it.
Yes i understand it thanks for the explanation, but issue would be how the 
application knows whats the right configuration for these, in application per 
se they would think it would be always right to run all the tasks in the given 
container than launching more containers. If there was a way for admin to 
enforce it it would be usefull. If its just client level configuration it just 
adds into already long list of configurations and users will not be clear what 
to configure for it. And besides would it be better to have just how many tasks 
can reuse a given container and try to avoid for Map and Reduce seperately ?

Btw it could be also good to introduce a metric for number of Map Tasks or 
Reduce tasks which has reused the containers

bq. If you want to have a try this feature, you can apply MAPREDUCE-6773, 
MAPREDUCE-6781, MAPREDUCE-6784, MAPREDUCE-6785, MAPREDUCE-6786 and then try 
this feature. 
Sure Deva will try it over the weekend and update you, anyway started to take 
look at them

bq. Here we should note that the whole container log which is displaying for 
TaskAttempt is not applicable to the TaskAttempt and the log can be identified 
easily which part applicable to it.
This was the problem which we generally faced and difficult for the customers 
to understand that entire log is not for the task attempt, so was wondering to 
have any better approach to this. 








> MR AM should reuse containers for Map/Reduce Tasks
> --
>
> Key: MAPREDUCE-6749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster, mrv2
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf
>
>
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers 
> for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks

2016-10-05 Thread Devaraj K (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550874#comment-15550874
 ] 

Devaraj K commented on MAPREDUCE-6749:
--

Thanks [~Naganarasimha] for looking into this and looking forward to work on 
this with you.

{quote}
Is it required to have number of total Maps(/reducers) to be run using 
container reuse? i felt we can just keep whether to reuse the containers are 
not and one assumption that map containers will be reused only for map and 
reduce only for reduce
{quote}
I think the limit configuration for no of map/reduce reuse containers would 
allow other applications to start running without waiting for the Job to be 
finished when reuse is enabled. If there is a big Job running which could 
occupy the entire cluster, and then any high priority application gets 
submitted this limit for maps/reduce container would probably give a room for 
high priority application to start running without preempting the containers of 
the previous Job. By default there is no limit for number of containers to be 
reused and if any user/Job wanted to have this constraint they can configure 
it. 

{quote}
did not understand the need of RMContainerReuseRequestor.java, may be i will go 
through MAPREDUCE-6773 but was expecting more like just TaskAttemptIimpl gets 
TA_DONE event was expecting it do have a new ReuseContainerTransition where in 
if required it will reassign a new task to it else will continue to 
CLEANUP_CONTAINER_TRANSITION. may be we can discuss more on this in 
MAPREDUCE-6773 ?
{quote}
RMContainerReuseRequestor has the functionality for checking the container 
reuse constraints and adding to reuse containers if it is eligible. This class 
abstracts the reuse functionality from regular RMContainerRequestor and 
RMContainerReuseRequestor will be used only when the RMContainer reuse is 
enabled. If you want to have a try this feature, you can apply MAPREDUCE-6773, 
MAPREDUCE-6781, MAPREDUCE-6784, MAPREDUCE-6785, MAPREDUCE-6786 and then try 
this feature. I would be happy to get your feedback or suggestions if any.

{quote}
If we have some conclusion on issue 2, then we can think of whether only 
TaskAttemptImpl state is sufficient or we need to modify others too.
{quote}
Sure, We can discuss it further if required.

{quote} Logs generated by the tasks in a container will be appended in the same 
container log file we need to ensure we are able to display trhough JHS and 
other places where ever required to display
{quote}
TaskAttempt is linked to the container and the corresponding container log 
would be displayed for TaskAttempt through the JHS. Here we should note that 
the whole container log which is displaying for TaskAttempt is not applicable 
to the TaskAttempt and the log can be identified easily which part applicable 
to it. 

> MR AM should reuse containers for Map/Reduce Tasks
> --
>
> Key: MAPREDUCE-6749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster, mrv2
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf
>
>
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers 
> for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks

2016-10-05 Thread Naganarasimha G R (JIRA)

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549700#comment-15549700
 ] 

Naganarasimha G R commented on MAPREDUCE-6749:
--

Thanks [~devaraj.k] for the document and raising these jira's, it has been 
always discussing but no progress on this topic lets try to make some progress 
for this feature here .
Few queries:
# Is it required to have number of total Maps(/reducers) to be run using 
container reuse? i felt we can just keep whether to reuse the containers are 
not and one assumption that map containers will be reused only for map and 
reduce only for reduce 
# did not understand the need of *RMContainerReuseRequestor.java*, may be i 
will go through MAPREDUCE-6773 but was expecting more like just  
TaskAttemptIimpl gets TA_DONE event was expecting it do have a new 
ReuseContainerTransition where in if required it will reassign a new task to it 
else will continue to *CLEANUP_CONTAINER_TRANSITION*. may be we can discuss 
more on this in MAPREDUCE-6773 ?
# If we have some conclusion on issue 2, then we can think of whether only 
*TaskAttemptImpl* state is sufficient or we need to modify others too.
# *Logs generated by the tasks in a container will be appended in the same 
container log file* we need to ensure we are able to display trhough JHS and 
other places where ever required to display

> MR AM should reuse containers for Map/Reduce Tasks
> --
>
> Key: MAPREDUCE-6749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: applicationmaster, mrv2
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf
>
>
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers 
> for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org