[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288841#comment-17288841 ] Devaraj Kavali commented on MAPREDUCE-6749: --- Thanks [~brahmareddy] and [~BilwaST] for driving this and for the test report. [~brahmareddy] Please feel free to review the patches and merge to the branch. I will also have a look at those. > MR AM should reuse containers for Map/Reduce Tasks > -- > > Key: MAPREDUCE-6749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Devaraj Kavali >Assignee: Devaraj Kavali >Priority: Major > Attachments: Container Reuse Performance Report.pdf, > MAPREDUCE-6749-Container Reuse-v0.pdf > > > It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers > for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288837#comment-17288837 ] Brahma Reddy Battula commented on MAPREDUCE-6749: - [~BilwaST] thanks for attaching the performance report. Looks performance is better when we enable this feaute.. It can be more benefit when more number of container's are there.. you can give how much performabce gain with this also. [~devaraj] can we merge pending jira's and merge this feature..THinking to have for 3.3.1 relase based on stablity. > MR AM should reuse containers for Map/Reduce Tasks > -- > > Key: MAPREDUCE-6749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Devaraj Kavali >Assignee: Devaraj Kavali >Priority: Major > Attachments: Container Reuse Performance Report.pdf, > MAPREDUCE-6749-Container Reuse-v0.pdf > > > It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers > for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288829#comment-17288829 ] Brahma Reddy Battula commented on MAPREDUCE-7169: - [~ahussein] , [~lichen1109] and [~epayne] [~BilwaST] this Jira marked for 3.3.1 release which is going to release near future, please let me know update on this. > Speculative attempts should not run on the same node > > > Key: MAPREDUCE-7169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: yarn >Affects Versions: 2.7.2 >Reporter: Lee chen >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch, > MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch, > MAPREDUCE-7169.006.patch, MAPREDUCE-7169.007.patch, > image-2018-12-03-09-54-07-859.png > > > I found in all versions of yarn, Speculative Execution may set the > speculative task to the node of original task.What i have read is only it > will try to have one more task attempt. haven't seen any place mentioning not > on same node.It is unreasonable.If the node have some problems lead to tasks > execution will be very slow. and then placement the speculative task to same > node cannot help the problematic task. > In our cluster （version 2.7.2，2700 nodes），this phenomenon appear > almost everyday. > !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288590#comment-17288590
]
Eric Badger commented on MAPREDUCE-7320:
I think leaving the test logs around is a feature, not a bug. I agree with
[~Jim_Brennan] on keeping them around and deleting them at the start of the
next run of unit tests
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288425#comment-17288425
]
Ahmed Hussein commented on MAPREDUCE-7320:
--
You have a reasonable point [~Jim_Brennan]. I agree it would be inconvenient to
modify the code just to look to the output and the logs.
I will disable the cleaning in {{@AfterClass}}.
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288420#comment-17288420
]
Jim Brennan commented on MAPREDUCE-7320:
I would prefer to cleanup at the start of the test. I know we have had cases
in the past where I needed to look at these logs after our automated unit test
builds. I wouldn't want to have to modify the code to enable that.
I am curious how others feel about this?
cc: [~epayne], [~jeagles], [~ebadger], [~jhung]
> ClusterMapReduceTestCase does not clean directories
> ---
>
> Key: MAPREDUCE-7320
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of
> directories and folders without cleaning them up.
> For example:
> {code:bash}
> men test -Dtest=TestMRJobClient{code}
> generates the following directories:
> {code:bash}
> - target
>-+ ConfigurableMiniMRCluster_315090884
>-+ ConfigurableMiniMRCluster_1335188990
>-+ ConfigurableMiniMRCluster_1973037511
>-+ test-dir
> -+ dfs
> -+ hadopp-XYZ-01
> -+ hadopp-XYZ-02
> -+ hadopp-XYZ-03
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7323) Remove job_history_summary.py
[
https://issues.apache.org/jira/browse/MAPREDUCE-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka updated MAPREDUCE-7323:
-
Fix Version/s: 3.2.3
2.10.2
3.3.1
3.1.5
3.4.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to all the active branches.
> Remove job_history_summary.py
> -
>
> Key: MAPREDUCE-7323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7323
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.1.5, 3.3.1, 2.10.2, 3.2.3
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Fix the following syntax error in Python 3:
> {noformat}
> aajisaka@b23b6a126ee7:~/hadoop$ python3
> hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/job_history_summary.py
> File
> "hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/job_history_summary.py",
> line 73
> print "Name reduce-output-bytes shuffle-finish reduce-finish"
> ^
> SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Name
> reduce-output-bytes shuffle-finish reduce-finish")?
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7323) Remove job_history_summary.py
[
https://issues.apache.org/jira/browse/MAPREDUCE-7323?focusedWorklogId=555823=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-555823
]
ASF GitHub Bot logged work on MAPREDUCE-7323:
-
Author: ASF GitHub Bot
Created on: 22/Feb/21 14:15
Start Date: 22/Feb/21 14:15
Worklog Time Spent: 10m
Work Description: aajisaka commented on pull request #2712:
URL: https://github.com/apache/hadoop/pull/2712#issuecomment-783405188
Merged. Thank you @tasanuma
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 555823)
Time Spent: 0.5h (was: 20m)
> Remove job_history_summary.py
> -
>
> Key: MAPREDUCE-7323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7323
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Fix the following syntax error in Python 3:
> {noformat}
> aajisaka@b23b6a126ee7:~/hadoop$ python3
> hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/job_history_summary.py
> File
> "hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/job_history_summary.py",
> line 73
> print "Name reduce-output-bytes shuffle-finish reduce-finish"
> ^
> SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Name
> reduce-output-bytes shuffle-finish reduce-finish")?
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work logged] (MAPREDUCE-7323) Remove job_history_summary.py
[
https://issues.apache.org/jira/browse/MAPREDUCE-7323?focusedWorklogId=555822=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-555822
]
ASF GitHub Bot logged work on MAPREDUCE-7323:
-
Author: ASF GitHub Bot
Created on: 22/Feb/21 14:15
Start Date: 22/Feb/21 14:15
Worklog Time Spent: 10m
Work Description: aajisaka merged pull request #2712:
URL: https://github.com/apache/hadoop/pull/2712
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 555822)
Time Spent: 20m (was: 10m)
> Remove job_history_summary.py
> -
>
> Key: MAPREDUCE-7323
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7323
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Fix the following syntax error in Python 3:
> {noformat}
> aajisaka@b23b6a126ee7:~/hadoop$ python3
> hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/job_history_summary.py
> File
> "hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/job_history_summary.py",
> line 73
> print "Name reduce-output-bytes shuffle-finish reduce-finish"
> ^
> SyntaxError: Missing parentheses in call to 'print'. Did you mean print("Name
> reduce-output-bytes shuffle-finish reduce-finish")?
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
