[jira] [Commented] (MAPREDUCE-7261) Memory efficiency in speculator
[
https://issues.apache.org/jira/browse/MAPREDUCE-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095538#comment-17095538
]
Hadoop QA commented on MAPREDUCE-7261:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
57s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
The patch generated 0 new + 87 unchanged - 6 fixed = 87 total (was 93) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m
14s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7776/artifact/out/Dockerfile
|
| JIRA Issue | MAPREDUCE-7261 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13001598/MAPREDUCE-7261.003.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 3eab61d82ec1 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 410c605aec3 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
[jira] [Comment Edited] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[
https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095502#comment-17095502
]
Bilwa S T edited comment on MAPREDUCE-7169 at 4/29/20, 2:40 PM:
Hi [~ahussein]
{code:java}
The same node will be picked if there are no other available nodes. In
MAPREDUCE-7169.005.patch , what is the expected behavior if the resources
available to run the taskAttempt are only available on the same node? I do not
see this case in the unit test.{code}
Container is not assigned until resources are available on other node. Task
will wait until it gets container on other node. As per use case we do not want
container to be launched on same node as node might have a problem
was (Author: bilwast):
Hi [~ahussein]
{code:java}
The same node will be picked if there are no other available nodes. In
MAPREDUCE-7169.005.patch , what is the expected behavior if the resources
available to run the taskAttempt are only available on the same node? I do not
see this case in the unit test.{code}
Container is not assigned until resources are available on other node.
> Speculative attempts should not run on the same node
>
>
> Key: MAPREDUCE-7169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: yarn
>Affects Versions: 2.7.2
>Reporter: Lee chen
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch,
> MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch,
> image-2018-12-03-09-54-07-859.png
>
>
> I found in all versions of yarn, Speculative Execution may set the
> speculative task to the node of original task.What i have read is only it
> will try to have one more task attempt. haven't seen any place mentioning not
> on same node.It is unreasonable.If the node have some problems lead to tasks
> execution will be very slow. and then placement the speculative task to same
> node cannot help the problematic task.
> In our cluster （version 2.7.2，2700 nodes），this phenomenon appear
> almost everyday.
> !image-2018-12-03-09-54-07-859.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[
https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095502#comment-17095502
]
Bilwa S T commented on MAPREDUCE-7169:
--
Hi [~ahussein]
{code:java}
The same node will be picked if there are no other available nodes. In
MAPREDUCE-7169.005.patch , what is the expected behavior if the resources
available to run the taskAttempt are only available on the same node? I do not
see this case in the unit test.{code}
Container is not assigned until resources are available on other node.
> Speculative attempts should not run on the same node
>
>
> Key: MAPREDUCE-7169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: yarn
>Affects Versions: 2.7.2
>Reporter: Lee chen
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch,
> MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch,
> image-2018-12-03-09-54-07-859.png
>
>
> I found in all versions of yarn, Speculative Execution may set the
> speculative task to the node of original task.What i have read is only it
> will try to have one more task attempt. haven't seen any place mentioning not
> on same node.It is unreasonable.If the node have some problems lead to tasks
> execution will be very slow. and then placement the speculative task to same
> node cannot help the problematic task.
> In our cluster （version 2.7.2，2700 nodes），this phenomenon appear
> almost everyday.
> !image-2018-12-03-09-54-07-859.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[
https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095491#comment-17095491
]
Ahmed Hussein commented on MAPREDUCE-7169:
--
Thanks [~BilwaST] !
{quote}With change like this, there are chances that task attempt will get
launched on same node where it was launched which wouldn't solve the problem.
If we blacklist node then in next iteration of containers assigned it will be
launched on other node.
{quote}
The same node will be picked if there are no other available nodes. In
[^MAPREDUCE-7169.005.patch] , what is the expected behavior if the resources
available to run the taskAttempt are only available on the same node? I do not
see this case in the unit test.
> Speculative attempts should not run on the same node
>
>
> Key: MAPREDUCE-7169
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: yarn
>Affects Versions: 2.7.2
>Reporter: Lee chen
>Assignee: Bilwa S T
>Priority: Major
> Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch,
> MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch,
> image-2018-12-03-09-54-07-859.png
>
>
> I found in all versions of yarn, Speculative Execution may set the
> speculative task to the node of original task.What i have read is only it
> will try to have one more task attempt. haven't seen any place mentioning not
> on same node.It is unreasonable.If the node have some problems lead to tasks
> execution will be very slow. and then placement the speculative task to same
> node cannot help the problematic task.
> In our cluster （version 2.7.2，2700 nodes），this phenomenon appear
> almost everyday.
> !image-2018-12-03-09-54-07-859.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7261) Memory efficiency in speculator
[
https://issues.apache.org/jira/browse/MAPREDUCE-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7261:
-
Attachment: MAPREDUCE-7261.003.patch
> Memory efficiency in speculator
>
>
> Key: MAPREDUCE-7261
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7261
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.001.patch, MAPREDUCE-7261.002.patch,
> MAPREDUCE-7261.003.patch
>
>
> The data structures in speculator and runtime-estimator are bloating. Data
> elements such as (taskID, TA-ID, task stats, tasks speculated, tasks
> finished..etc) are added to the concurrent maps but never removed.
> For long running jobs, there are couple of issues:
> # memory leakage: the speculator memory usage increases over time.
> # performance: keeping large structures in the heap affects the performance
> due to locality and cache misses.
> *Suggested Fixes:*
> - When a TA transitions to {{MoveContainerToSucceededFinishingTransition}},
> the TA notifies the speculator. The latter handles the event by cleaning the
> internal structure accordingly.
> - When a task transitions is failed/killed, the speculator is notified to
> clean the internal data structure.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7261) Memory efficiency in speculator
[
https://issues.apache.org/jira/browse/MAPREDUCE-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7261:
-
Attachment: HDFS-10499-branch-2.10.003.patch
> Memory efficiency in speculator
>
>
> Key: MAPREDUCE-7261
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7261
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.001.patch, MAPREDUCE-7261.002.patch
>
>
> The data structures in speculator and runtime-estimator are bloating. Data
> elements such as (taskID, TA-ID, task stats, tasks speculated, tasks
> finished..etc) are added to the concurrent maps but never removed.
> For long running jobs, there are couple of issues:
> # memory leakage: the speculator memory usage increases over time.
> # performance: keeping large structures in the heap affects the performance
> due to locality and cache misses.
> *Suggested Fixes:*
> - When a TA transitions to {{MoveContainerToSucceededFinishingTransition}},
> the TA notifies the speculator. The latter handles the event by cleaning the
> internal structure accordingly.
> - When a task transitions is failed/killed, the speculator is notified to
> clean the internal data structure.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7261) Memory efficiency in speculator
[
https://issues.apache.org/jira/browse/MAPREDUCE-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7261:
-
Attachment: (was: HDFS-10499-branch-2.10.003.patch)
> Memory efficiency in speculator
>
>
> Key: MAPREDUCE-7261
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7261
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Ahmed Hussein
>Assignee: Ahmed Hussein
>Priority: Minor
> Attachments: MAPREDUCE-7261.001.patch, MAPREDUCE-7261.002.patch
>
>
> The data structures in speculator and runtime-estimator are bloating. Data
> elements such as (taskID, TA-ID, task stats, tasks speculated, tasks
> finished..etc) are added to the concurrent maps but never removed.
> For long running jobs, there are couple of issues:
> # memory leakage: the speculator memory usage increases over time.
> # performance: keeping large structures in the heap affects the performance
> due to locality and cache misses.
> *Suggested Fixes:*
> - When a TA transitions to {{MoveContainerToSucceededFinishingTransition}},
> the TA notifies the speculator. The latter handles the event by cleaning the
> internal structure accordingly.
> - When a task transitions is failed/killed, the speculator is notified to
> clean the internal data structure.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7274) Enable to limit running map and reduce tasks when job is running
[
https://issues.apache.org/jira/browse/MAPREDUCE-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095359#comment-17095359
]
Hadoop QA commented on MAPREDUCE-7274:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 8m
18s{color} | {color:red} Docker failed to build yetus/hadoop:e213563c3cb.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | MAPREDUCE-7274 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13001579/MAPREDUCE-7274-branch-3.2.0-v2.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7775/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Enable to limit running map and reduce tasks when job is running
>
>
> Key: MAPREDUCE-7274
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7274
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mr-am, mrv2
>Affects Versions: 3.2.0
>Reporter: Chengwei Wang
>Priority: Major
> Attachments: MAPREDUCE-7274-branch-3.2.0-v2.patch,
> MAPREDUCE-7274-branch-3.2.0.patch
>
>
> MRv2 enabled users to control the number of map or reduce tasks running
> simultaneously by configuration *_mapreduce.job.running.map.limit_* or
> _*mapreduce.job.running.reduce.limit*._ But users can only set limit number
> before submitting the job to rm. So, it's meaningful to enable users to set
> the limit of running map or reduce tasks when job is running, which can help
> users to restrict resource usage of job and give resources to high-priority
> job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7274) Enable to limit running map and reduce tasks when job is running
[ https://issues.apache.org/jira/browse/MAPREDUCE-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095254#comment-17095254 ] Chengwei Wang edited comment on MAPREDUCE-7274 at 4/29/20, 11:10 AM: - Upload patch v2. Add unit test, modify some improper naming and fix the patch does not apply to branch-3.2.0. was (Author: smarthan): Upload patch v2. Add unit test and modify some improper naming. > Enable to limit running map and reduce tasks when job is running > > > Key: MAPREDUCE-7274 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7274 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mr-am, mrv2 >Affects Versions: 3.2.0 >Reporter: Chengwei Wang >Priority: Major > Attachments: MAPREDUCE-7274-branch-3.2.0-v2.patch, > MAPREDUCE-7274-branch-3.2.0.patch > > > MRv2 enabled users to control the number of map or reduce tasks running > simultaneously by configuration *_mapreduce.job.running.map.limit_* or > _*mapreduce.job.running.reduce.limit*._ But users can only set limit number > before submitting the job to rm. So, it's meaningful to enable users to set > the limit of running map or reduce tasks when job is running, which can help > users to restrict resource usage of job and give resources to high-priority > job. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7274) Enable to limit running map and reduce tasks when job is running
[ https://issues.apache.org/jira/browse/MAPREDUCE-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengwei Wang updated MAPREDUCE-7274: - Attachment: MAPREDUCE-7274-branch-3.2.0-v2.patch > Enable to limit running map and reduce tasks when job is running > > > Key: MAPREDUCE-7274 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7274 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mr-am, mrv2 >Affects Versions: 3.2.0 >Reporter: Chengwei Wang >Priority: Major > Attachments: MAPREDUCE-7274-branch-3.2.0-v2.patch, > MAPREDUCE-7274-branch-3.2.0.patch > > > MRv2 enabled users to control the number of map or reduce tasks running > simultaneously by configuration *_mapreduce.job.running.map.limit_* or > _*mapreduce.job.running.reduce.limit*._ But users can only set limit number > before submitting the job to rm. So, it's meaningful to enable users to set > the limit of running map or reduce tasks when job is running, which can help > users to restrict resource usage of job and give resources to high-priority > job. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7274) Enable to limit running map and reduce tasks when job is running
[ https://issues.apache.org/jira/browse/MAPREDUCE-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengwei Wang updated MAPREDUCE-7274: - Attachment: (was: MAPREDUCE-7274-branch-3.2.0-v2.patch) > Enable to limit running map and reduce tasks when job is running > > > Key: MAPREDUCE-7274 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7274 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mr-am, mrv2 >Affects Versions: 3.2.0 >Reporter: Chengwei Wang >Priority: Major > Attachments: MAPREDUCE-7274-branch-3.2.0.patch > > > MRv2 enabled users to control the number of map or reduce tasks running > simultaneously by configuration *_mapreduce.job.running.map.limit_* or > _*mapreduce.job.running.reduce.limit*._ But users can only set limit number > before submitting the job to rm. So, it's meaningful to enable users to set > the limit of running map or reduce tasks when job is running, which can help > users to restrict resource usage of job and give resources to high-priority > job. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7274) Enable to limit running map and reduce tasks when job is running
[
https://issues.apache.org/jira/browse/MAPREDUCE-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095258#comment-17095258
]
Hadoop QA commented on MAPREDUCE-7274:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 13s{color}
| {color:red} MAPREDUCE-7274 does not apply to branch-3.2.0. Rebase required?
Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | MAPREDUCE-7274 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13001559/MAPREDUCE-7274-branch-3.2.0-v2.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7774/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.12.0 https://yetus.apache.org |
This message was automatically generated.
> Enable to limit running map and reduce tasks when job is running
>
>
> Key: MAPREDUCE-7274
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7274
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mr-am, mrv2
>Affects Versions: 3.2.0
>Reporter: Chengwei Wang
>Priority: Major
> Attachments: MAPREDUCE-7274-branch-3.2.0-v2.patch,
> MAPREDUCE-7274-branch-3.2.0.patch
>
>
> MRv2 enabled users to control the number of map or reduce tasks running
> simultaneously by configuration *_mapreduce.job.running.map.limit_* or
> _*mapreduce.job.running.reduce.limit*._ But users can only set limit number
> before submitting the job to rm. So, it's meaningful to enable users to set
> the limit of running map or reduce tasks when job is running, which can help
> users to restrict resource usage of job and give resources to high-priority
> job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7274) Enable to limit running map and reduce tasks when job is running
[ https://issues.apache.org/jira/browse/MAPREDUCE-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17095254#comment-17095254 ] Chengwei Wang commented on MAPREDUCE-7274: -- Upload patch v2. Add unit test and modify some improper naming. > Enable to limit running map and reduce tasks when job is running > > > Key: MAPREDUCE-7274 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7274 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mr-am, mrv2 >Affects Versions: 3.2.0 >Reporter: Chengwei Wang >Priority: Major > Attachments: MAPREDUCE-7274-branch-3.2.0-v2.patch, > MAPREDUCE-7274-branch-3.2.0.patch > > > MRv2 enabled users to control the number of map or reduce tasks running > simultaneously by configuration *_mapreduce.job.running.map.limit_* or > _*mapreduce.job.running.reduce.limit*._ But users can only set limit number > before submitting the job to rm. So, it's meaningful to enable users to set > the limit of running map or reduce tasks when job is running, which can help > users to restrict resource usage of job and give resources to high-priority > job. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7274) Enable to limit running map and reduce tasks when job is running
[ https://issues.apache.org/jira/browse/MAPREDUCE-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chengwei Wang updated MAPREDUCE-7274: - Attachment: MAPREDUCE-7274-branch-3.2.0-v2.patch > Enable to limit running map and reduce tasks when job is running > > > Key: MAPREDUCE-7274 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7274 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mr-am, mrv2 >Affects Versions: 3.2.0 >Reporter: Chengwei Wang >Priority: Major > Attachments: MAPREDUCE-7274-branch-3.2.0-v2.patch, > MAPREDUCE-7274-branch-3.2.0.patch > > > MRv2 enabled users to control the number of map or reduce tasks running > simultaneously by configuration *_mapreduce.job.running.map.limit_* or > _*mapreduce.job.running.reduce.limit*._ But users can only set limit number > before submitting the job to rm. So, it's meaningful to enable users to set > the limit of running map or reduce tasks when job is running, which can help > users to restrict resource usage of job and give resources to high-priority > job. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
