[jira] [Created] (MAPREDUCE-6963) MR Map or Reduce specified node label is missing due to concurrent task limits
wei created MAPREDUCE-6963:
--
Summary: MR Map or Reduce specified node label is missing due to
concurrent task limits
Key: MAPREDUCE-6963
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6963
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Affects Versions: 2.8.1
Reporter: wei
In RMContainerAllocator#applyConcurrentTaskLimits, we limit the degree of task
parallelism but not consider the node label specified for the ResourceRequest.
{code:java}
private void applyConcurrentTaskLimits() {
...
setRequestLimit(PRIORITY_FAST_FAIL_MAP, mapResourceRequest,
failedMapRequestLimit);
setRequestLimit(PRIORITY_MAP, mapResourceRequest, normalMapRequestLimit);
}
...
setRequestLimit(PRIORITY_REDUCE, reduceResourceRequest,
reduceRequestLimit);
}
}
{code}
Then we call the applyRequestLimits in RMContainerAllocator#makeRemoteRequest
to apply the request limits. When the req.getNumContainers() > limit conditions
are met，the original ResourceRequest(ask) will be replaced by the reqLimit
which was generated by the above.
{code:java}
private void applyRequestLimits() {
.
// update an existing ask or send a new one if updating
if (ask.remove(req) || requestLimitsToUpdate.contains(req)) {
ResourceRequest newReq = req.getNumContainers() > limit
? reqLimit : req;
ask.add(newReq);
LOG.info("Applying ask limit of " + newReq.getNumContainers()
+ " for priority:" + reqLimit.getPriority()
+ " and capability:" + reqLimit.getCapability());
}
...
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Reopened] (MAPREDUCE-6947) Moving logging APIs over to slf4j in hadoop-mapreduce-examples
[ https://issues.apache.org/jira/browse/MAPREDUCE-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira Ajisaka reopened MAPREDUCE-6947: -- Reverted. > Moving logging APIs over to slf4j in hadoop-mapreduce-examples > --- > > Key: MAPREDUCE-6947 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6947 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Gergely Novák >Assignee: Gergely Novák > Attachments: MAPREDUCE-6947.001.patch, MAPREDUCE-6947.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6962) Remove unused settings from log4j.properties
[
https://issues.apache.org/jira/browse/MAPREDUCE-6962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira Ajisaka resolved MAPREDUCE-6962.
--
Resolution: Fixed
Dup of HADOOP-7308, closing.
> Remove unused settings from log4j.properties
>
>
> Key: MAPREDUCE-6962
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6962
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Akira Ajisaka
>Priority: Minor
> Labels: newbie
>
> {noformat:title=hadoop-common-project/hadoop-common/src/main/conf/log4j.properties}
> #Default values
> hadoop.tasklog.taskid=null
> hadoop.tasklog.iscleanup=false
> hadoop.tasklog.noKeepSplits=4
> hadoop.tasklog.totalLogFileSize=100
> hadoop.tasklog.purgeLogSplits=true
> hadoop.tasklog.logsRetainHours=12
> {noformat}
> hadoop.tasklog.noKeepSplits, purgeLogSplits, and logsRetainHours are not used
> anywhere and can be removed. These parameters were removed by HADOOP-1553 10
> years ago.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/528/ [Sep 19, 2017 9:45:41 AM] (kai.zheng) HDFS-12479. Some misuses of lock in DFSStripedOutputStream. Contributed [Sep 19, 2017 10:53:11 AM] (stevel) Revert "HADOOP-14771. hadoop-client does not include hadoop-yarn-client. [Sep 19, 2017 1:45:05 PM] (jlowe) Revert "MAPREDUCE-6958. Shuffle audit logger should log size of shuffle [Sep 19, 2017 2:13:17 PM] (jlowe) MAPREDUCE-6958. Shuffle audit logger should log size of shuffle [Sep 19, 2017 3:35:15 PM] (epayne) MAPREDUCE-6960. Shuffle Handler prints disk error stack traces for every [Sep 19, 2017 5:07:07 AM] (brahma) HDFS-12480. TestNameNodeMetrics#testTransactionAndCheckpointMetrics [Sep 19, 2017 5:55:45 AM] (brahma) HDFS-11799. Introduce a config to allow setting up write pipeline with [Sep 19, 2017 6:50:01 PM] (lei) HDFS-12449. TestReconstructStripedFile.testNNSendsErasureCodingTasks [Sep 19, 2017 8:44:42 PM] (wang) HDFS-12444. Reduce runtime of TestWriteReadStripedFile. Contributed by [Sep 19, 2017 8:48:23 PM] (wang) HDFS-12445. Correct spellings of choosen to chosen. Contributed by hu [Sep 19, 2017 11:42:20 PM] (wang) HDFS-12437. Fix test setup in TestLeaseRecoveryStriped. [Sep 20, 2017 1:31:15 AM] (junping_du) YARN-7186. Fix finicky TestContainerManager tests. Contributed by Arun [Sep 20, 2017 1:28:31 AM] (yufei) YARN-6499. Remove the doc about Schedulable#redistributeShare(). [Error replacing 'FILE' - Workspace is not accessible] - To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6962) Remove unused settings from log4j.properties
Akira Ajisaka created MAPREDUCE-6962:
Summary: Remove unused settings from log4j.properties
Key: MAPREDUCE-6962
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6962
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Akira Ajisaka
Priority: Minor
{noformat:title=hadoop-common-project/hadoop-common/src/main/conf/log4j.properties}
#Default values
hadoop.tasklog.taskid=null
hadoop.tasklog.iscleanup=false
hadoop.tasklog.noKeepSplits=4
hadoop.tasklog.totalLogFileSize=100
hadoop.tasklog.purgeLogSplits=true
hadoop.tasklog.logsRetainHours=12
{noformat}
hadoop.tasklog.noKeepSplits, purgeLogSplits, and logsRetainHours is not used
anywhere and can be removed. These parameters were removed by HADOOP-1553 10
years ago.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
