[jira] [Commented] (MAPREDUCE-7100) Provide options to skip adding resource request for data-local and rack-local respectively
[
https://issues.apache.org/jira/browse/MAPREDUCE-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499689#comment-16499689
]
Xiang Li commented on MAPREDUCE-7100:
-
The allocation of containing is more quick if I disabled adding requests for
rack-local. But the MR job summary shows:
{code}
Rack-local map tasks=xxx
{code}
which is quite questionable to me, because I did not request rack-local
containers, why are there rack-local map tasks.
> Provide options to skip adding resource request for data-local and rack-local
> respectively
> --
>
> Key: MAPREDUCE-7100
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7100
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster
>Reporter: Xiang Li
>Priority: Minor
>
> We are using hadoop 2.7.3 and the computing layer is running out of the
> storage cluster (that is, node managers are running on a different set of
> nodes from data nodes). The problem we meet is that the container allocation
> is quite slow for some jobs.
> After some debugging, we found that in
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor#addContainerReq()
> (the following code is from trunk, not 2.7.3)
> {code}
> protected void addContainerReq(ContainerRequest req) {
> // Create resource requests
> for (String host : req.hosts) {
> // Data-local
> if (!isNodeBlacklisted(host)) {
> addResourceRequest(req.priority, host, req.capability,
> null);
> }
> }
> // Nothing Rack-local for now
> for (String rack : req.racks) {
> addResourceRequest(req.priority, rack, req.capability,
> null);
> }
> // Off-switch
> addResourceRequest(req.priority, ResourceRequest.ANY, req.capability,
> req.nodeLabelExpression);
> }
> {code}
> It seem that the request of data-local and rack-local could be skipped when
> computing layer is not the same as the storage cluster.
> If I get it correctly, req.hosts and req.racks are provided by InputFormat.
> If the mapper is to read HDFS, req.hosts is the corresponding data node and
> req.racks is its rack. The debug log of AM is like:
> {code}
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> addResourceRequest: applicationId=1 priority=20 resourceName=
> numContainers=1 #asks=1
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> addResourceRequest: applicationId=1 priority=20 resourceName=
> numContainers=1 #asks=2
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> addResourceRequest: applicationId=1 priority=20 resourceName=*
> numContainers=1 #asks=3
> {code}
> Although eventually, the resource request with resourceName= will
> not be satisfied (because the data node is not node manager) in RM, it could
> be better if AM does not request data-local or rack-local at the very
> beginning, when we already know that computer layer runs out of the storage
> cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7100) Provide options to skip adding resource request for data-local and rack-local respectively
[
https://issues.apache.org/jira/browse/MAPREDUCE-7100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499689#comment-16499689
]
Xiang Li edited comment on MAPREDUCE-7100 at 6/4/18 2:54 AM:
-
The allocation of containing is more quick if I disabled adding requests for
rack-local. But the MR job summary shows:
{code}
Rack-local map tasks=xxx
{code}
which is quite questionable to me, because I did not request rack-local
containers, why are there rack-local map tasks?
was (Author: water):
The allocation of containing is more quick if I disabled adding requests for
rack-local. But the MR job summary shows:
{code}
Rack-local map tasks=xxx
{code}
which is quite questionable to me, because I did not request rack-local
containers, why are there rack-local map tasks.
> Provide options to skip adding resource request for data-local and rack-local
> respectively
> --
>
> Key: MAPREDUCE-7100
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7100
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster
>Reporter: Xiang Li
>Priority: Minor
>
> We are using hadoop 2.7.3 and the computing layer is running out of the
> storage cluster (that is, node managers are running on a different set of
> nodes from data nodes). The problem we meet is that the container allocation
> is quite slow for some jobs.
> After some debugging, we found that in
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor#addContainerReq()
> (the following code is from trunk, not 2.7.3)
> {code}
> protected void addContainerReq(ContainerRequest req) {
> // Create resource requests
> for (String host : req.hosts) {
> // Data-local
> if (!isNodeBlacklisted(host)) {
> addResourceRequest(req.priority, host, req.capability,
> null);
> }
> }
> // Nothing Rack-local for now
> for (String rack : req.racks) {
> addResourceRequest(req.priority, rack, req.capability,
> null);
> }
> // Off-switch
> addResourceRequest(req.priority, ResourceRequest.ANY, req.capability,
> req.nodeLabelExpression);
> }
> {code}
> It seem that the request of data-local and rack-local could be skipped when
> computing layer is not the same as the storage cluster.
> If I get it correctly, req.hosts and req.racks are provided by InputFormat.
> If the mapper is to read HDFS, req.hosts is the corresponding data node and
> req.racks is its rack. The debug log of AM is like:
> {code}
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> addResourceRequest: applicationId=1 priority=20 resourceName=
> numContainers=1 #asks=1
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> addResourceRequest: applicationId=1 priority=20 resourceName=
> numContainers=1 #asks=2
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
> addResourceRequest: applicationId=1 priority=20 resourceName=*
> numContainers=1 #asks=3
> {code}
> Although eventually, the resource request with resourceName= will
> not be satisfied (because the data node is not node manager) in RM, it could
> be better if AM does not request data-local or rack-local at the very
> beginning, when we already know that computer layer runs out of the storage
> cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7099) Daily test result fails in MapReduce JobClient though there isn't any error
[
https://issues.apache.org/jira/browse/MAPREDUCE-7099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16499623#comment-16499623
]
Takanobu Asanuma commented on MAPREDUCE-7099:
-
Thanks for reporting it, [~miklos.szeg...@cloudera.com]. I haven't found the
cause yet. I will further investigate the tests you reported.
> Daily test result fails in MapReduce JobClient though there isn't any error
> ---
>
> Key: MAPREDUCE-7099
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7099
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build, test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Critical
>
> Looks like the test result in MapReduce JobClient always fails lately. Please
> see the results of hadoop-qbt-trunk-java8-linux-x86:
>
> [https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/]/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
> {noformat}
> [INFO] Results:
> [INFO]
> [WARNING] Tests run: 565, Failures: 0, Errors: 0, Skipped: 10
> [INFO]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 02:06 h
> [INFO] Finished at: 2018-05-30T12:32:39+00:00
> [INFO] Final Memory: 25M/645M
> [INFO]
>
> [WARNING] The requested profile "parallel-tests" could not be activated
> because it does not exist.
> [WARNING] The requested profile "shelltest" could not be activated because it
> does not exist.
> [WARNING] The requested profile "native" could not be activated because it
> does not exist.
> [WARNING] The requested profile "yarn-ui" could not be activated because it
> does not exist.
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on
> project hadoop-mapreduce-client-jobclient: There was a timeout or other error
> in the fork -> [Help 1]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7104) TestHistoryViewerPrinter fails on windows.
[ https://issues.apache.org/jira/browse/MAPREDUCE-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated MAPREDUCE-7104: - Resolution: Duplicate Status: Resolved (was: Patch Available) > TestHistoryViewerPrinter fails on windows. > -- > > Key: MAPREDUCE-7104 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7104 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Minor > Attachments: MAPREDUCE-7104.001.patch > > > All tests in > [TestHistoryViewerPrinter|https://builds.apache.org/job/hadoop-trunk-win/485/testReport/org.apache.hadoop.mapreduce.jobhistory/TestHistoryViewerPrinter/testHumanPrinter/] > are failing in Windows due to difference in line separator on Windows. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7104) TestHistoryViewerPrinter fails on windows.
[ https://issues.apache.org/jira/browse/MAPREDUCE-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated MAPREDUCE-7104: - Attachment: MAPREDUCE-7104.001.patch Status: Patch Available (was: Open) [~elgoiri] could you please review it. > TestHistoryViewerPrinter fails on windows. > -- > > Key: MAPREDUCE-7104 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7104 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Minor > Attachments: MAPREDUCE-7104.001.patch > > > All tests in > [TestHistoryViewerPrinter|https://builds.apache.org/job/hadoop-trunk-win/485/testReport/org.apache.hadoop.mapreduce.jobhistory/TestHistoryViewerPrinter/testHumanPrinter/] > are failing in Windows due to difference in line separator on Windows. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7104) TestHistoryViewerPrinter fails on windows.
[ https://issues.apache.org/jira/browse/MAPREDUCE-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated MAPREDUCE-7104: - Description: All tests in [TestHistoryViewerPrinter|https://builds.apache.org/job/hadoop-trunk-win/485/testReport/org.apache.hadoop.mapreduce.jobhistory/TestHistoryViewerPrinter/testHumanPrinter/] are failing in Windows due to difference in line separator on Windows. > TestHistoryViewerPrinter fails on windows. > -- > > Key: MAPREDUCE-7104 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7104 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Abhishek Modi >Assignee: Abhishek Modi >Priority: Minor > > All tests in > [TestHistoryViewerPrinter|https://builds.apache.org/job/hadoop-trunk-win/485/testReport/org.apache.hadoop.mapreduce.jobhistory/TestHistoryViewerPrinter/testHumanPrinter/] > are failing in Windows due to difference in line separator on Windows. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7104) TestHistoryViewerPrinter fails on windows.
Abhishek Modi created MAPREDUCE-7104: Summary: TestHistoryViewerPrinter fails on windows. Key: MAPREDUCE-7104 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7104 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Abhishek Modi Assignee: Abhishek Modi -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
