[jira] [Commented] (MAPREDUCE-4815) Speed up FileOutputCommitter#commitJob for many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14356967#comment-14356967 ] Hudson commented on MAPREDUCE-4815: --- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #120 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/120/]) MAPREDUCE-4815. Speed up FileOutputCommitter#commitJob for many output files. (Siqi Li via gera) (gera: rev aa92b764a7ddb888d097121c4d610089a0053d11) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java Speed up FileOutputCommitter#commitJob for many output files Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Labels: perfomance Fix For: 2.7.0 Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v11.patch, MAPREDUCE-4815.v12.patch, MAPREDUCE-4815.v13.patch, MAPREDUCE-4815.v14.patch, MAPREDUCE-4815.v15.patch, MAPREDUCE-4815.v16.patch, MAPREDUCE-4815.v17.patch, MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch, MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4815) Speed up FileOutputCommitter#commitJob for many output files
[ https://issues.apache.org/jira/browse/MAPREDUCE-4815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14355444#comment-14355444 ] Hudson commented on MAPREDUCE-4815: --- FAILURE: Integrated in Hadoop-trunk-Commit #7299 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7299/]) MAPREDUCE-4815. Speed up FileOutputCommitter#commitJob for many output files. (Siqi Li via gera) (gera: rev aa92b764a7ddb888d097121c4d610089a0053d11) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestFileOutputCommitter.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestFileOutputCommitter.java Speed up FileOutputCommitter#commitJob for many output files Key: MAPREDUCE-4815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4815 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.3, 2.0.1-alpha, 2.4.1 Reporter: Jason Lowe Assignee: Siqi Li Labels: perfomance Attachments: MAPREDUCE-4815.v10.patch, MAPREDUCE-4815.v11.patch, MAPREDUCE-4815.v12.patch, MAPREDUCE-4815.v13.patch, MAPREDUCE-4815.v14.patch, MAPREDUCE-4815.v15.patch, MAPREDUCE-4815.v16.patch, MAPREDUCE-4815.v17.patch, MAPREDUCE-4815.v3.patch, MAPREDUCE-4815.v4.patch, MAPREDUCE-4815.v5.patch, MAPREDUCE-4815.v6.patch, MAPREDUCE-4815.v7.patch, MAPREDUCE-4815.v8.patch, MAPREDUCE-4815.v9.patch If a job generates many files to commit then the commitJob method call at the end of the job can take minutes. This is a performance regression from 1.x, as 1.x had the tasks commit directly to the final output directory as they were completing and commitJob had very little to do. The commit work was processed in parallel and overlapped the processing of outstanding tasks. In 0.23/2.x, the commit is single-threaded and waits until all tasks have completed before commencing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359867#comment-14359867
]
zhihai xu commented on MAPREDUCE-6265:
--
I uploaded a patch MAPREDUCE-6265.000.patch for review.
The initial thread pool size is a important parameter for ContainerLauncherImpl.
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359899#comment-14359899
]
Hadoop QA commented on MAPREDUCE-6265:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704343/MAPREDUCE-6265.000.patch
against trunk revision 8212877.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5285//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5285//console
This message is automatically generated.
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated MAPREDUCE-6265:
-
Attachment: MAPREDUCE-6265.000.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359929#comment-14359929
]
zhihai xu commented on MAPREDUCE-6265:
--
Hi [~ozawa], The patch passed all the tests. Do you have time to review it?
thanks
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated MAPREDUCE-6265:
-
Status: Patch Available (was: Open)
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359952#comment-14359952
]
Tsuyoshi Ozawa commented on MAPREDUCE-6265:
---
Sure, I'll check it.
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5653) DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359063#comment-14359063 ] Allen Wittenauer commented on MAPREDUCE-5653: - bq. Allen, do you think there's more than just this one Xmx passthrough that's affecting DistCP? Yes. but those are outside the scope of this jira. bq. Hadn't seen MAPREDUCE-5785. Agree that that's an excellent direction. As I mentioned way above, for the same reasons that '5785 is in trunk, this change is also only in trunk. Hypothetically, they should work very well together. DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb Key: MAPREDUCE-5653 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5653 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 0.23.9, 2.2.0 Reporter: Mithun Radhakrishnan Assignee: Ratandeep Ratti Fix For: 3.0.0 Attachments: MAPREDUCE-5653.branch-0.23.patch, MAPREDUCE-5653.branch-2.patch, MAPREDUCE-5653.trunk.2.patch, MAPREDUCE-5653.trunk.patch When a DistCp job is run through Oozie (through a Java action that launches DistCp), one sees that mapred.child.java.opts as set from the caller is honoured by DistCp. But, DistCp doesn't seem to honour any overrides for configs mapreduce.[map,reduce].memory.mb. Problem has been identified. I'll post a patch shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5653) DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb
[
https://issues.apache.org/jira/browse/MAPREDUCE-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358974#comment-14358974
]
Philip Zeyliger commented on MAPREDUCE-5653:
You could make an argument that DistCp, as a Yarn application, knows better
than the defaults about how much memory it uses. I.e., that the bug is that
DistCp isn't setting both intimately related settings
({{mapred.job.{map|reduce}.memory.mb}} and {{mapreduce.map.java.opts}}, but
rather than just one. If the defaults in your cluster were to use a lot of
memory, and DistCP uses very little (after all, it's copying a buffer around),
it's wasteful.
DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb
Key: MAPREDUCE-5653
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5653
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp
Affects Versions: 0.23.9, 2.2.0
Reporter: Mithun Radhakrishnan
Assignee: Ratandeep Ratti
Fix For: 3.0.0
Attachments: MAPREDUCE-5653.branch-0.23.patch,
MAPREDUCE-5653.branch-2.patch, MAPREDUCE-5653.trunk.2.patch,
MAPREDUCE-5653.trunk.patch
When a DistCp job is run through Oozie (through a Java action that launches
DistCp), one sees that mapred.child.java.opts as set from the caller is
honoured by DistCp. But, DistCp doesn't seem to honour any overrides for
configs mapreduce.[map,reduce].memory.mb.
Problem has been identified. I'll post a patch shortly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5653) DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14358994#comment-14358994 ] Allen Wittenauer commented on MAPREDUCE-5653: - bq. You could make an argument that DistCp, as a Yarn application, knows better than the defaults about how much memory it uses. One could, but one pass through the code makes that argument kind of moot given there is zero logic currently for distcp to have that level of intelligence. bq. If the defaults in your cluster were to use a lot of memory, and DistCP uses very little (after all, it's copying a buffer around), it's wasteful. Part of the argument around MAPREDUCE-5785 revolves around the fact that one doesn't actually want to set defaults anymore for MapReduce applications. DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb Key: MAPREDUCE-5653 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5653 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 0.23.9, 2.2.0 Reporter: Mithun Radhakrishnan Assignee: Ratandeep Ratti Fix For: 3.0.0 Attachments: MAPREDUCE-5653.branch-0.23.patch, MAPREDUCE-5653.branch-2.patch, MAPREDUCE-5653.trunk.2.patch, MAPREDUCE-5653.trunk.patch When a DistCp job is run through Oozie (through a Java action that launches DistCp), one sees that mapred.child.java.opts as set from the caller is honoured by DistCp. But, DistCp doesn't seem to honour any overrides for configs mapreduce.[map,reduce].memory.mb. Problem has been identified. I'll post a patch shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5653) DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359045#comment-14359045 ] Philip Zeyliger commented on MAPREDUCE-5653: Allen, do you think there's more than just this one Xmx passthrough that's affecting DistCP? There's not much smarts it needs: it's not like it's every doing anything besides copying files. Hadn't seen MAPREDUCE-5785. Agree that that's an excellent direction. DistCp does not honour config-overrides for mapreduce.[map,reduce].memory.mb Key: MAPREDUCE-5653 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5653 Project: Hadoop Map/Reduce Issue Type: Bug Components: distcp Affects Versions: 0.23.9, 2.2.0 Reporter: Mithun Radhakrishnan Assignee: Ratandeep Ratti Fix For: 3.0.0 Attachments: MAPREDUCE-5653.branch-0.23.patch, MAPREDUCE-5653.branch-2.patch, MAPREDUCE-5653.trunk.2.patch, MAPREDUCE-5653.trunk.patch When a DistCp job is run through Oozie (through a Java action that launches DistCp), one sees that mapred.child.java.opts as set from the caller is honoured by DistCp. But, DistCp doesn't seem to honour any overrides for configs mapreduce.[map,reduce].memory.mb. Problem has been identified. I'll post a patch shortly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6266) Job#getTrackingURL should consistently return a proper URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-6266: -- Status: Patch Available (was: Open) Job#getTrackingURL should consistently return a proper URL -- Key: MAPREDUCE-6266 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6266 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: MAPREDUCE-6266.001.patch When a job is running, Job#getTrackingURL returns a proper URL like: http://RM_IP:8088/proxy/application_1424910897258_0004/ Once a job is finished and the job has moved to the JHS, then Job#getTrackingURL returns a URL without the protocol like: JHS_IP:19888/jobhistory/job/job_1424910897258_0004 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6266) Job#getTrackingURL should consistently return a proper URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-6266: -- Attachment: MAPREDUCE-6266.001.patch - Call MRWebAppUtil#getApplicationWebURLOnJHSWithScheme() instead of MRWebAppUtil#getApplicationWebURLOnJHSWithoutScheme() - Retry method call 3 times in order to lessen chance of returning an empty string Job#getTrackingURL should consistently return a proper URL -- Key: MAPREDUCE-6266 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6266 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: MAPREDUCE-6266.001.patch When a job is running, Job#getTrackingURL returns a proper URL like: http://RM_IP:8088/proxy/application_1424910897258_0004/ Once a job is finished and the job has moved to the JHS, then Job#getTrackingURL returns a URL without the protocol like: JHS_IP:19888/jobhistory/job/job_1424910897258_0004 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6266) Job#getTrackingURL should consistently return a proper URL
[
https://issues.apache.org/jira/browse/MAPREDUCE-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14359649#comment-14359649
]
Hadoop QA commented on MAPREDUCE-6266:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704287/MAPREDUCE-6266.001.patch
against trunk revision 8212877.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5284//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5284//console
This message is automatically generated.
Job#getTrackingURL should consistently return a proper URL
--
Key: MAPREDUCE-6266
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6266
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Ray Chiang
Assignee: Ray Chiang
Priority: Minor
Labels: supportability
Attachments: MAPREDUCE-6266.001.patch
When a job is running, Job#getTrackingURL returns a proper URL like:
http://RM_IP:8088/proxy/application_1424910897258_0004/
Once a job is finished and the job has moved to the JHS, then
Job#getTrackingURL returns a URL without the protocol like:
JHS_IP:19888/jobhistory/job/job_1424910897258_0004
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4742) Fix typo in nnbench#displayUsage
[ https://issues.apache.org/jira/browse/MAPREDUCE-4742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-4742: -- Resolution: Fixed Fix Version/s: 2.8.0 Target Version/s: 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed this to trunk and branch-2. Thanks [~xieliang007] for your contribution. Fix typo in nnbench#displayUsage Key: MAPREDUCE-4742 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4742 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.2-alpha, 0.23.4, 2.6.0 Reporter: Liang Xie Priority: Trivial Fix For: 2.8.0 Attachments: MAPREDUCE-4742.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-4742) Tiny nnbench displayUsage
[ https://issues.apache.org/jira/browse/MAPREDUCE-4742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-4742: -- Affects Version/s: 2.6.0 Tiny nnbench displayUsage - Key: MAPREDUCE-4742 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4742 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.2-alpha, 0.23.4, 2.6.0 Reporter: Liang Xie Priority: Trivial Attachments: MAPREDUCE-4742.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
