[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360873#comment-14360873
]
zhihai xu commented on MAPREDUCE-6265:
--
[~ozawa], yes, that is a very good suggestion, I uploaded a new patch
MAPREDUCE-6265.001.patch which addressed your comment. Please review it.
many thanks for the review.
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360963#comment-14360963
]
Hadoop QA commented on MAPREDUCE-6265:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704478/MAPREDUCE-6265.001.patch
against trunk revision f446669.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5288//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5288//console
This message is automatically generated.
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated MAPREDUCE-6265:
-
Attachment: MAPREDUCE-6265.001.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated MAPREDUCE-6265:
-
Attachment: MAPREDUCE-6265.002.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch,
MAPREDUCE-6265.002.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14361602#comment-14361602
]
Tsuyoshi Ozawa commented on MAPREDUCE-6265:
---
I meant containerlauncher.thread-initial-size should be
containerlauncher.*threadpool*-initial-size;
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14361601#comment-14361601
]
Tsuyoshi Ozawa commented on MAPREDUCE-6265:
---
[~zxu] should we also update the following line(name of property name itself)?
{code}
+ public static final String MR_AM_CONTAINERLAUNCHER_THREADPOOL_INITIAL_SIZE =
+ MR_AM_PREFIX+containerlauncher.thread-initial-size;
{code}
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14361609#comment-14361609
]
zhihai xu commented on MAPREDUCE-6265:
--
I uploaded a new patch MAPREDUCE-6265.002.patch to fix this issue.
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch,
MAPREDUCE-6265.002.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14361608#comment-14361608
]
zhihai xu commented on MAPREDUCE-6265:
--
[~ozawa], good catch, I forget to update the string value. thanks for your
thorough review.
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch,
MAPREDUCE-6265.002.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14361615#comment-14361615
]
Hadoop QA commented on MAPREDUCE-6265:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704570/MAPREDUCE-6265.002.patch
against trunk revision 32741cf.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5289//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5289//console
This message is automatically generated.
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch,
MAPREDUCE-6265.002.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6266) Job#getTrackingURL should consistently return a proper URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ray Chiang updated MAPREDUCE-6266: -- Attachment: MAPREDUCE-6266.002.patch Add minor unit test update Job#getTrackingURL should consistently return a proper URL -- Key: MAPREDUCE-6266 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6266 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: MAPREDUCE-6266.001.patch, MAPREDUCE-6266.002.patch When a job is running, Job#getTrackingURL returns a proper URL like: http://RM_IP:8088/proxy/application_1424910897258_0004/ Once a job is finished and the job has moved to the JHS, then Job#getTrackingURL returns a URL without the protocol like: JHS_IP:19888/jobhistory/job/job_1424910897258_0004 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6266) Job#getTrackingURL should consistently return a proper URL
[
https://issues.apache.org/jira/browse/MAPREDUCE-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360061#comment-14360061
]
Hadoop QA commented on MAPREDUCE-6266:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704372/MAPREDUCE-6266.002.patch
against trunk revision 387f271.
{color:red}-1 @author{color}. The patch appears to contain @author tags
which the Hadoop community has agreed to not allow in code contributions.
{color:green}+1 tests included{color}. The patch appears to include new
or modified test files.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5286//console
This message is automatically generated.
Job#getTrackingURL should consistently return a proper URL
--
Key: MAPREDUCE-6266
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6266
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Ray Chiang
Assignee: Ray Chiang
Priority: Minor
Labels: supportability
Attachments: MAPREDUCE-6266.001.patch, MAPREDUCE-6266.002.patch
When a job is running, Job#getTrackingURL returns a proper URL like:
http://RM_IP:8088/proxy/application_1424910897258_0004/
Once a job is finished and the job has moved to the JHS, then
Job#getTrackingURL returns a URL without the protocol like:
JHS_IP:19888/jobhistory/job/job_1424910897258_0004
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6266) Job#getTrackingURL should consistently return a proper URL
[
https://issues.apache.org/jira/browse/MAPREDUCE-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360096#comment-14360096
]
Hadoop QA commented on MAPREDUCE-6266:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12704372/MAPREDUCE-6266.002.patch
against trunk revision 387f271.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:
org.apache.hadoop.mapreduce.v2.hs.TestJobHistoryEntities
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5287//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5287//console
This message is automatically generated.
Job#getTrackingURL should consistently return a proper URL
--
Key: MAPREDUCE-6266
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6266
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Ray Chiang
Assignee: Ray Chiang
Priority: Minor
Labels: supportability
Attachments: MAPREDUCE-6266.001.patch, MAPREDUCE-6266.002.patch
When a job is running, Job#getTrackingURL returns a proper URL like:
http://RM_IP:8088/proxy/application_1424910897258_0004/
Once a job is finished and the job has moved to the JHS, then
Job#getTrackingURL returns a URL without the protocol like:
JHS_IP:19888/jobhistory/job/job_1424910897258_0004
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6266) Job#getTrackingURL should consistently return a proper URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360420#comment-14360420 ] Ray Chiang commented on MAPREDUCE-6266: --- The call to Job.getTrackingURL() occasionally fails quietly (due to the empty catch statement and probably network hiccups). This will put in a limited retry to make the occurrence less frequent. I can take out that part if it's undesirable. Job#getTrackingURL should consistently return a proper URL -- Key: MAPREDUCE-6266 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6266 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: MAPREDUCE-6266.001.patch, MAPREDUCE-6266.002.patch When a job is running, Job#getTrackingURL returns a proper URL like: http://RM_IP:8088/proxy/application_1424910897258_0004/ Once a job is finished and the job has moved to the JHS, then Job#getTrackingURL returns a URL without the protocol like: JHS_IP:19888/jobhistory/job/job_1424910897258_0004 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6266) Job#getTrackingURL should consistently return a proper URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360446#comment-14360446 ] Ray Chiang commented on MAPREDUCE-6266: --- RE: failed unit test Interesting, testing on my standalone machine gives me one set of results on the TestJobHistoryEntities unit test. Testing on Jenkins seems to give a valid URL. Job#getTrackingURL should consistently return a proper URL -- Key: MAPREDUCE-6266 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6266 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: MAPREDUCE-6266.001.patch, MAPREDUCE-6266.002.patch When a job is running, Job#getTrackingURL returns a proper URL like: http://RM_IP:8088/proxy/application_1424910897258_0004/ Once a job is finished and the job has moved to the JHS, then Job#getTrackingURL returns a URL without the protocol like: JHS_IP:19888/jobhistory/job/job_1424910897258_0004 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6266) Job#getTrackingURL should consistently return a proper URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360444#comment-14360444 ] Junping Du commented on MAPREDUCE-6266: --- Can we dig out the reason why Job.getTrackingURL() occasionally fails? It could be some bug hidden there. If so, it would be great if we can fix it also in this JIRA. Job#getTrackingURL should consistently return a proper URL -- Key: MAPREDUCE-6266 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6266 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: MAPREDUCE-6266.001.patch, MAPREDUCE-6266.002.patch When a job is running, Job#getTrackingURL returns a proper URL like: http://RM_IP:8088/proxy/application_1424910897258_0004/ Once a job is finished and the job has moved to the JHS, then Job#getTrackingURL returns a URL without the protocol like: JHS_IP:19888/jobhistory/job/job_1424910897258_0004 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6266) Job#getTrackingURL should consistently return a proper URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360447#comment-14360447 ] Ray Chiang commented on MAPREDUCE-6266: --- Okay, I'll take a look and see if I can replicate the issue. Job#getTrackingURL should consistently return a proper URL -- Key: MAPREDUCE-6266 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6266 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: MAPREDUCE-6266.001.patch, MAPREDUCE-6266.002.patch When a job is running, Job#getTrackingURL returns a proper URL like: http://RM_IP:8088/proxy/application_1424910897258_0004/ Once a job is finished and the job has moved to the JHS, then Job#getTrackingURL returns a URL without the protocol like: JHS_IP:19888/jobhistory/job/job_1424910897258_0004 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6266) Job#getTrackingURL should consistently return a proper URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-6266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360368#comment-14360368 ] Junping Du commented on MAPREDUCE-6266: --- Thanks [~rchiang] for reporting an issue and working on a patch here. One question: why we add a loop here in getting URL? Job#getTrackingURL should consistently return a proper URL -- Key: MAPREDUCE-6266 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6266 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: Ray Chiang Assignee: Ray Chiang Priority: Minor Labels: supportability Attachments: MAPREDUCE-6266.001.patch, MAPREDUCE-6266.002.patch When a job is running, Job#getTrackingURL returns a proper URL like: http://RM_IP:8088/proxy/application_1424910897258_0004/ Once a job is finished and the job has moved to the JHS, then Job#getTrackingURL returns a URL without the protocol like: JHS_IP:19888/jobhistory/job/job_1424910897258_0004 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6265) make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread pool size to launch/kill containers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14360641#comment-14360641
]
Tsuyoshi Ozawa commented on MAPREDUCE-6265:
---
[~zxu] looks good to me overall. Minor nits: I think the name of the property
should be yarn.app.mapreduce.am.containerlauncher.*threadpool*-initial-size and
MR_AM_CONTAINERLAUNCHER_*THREADPOOL*_INITIAL_SIZE for accuracy since it's a bit
difficult to understand what thread initial size means. What do you think?
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers.
--
Key: MAPREDUCE-6265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Reporter: zhihai xu
Assignee: zhihai xu
Attachments: MAPREDUCE-6265.000.patch
make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better
control the thread pool size to launch/kill containers
Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at
{code}
protected static final int INITIAL_POOL_SIZE = 10;
{code}
We should make it configurable because the thread pool size will be decided
by INITIAL_POOL_SIZE, limitOnPoolSize and number of node used by the AM.
Since we already made limitOnPoolSize configurable, it make senses to also
make INITIAL_POOL_SIZE configurable to better manage the thread pool size.
We saw some issue due to the small thread pool size when some node is down.
The recovery from a shutdown node take very long time due to all the
ContainerLauncher threads are blocked by IPC client connection to the
shutdown node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
