[
https://issues.apache.org/jira/browse/HDFS-13935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhoushulin updated HDFS-13935:
------------------------------
Description:
The timeout used in "setReadTimeout" in "Fetcher.java" and
"JobEndNotifier.java" in MapReduce is configurable as follows:
{code:title="/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java"}
class Fetcher<K,V> extends Thread {
...
this.readTimeout =
job.getInt(MRJobConfig.SHUFFLE_READ_TIMEOUT, DEFAULT_READ_TIMEOUT);
...
private void setupShuffleConnection(String encHash) {
// put url hash into http header
connection.addRequestProperty(
SecureShuffleUtils.HTTP_HEADER_URL_HASH, encHash);
// set the read timeout
connection.setReadTimeout(readTimeout);
// put shuffle version into http header
connection.addRequestProperty(ShuffleHeader.HTTP_HEADER_NAME,
ShuffleHeader.DEFAULT_HTTP_HEADER_NAME);
connection.addRequestProperty(ShuffleHeader.HTTP_HEADER_VERSION,
ShuffleHeader.DEFAULT_HTTP_HEADER_VERSION);
}
...
}
{code}
{code:title="/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java"}
public class JobEndNotifier implements Configurable {
...
timeout = conf.getInt(JobContext.MR_JOB_END_NOTIFICATION_TIMEOUT,
JobContext.DEFAULT_MR_JOB_END_NOTIFICATION_TIMEOUT);
...
protected boolean notifyURLOnce() {
...
conn.setReadTimeout(timeout);
...
}
...
}
{code}
The similar situations happen in "DFSZKFailoverController.java" in HDFS.
However, the "setReadTimeout()" in "FederationUtil.java",
"CredentialBasedAccessTokenProvider.java", and
"ConfRefreshTokenBasedAccessTokenProvider.java" call constants as their
argument. For example:
{code:title="/org/apache/hadoop/hdfs/server/federation/router/FederationUtil.java"}
public final class FederationUtil {
...
public static JSONArray getJmx(String beanQuery, String webAddress) {
...
URLConnection conn = jmxURL.openConnection();
conn.setConnectTimeout(5 * 1000);
conn.setReadTimeout(5 * 1000);
...
}
...
}
{code}
I think these constant should be also configurable for different requirements
and environments.
The similar situation happens on method "setConnectTimeout()", so it should be
also configurable.
was:
The timeout used in "setReadTimeout" in "Fetcher.java" and
"JobEndNotifier.java" in MapReduce is configurable as follows:
{code:title="/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java"}
class Fetcher<K,V> extends Thread {
...
this.readTimeout =
job.getInt(MRJobConfig.SHUFFLE_READ_TIMEOUT, DEFAULT_READ_TIMEOUT);
...
private void setupShuffleConnection(String encHash) {
// put url hash into http header
connection.addRequestProperty(
SecureShuffleUtils.HTTP_HEADER_URL_HASH, encHash);
// set the read timeout
connection.setReadTimeout(readTimeout);
// put shuffle version into http header
connection.addRequestProperty(ShuffleHeader.HTTP_HEADER_NAME,
ShuffleHeader.DEFAULT_HTTP_HEADER_NAME);
connection.addRequestProperty(ShuffleHeader.HTTP_HEADER_VERSION,
ShuffleHeader.DEFAULT_HTTP_HEADER_VERSION);
}
...
}
{code}
{code:title="/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java"}
public class JobEndNotifier implements Configurable {
...
timeout = conf.getInt(JobContext.MR_JOB_END_NOTIFICATION_TIMEOUT,
JobContext.DEFAULT_MR_JOB_END_NOTIFICATION_TIMEOUT);
...
protected boolean notifyURLOnce() {
...
conn.setReadTimeout(timeout);
...
}
...
}
{code}
The similar situations happen in "DFSZKFailoverController.java" in HDFS.
However, the "setReadTimeout()" in "FederationUtil.java",
"CredentialBasedAccessTokenProvider.java", and
"ConfRefreshTokenBasedAccessTokenProvider.java" call constants as their
argument. For example:
{code:title="/org/apache/hadoop/hdfs/server/federation/router/FederationUtil.java"}
public final class FederationUtil {
...
public static JSONArray getJmx(String beanQuery, String webAddress) {
...
URLConnection conn = jmxURL.openConnection();
conn.setConnectTimeout(5 * 1000);
...
}
...
}
{code}
I think these constant should be also configurable for different requirements
and environments.
> The timeout used in “setReadTimeout()” should be configurable
> -------------------------------------------------------------
>
> Key: HDFS-13935
> URL: https://issues.apache.org/jira/browse/HDFS-13935
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 3.1.1
> Reporter: zhoushulin
> Priority: Major
>
> The timeout used in "setReadTimeout" in "Fetcher.java" and
> "JobEndNotifier.java" in MapReduce is configurable as follows:
> {code:title="/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java"}
> class Fetcher<K,V> extends Thread {
> ...
> this.readTimeout =
> job.getInt(MRJobConfig.SHUFFLE_READ_TIMEOUT, DEFAULT_READ_TIMEOUT);
> ...
> private void setupShuffleConnection(String encHash) {
> // put url hash into http header
> connection.addRequestProperty(
> SecureShuffleUtils.HTTP_HEADER_URL_HASH, encHash);
> // set the read timeout
> connection.setReadTimeout(readTimeout);
> // put shuffle version into http header
> connection.addRequestProperty(ShuffleHeader.HTTP_HEADER_NAME,
> ShuffleHeader.DEFAULT_HTTP_HEADER_NAME);
> connection.addRequestProperty(ShuffleHeader.HTTP_HEADER_VERSION,
> ShuffleHeader.DEFAULT_HTTP_HEADER_VERSION);
> }
> ...
> }
> {code}
> {code:title="/org/apache/hadoop/mapreduce/v2/app/JobEndNotifier.java"}
> public class JobEndNotifier implements Configurable {
> ...
> timeout = conf.getInt(JobContext.MR_JOB_END_NOTIFICATION_TIMEOUT,
> JobContext.DEFAULT_MR_JOB_END_NOTIFICATION_TIMEOUT);
> ...
> protected boolean notifyURLOnce() {
> ...
> conn.setReadTimeout(timeout);
> ...
> }
> ...
> }
> {code}
> The similar situations happen in "DFSZKFailoverController.java" in HDFS.
> However, the "setReadTimeout()" in "FederationUtil.java",
> "CredentialBasedAccessTokenProvider.java", and
> "ConfRefreshTokenBasedAccessTokenProvider.java" call constants as their
> argument. For example:
> {code:title="/org/apache/hadoop/hdfs/server/federation/router/FederationUtil.java"}
> public final class FederationUtil {
> ...
> public static JSONArray getJmx(String beanQuery, String webAddress) {
> ...
> URLConnection conn = jmxURL.openConnection();
> conn.setConnectTimeout(5 * 1000);
> conn.setReadTimeout(5 * 1000);
> ...
> }
> ...
> }
> {code}
> I think these constant should be also configurable for different requirements
> and environments.
> The similar situation happens on method "setConnectTimeout()", so it should
> be also configurable.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
