[jira] [Commented] (MAPREDUCE-6696) Add a configuration to limit the number of map tasks allowed per job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290497#comment-15290497 ] zhihai xu commented on MAPREDUCE-6696: -- Thanks [~jianhe]! These are good suggestions. I uploaded a new patch MAPREDUCE-6696.003.patch which addressed all your comments, Please review it thanks. > Add a configuration to limit the number of map tasks allowed per job. > - > > Key: MAPREDUCE-6696 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6696 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission >Affects Versions: 2.8.0 >Reporter: zhihai xu >Assignee: zhihai xu > Attachments: MAPREDUCE-6696.000.patch, MAPREDUCE-6696.001.patch, > MAPREDUCE-6696.002.patch, MAPREDUCE-6696.003.patch > > > Add a configuration "mapreduce.job.max.map" to limit the number of map tasks > allowed per job. It will be useful for Hadoop admin to save Hadoop cluster > resource by preventing users from submitting big mapreduce jobs. A mapredeuce > job with too many mappers may fail with OOM after running for long time. It > will be a big waste. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6696) Add a configuration to limit the number of map tasks allowed per job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated MAPREDUCE-6696: - Attachment: MAPREDUCE-6696.003.patch > Add a configuration to limit the number of map tasks allowed per job. > - > > Key: MAPREDUCE-6696 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6696 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission >Affects Versions: 2.8.0 >Reporter: zhihai xu >Assignee: zhihai xu > Attachments: MAPREDUCE-6696.000.patch, MAPREDUCE-6696.001.patch, > MAPREDUCE-6696.002.patch, MAPREDUCE-6696.003.patch > > > Add a configuration "mapreduce.job.max.map" to limit the number of map tasks > allowed per job. It will be useful for Hadoop admin to save Hadoop cluster > resource by preventing users from submitting big mapreduce jobs. A mapredeuce > job with too many mappers may fail with OOM after running for long time. It > will be a big waste. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6565) Configuration to use host name in delegation token service is not read from job.xml during MapReduce job execution.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290467#comment-15290467
]
Yuren Wu commented on MAPREDUCE-6565:
-
further thoughts on this fix.. the multi-home network setup for hadoop is
getting some attention in industry. The security token design using toke +
service name does not have updates to accommodate the complex network setup. HA
This quick fix just get by with executing mapreduce jobs. However, I would
suggest to create a new request to address the multi-home network and token
handling in a more organized effort. Security package has very succinct log and
it took me quite while to track down the issue. Properties under
hadoop.security should be handled in a single code base to interact with
various components. Credential token services such as retrieve/clone are
handled by different packages in multiple components. Current code is really
difficult to understand and manage.
> Configuration to use host name in delegation token service is not read from
> job.xml during MapReduce job execution.
> ---
>
> Key: MAPREDUCE-6565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6565
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chris Nauroth
>
> By default, the service field of a delegation token is populated based on
> server IP address. Setting {{hadoop.security.token.service.use_ip}} to
> {{false}} changes this behavior to use host name instead of IP address.
> However, this configuration property is not read from job.xml. Instead, it's
> read from a separate {{Configuration}} instance created during static
> initialization of {{SecurityUtil}}. This does not work correctly with
> MapReduce jobs if the framework is distributed by setting
> {{mapreduce.application.framework.path}} and the
> {{mapreduce.application.classpath}} is isolated to avoid reading
> core-site.xml from the cluster nodes. MapReduce tasks will fail to
> authenticate to HDFS, because they'll try to find a delegation token based on
> the NameNode IP address, even though at job submission time the tokens were
> generated using the host name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6565) Configuration to use host name in delegation token service is not read from job.xml during MapReduce job execution.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290449#comment-15290449
]
Yuren Wu commented on MAPREDUCE-6565:
-
a quick place to fix the map reduce issue is to add the following line into the
YarnClient.java
LOG.info("YARN CHILD CHECK SECURITY SETTING
USE_IP:"+job.get(CommonConfigurationKeys.HADOOP_SECURITY_TOKEN_SERVICE_USE_IP));
// get useIp flag for KMS
boolean useIp = job.getBoolean(
CommonConfigurationKeys.HADOOP_SECURITY_TOKEN_SERVICE_USE_IP,
CommonConfigurationKeys.HADOOP_SECURITY_TOKEN_SERVICE_USE_IP_DEFAULT);
LOG.debug("set securityutil token service use ip value from config."
+useIp);
SecurityUtil.setTokenServiceUseIp(useIp);
> Configuration to use host name in delegation token service is not read from
> job.xml during MapReduce job execution.
> ---
>
> Key: MAPREDUCE-6565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6565
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chris Nauroth
>
> By default, the service field of a delegation token is populated based on
> server IP address. Setting {{hadoop.security.token.service.use_ip}} to
> {{false}} changes this behavior to use host name instead of IP address.
> However, this configuration property is not read from job.xml. Instead, it's
> read from a separate {{Configuration}} instance created during static
> initialization of {{SecurityUtil}}. This does not work correctly with
> MapReduce jobs if the framework is distributed by setting
> {{mapreduce.application.framework.path}} and the
> {{mapreduce.application.classpath}} is isolated to avoid reading
> core-site.xml from the cluster nodes. MapReduce tasks will fail to
> authenticate to HDFS, because they'll try to find a delegation token based on
> the NameNode IP address, even though at job submission time the tokens were
> generated using the host name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6703) Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated MAPREDUCE-6703: --- Status: Patch Available (was: Open) > Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers > -- > > Key: MAPREDUCE-6703 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6703 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: MAPREDUCE-6703.001.patch > > > YARN-2882 and YARN-4335 introduces the concept of container ExecutionTypes > and specifically OPPORTUNISTIC containers. > The default ExecutionType is GUARANTEED. This JIRA proposes to allow users to > provide hints via config to the MR framework as to the number of containers > it would like to schedule as OPPORTUNISTIC. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5044) Have AM trigger jstack on task attempts that timeout before killing them
[
https://issues.apache.org/jira/browse/MAPREDUCE-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290193#comment-15290193
]
Ming Ma commented on MAPREDUCE-5044:
[~eepayne] I agree with your suggestion. Let us postpone it to a later time.
* {{signalContainers}} was initially suggested as an ordered list of
{{signalContainer}}. So it could include requests from the same container or
requests from different containers. It is true that the only use case we know
of so far is to include requests from the same container.
* We also discussed introducing other commands besides linux signal, for
example sleep command used to pause between signals, in that way, the new API
could be just like
{noformat}
public static SignalContainerRequest newInstance(ContainerId containerId,
Iterable signals) {
...
}
{noformat}
* Will the {{required}} in the protocol buffer definition create any issue if
we do rolling upgrade from 2.8 to 2.9 and the 2.9 MR AM might send a list of
SignalContainerCommandProto to 2.8 NM? Maybe 2.8 NM just discards the message,
not a big deal. Regardless, that is a separate issue that we don't need to
address it here.
{noformat}
message SignalContainerRequestProto {
required SignalContainerCommandProto command = 2;
}
{noformat}
> Have AM trigger jstack on task attempts that timeout before killing them
>
>
> Key: MAPREDUCE-5044
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5044
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Affects Versions: 2.1.0-beta
>Reporter: Jason Lowe
>Assignee: Gera Shegalov
> Attachments: MAPREDUCE-5044.008.patch, MAPREDUCE-5044.009.patch,
> MAPREDUCE-5044.v01.patch, MAPREDUCE-5044.v02.patch, MAPREDUCE-5044.v03.patch,
> MAPREDUCE-5044.v04.patch, MAPREDUCE-5044.v05.patch, MAPREDUCE-5044.v06.patch,
> MAPREDUCE-5044.v07.local.patch, Screen Shot 2013-11-12 at 1.05.32 PM.png,
> Screen Shot 2013-11-12 at 1.06.04 PM.png
>
>
> When an AM expires a task attempt it would be nice if it triggered a jstack
> output via SIGQUIT before killing the task attempt. This would be invaluable
> for helping users debug their hung tasks, especially if they do not have
> shell access to the nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6696) Add a configuration to limit the number of map tasks allowed per job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290171#comment-15290171 ] Jian He commented on MAPREDUCE-6696: also, may be throw IllegalArgumentException instead of RuntimeException ? > Add a configuration to limit the number of map tasks allowed per job. > - > > Key: MAPREDUCE-6696 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6696 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission >Affects Versions: 2.8.0 >Reporter: zhihai xu >Assignee: zhihai xu > Attachments: MAPREDUCE-6696.000.patch, MAPREDUCE-6696.001.patch, > MAPREDUCE-6696.002.patch > > > Add a configuration "mapreduce.job.max.map" to limit the number of map tasks > allowed per job. It will be useful for Hadoop admin to save Hadoop cluster > resource by preventing users from submitting big mapreduce jobs. A mapredeuce > job with too many mappers may fail with OOM after running for long time. It > will be a big waste. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6696) Add a configuration to limit the number of map tasks allowed per job.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290170#comment-15290170
]
Jian He commented on MAPREDUCE-6696:
I see, thanks for your explanation. patch looks good to me, minor nit:
may be useful to print the current number of map tasks too in the exception
message ? just to be more clear.
{code}
new RuntimeException("The number of map tasks exceeded limit " +
maxMaps);
{code}
> Add a configuration to limit the number of map tasks allowed per job.
> -
>
> Key: MAPREDUCE-6696
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6696
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: job submission
>Affects Versions: 2.8.0
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: MAPREDUCE-6696.000.patch, MAPREDUCE-6696.001.patch,
> MAPREDUCE-6696.002.patch
>
>
> Add a configuration "mapreduce.job.max.map" to limit the number of map tasks
> allowed per job. It will be useful for Hadoop admin to save Hadoop cluster
> resource by preventing users from submitting big mapreduce jobs. A mapredeuce
> job with too many mappers may fail with OOM after running for long time. It
> will be a big waste.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6565) Configuration to use host name in delegation token service is not read from job.xml during MapReduce job execution.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290047#comment-15290047
]
Yuren Wu commented on MAPREDUCE-6565:
-
need more thought around this.. just realized that setting this flag caused
hdfs delegation tokens cannot be looked up by ipc client..
the basic issue is that delegation tokens are populated by each individual
client and they do not have a conform protocol to follow to use ip or host
name.
i will post more findings and suggestions later.
> Configuration to use host name in delegation token service is not read from
> job.xml during MapReduce job execution.
> ---
>
> Key: MAPREDUCE-6565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6565
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chris Nauroth
>
> By default, the service field of a delegation token is populated based on
> server IP address. Setting {{hadoop.security.token.service.use_ip}} to
> {{false}} changes this behavior to use host name instead of IP address.
> However, this configuration property is not read from job.xml. Instead, it's
> read from a separate {{Configuration}} instance created during static
> initialization of {{SecurityUtil}}. This does not work correctly with
> MapReduce jobs if the framework is distributed by setting
> {{mapreduce.application.framework.path}} and the
> {{mapreduce.application.classpath}} is isolated to avoid reading
> core-site.xml from the cluster nodes. MapReduce tasks will fail to
> authenticate to HDFS, because they'll try to find a delegation token based on
> the NameNode IP address, even though at job submission time the tokens were
> generated using the host name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5044) Have AM trigger jstack on task attempts that timeout before killing them
[
https://issues.apache.org/jira/browse/MAPREDUCE-5044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289975#comment-15289975
]
Eric Payne commented on MAPREDUCE-5044:
---
[~mingma], thank you very much for the comments. I have one question:
{quote}
- ... it might be useful to rename signalContainer to signalContainers so that
we don't need to modify the API later, which means some new structure like
SignalContainersRequest. What is your take?
{quote}
I would rather not rename {{signalContainer}} to {{signalContainers}} because
{{signalContainers}} sounds to me like the purpose is to send one signal to
multiple containers rather than to send multiple signals to one container.
Calling it {{signalsContainer}} (plural {{signals}}) also sounds awkward. So, I
think {{signalContainer}} is the best option.
Regarding {{SignalContainerRequest}}, if we want the {{signalContainer}} API to
be fully compatible with sending multiple signals, I think
{{SignalContainerRequest}} would need to add an interface for
{{SignalContainerRequest#newInstance}} that included both pause and a list of
signals. Maybe something like this:
{code}
public static SignalContainerRequest newInstance(ContainerId containerId,
int pause, Iterable signals) {
...
}
{code}
I think it would be best to add that interface to {{SignalContainerRequest}} in
the future when we are ready to implement the rest of the "sending multiple
signals" feature. Thoughts?
> Have AM trigger jstack on task attempts that timeout before killing them
>
>
> Key: MAPREDUCE-5044
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5044
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Affects Versions: 2.1.0-beta
>Reporter: Jason Lowe
>Assignee: Gera Shegalov
> Attachments: MAPREDUCE-5044.008.patch, MAPREDUCE-5044.009.patch,
> MAPREDUCE-5044.v01.patch, MAPREDUCE-5044.v02.patch, MAPREDUCE-5044.v03.patch,
> MAPREDUCE-5044.v04.patch, MAPREDUCE-5044.v05.patch, MAPREDUCE-5044.v06.patch,
> MAPREDUCE-5044.v07.local.patch, Screen Shot 2013-11-12 at 1.05.32 PM.png,
> Screen Shot 2013-11-12 at 1.06.04 PM.png
>
>
> When an AM expires a task attempt it would be nice if it triggered a jstack
> output via SIGQUIT before killing the task attempt. This would be invaluable
> for helping users debug their hung tasks, especially if they do not have
> shell access to the nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6686) Add a way to download the job config from the mapred CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289654#comment-15289654 ] Hudson commented on MAPREDUCE-6686: --- SUCCESS: Integrated in Hadoop-trunk-Commit #9817 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9817/]) MAPREDUCE-6686. Add a way to download the job config from the mapred CLI (rkanter: rev 992a49353f0bdbc71534bfeeb134112c5624b94b) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapredCommands.md > Add a way to download the job config from the mapred CLI > > > Key: MAPREDUCE-6686 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6686 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 2.9.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.9.0 > > Attachments: MAPREDUCE-6686.001.patch, MAPREDUCE-6686.002.patch > > > It would be convenient if there was a way to easily grab the job > configuration via the CLI instead of having to find and go to the specific > HDFS location to grab it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6686) Add a way to download the job config from the mapred CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6686: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.9.0 Status: Resolved (was: Patch Available) Thanks for the review [~qwertymaniac]. Committed to trunk and branch-2! > Add a way to download the job config from the mapred CLI > > > Key: MAPREDUCE-6686 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6686 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 2.9.0 >Reporter: Robert Kanter >Assignee: Robert Kanter > Fix For: 2.9.0 > > Attachments: MAPREDUCE-6686.001.patch, MAPREDUCE-6686.002.patch > > > It would be convenient if there was a way to easily grab the job > configuration via the CLI instead of having to find and go to the specific > HDFS location to grab it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6565) Configuration to use host name in delegation token service is not read from job.xml during MapReduce job execution.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289614#comment-15289614
]
Yuren Wu commented on MAPREDUCE-6565:
-
Looks like no patch is available here. I propose to handle it in the
DFSUtil.createKeyProvider method.
// get useIp flag for KMS
boolean useIp = conf.getBoolean(
CommonConfigurationKeys.HADOOP_SECURITY_TOKEN_SERVICE_USE_IP,
CommonConfigurationKeys.HADOOP_SECURITY_TOKEN_SERVICE_USE_IP_DEFAULT);
LOG.debug("set
"+CommonConfigurationKeys.HADOOP_SECURITY_TOKEN_SERVICE_USE_IP +" value from
config." +useIp);
SecurityUtil.setTokenServiceUseIp(useIp);
> Configuration to use host name in delegation token service is not read from
> job.xml during MapReduce job execution.
> ---
>
> Key: MAPREDUCE-6565
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6565
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Chris Nauroth
>
> By default, the service field of a delegation token is populated based on
> server IP address. Setting {{hadoop.security.token.service.use_ip}} to
> {{false}} changes this behavior to use host name instead of IP address.
> However, this configuration property is not read from job.xml. Instead, it's
> read from a separate {{Configuration}} instance created during static
> initialization of {{SecurityUtil}}. This does not work correctly with
> MapReduce jobs if the framework is distributed by setting
> {{mapreduce.application.framework.path}} and the
> {{mapreduce.application.classpath}} is isolated to avoid reading
> core-site.xml from the cluster nodes. MapReduce tasks will fail to
> authenticate to HDFS, because they'll try to find a delegation token based on
> the NameNode IP address, even though at job submission time the tokens were
> generated using the host name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6703) Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
[
https://issues.apache.org/jira/browse/MAPREDUCE-6703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated MAPREDUCE-6703:
---
Attachment: MAPREDUCE-6703.001.patch
Uploading initial patch
* Changes to allow the MRAppMaster to request a mix of *OPPORTUNISTIC* and
*GUARANTEED* containers for it first attempt map tasks (reduce tasks and maps
to be retried are always GUARANTEED)
* User specifies number of *OPPORTUNISTIC* Containers via the
*mapreduce.job.num-opportunistic-maps-per-100* job conf : Default value is 0,
which implies all maps will be guaranteed. A value of 100 means all maps will
be requested as opportunistic. For any other value say _x_, the *FIRST* _x_
maps requested by the AM will be opportunistic. If the total number of maps for
the job < _x_, then *ALL* maps will be opportunistic
* The patch also contains a test case {{TestMROpportunisticMaps}} that uses the
*MiniYARNCluster* to run a simple external sorting MR job using the above flag.
*PS* : This feature currently requires Distributed Scheduling (YARN-2885), NM
Queuing (YARN-2883) and the AMRMProxyService (YARN-2884) to be turned on in the
yarn cluster.
> Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers
> --
>
> Key: MAPREDUCE-6703
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6703
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: MAPREDUCE-6703.001.patch
>
>
> YARN-2882 and YARN-4335 introduces the concept of container ExecutionTypes
> and specifically OPPORTUNISTIC containers.
> The default ExecutionType is GUARANTEED. This JIRA proposes to allow users to
> provide hints via config to the MR framework as to the number of containers
> it would like to schedule as OPPORTUNISTIC.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6700) Jobhistory server attempt and task table not loading maps/reduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-6700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288627#comment-15288627
]
Bibin A Chundatt commented on MAPREDUCE-6700:
-
[~rohithsharma]
{quote}
Bibin A Chundatt would you mind providing rebase patch to branch-2.8 please?
{quote}
Offline have asked Varun to upload don't have rights for to upload in
YARN-3840.
> Jobhistory server attempt and task table not loading maps/reduce
>
>
> Key: MAPREDUCE-6700
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6700
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
>
> Browser
> ===
> Chrome
> Steps to reproduce
> ==
> # Submit mapreduce application with 20 maps
> # Wait till completion of mapreduce application
> # Check maps attempts page
> {{jobhistory/attempts/job_1463446678437_0003/m/SUCCESSFUL}}
> and {{jobhistory/tasks/job_1463446678437_0003/m}} page
> Actual
> =
> Table not loading.
> Sort based on any column other than attempt contents are loaded.
> Column 0 is of *natural sorting* and not working. So waiting for ever to be
> sorted.
> {noformat}
> SCRIPT438: Object doesn't support property or method 'natural-asc'
> jquery.dataTables.min.js, line 86 character 179
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6700) Jobhistory server attempt and task table not loading maps/reduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-6700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt reassigned MAPREDUCE-6700:
---
Assignee: Bibin A Chundatt
> Jobhistory server attempt and task table not loading maps/reduce
>
>
> Key: MAPREDUCE-6700
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6700
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Blocker
>
> Browser
> ===
> Chrome
> Steps to reproduce
> ==
> # Submit mapreduce application with 20 maps
> # Wait till completion of mapreduce application
> # Check maps attempts page
> {{jobhistory/attempts/job_1463446678437_0003/m/SUCCESSFUL}}
> and {{jobhistory/tasks/job_1463446678437_0003/m}} page
> Actual
> =
> Table not loading.
> Sort based on any column other than attempt contents are loaded.
> Column 0 is of *natural sorting* and not working. So waiting for ever to be
> sorted.
> {noformat}
> SCRIPT438: Object doesn't support property or method 'natural-asc'
> jquery.dataTables.min.js, line 86 character 179
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6700) Jobhistory server attempt and task table not loading maps/reduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-6700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288557#comment-15288557
]
Rohith Sharma K S commented on MAPREDUCE-6700:
--
I have reopened the JIRA YARN-3840 and lets handle in the original JIRA only.
Till YARN-3840 is closed, I will keep open this JIRA for tracking.
> Jobhistory server attempt and task table not loading maps/reduce
>
>
> Key: MAPREDUCE-6700
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6700
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Blocker
>
> Browser
> ===
> Chrome
> Steps to reproduce
> ==
> # Submit mapreduce application with 20 maps
> # Wait till completion of mapreduce application
> # Check maps attempts page
> {{jobhistory/attempts/job_1463446678437_0003/m/SUCCESSFUL}}
> and {{jobhistory/tasks/job_1463446678437_0003/m}} page
> Actual
> =
> Table not loading.
> Sort based on any column other than attempt contents are loaded.
> Column 0 is of *natural sorting* and not working. So waiting for ever to be
> sorted.
> {noformat}
> SCRIPT438: Object doesn't support property or method 'natural-asc'
> jquery.dataTables.min.js, line 86 character 179
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
