David Mollitor created HIVE-24707:
-------------------------------------
Summary: Apply Sane Default for Tez Containers as Last Resort
Key: HIVE-24707
URL: https://issues.apache.org/jira/browse/HIVE-24707
Project: Hive
Issue Type: Improvement
Reporter: David Mollitor
{code:java|title=DagUtils.java}
public static Resource getContainerResource(Configuration conf) {
int memory = HiveConf.getIntVar(conf,
HiveConf.ConfVars.HIVETEZCONTAINERSIZE) > 0 ?
HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVETEZCONTAINERSIZE) :
conf.getInt(MRJobConfig.MAP_MEMORY_MB, MRJobConfig.DEFAULT_MAP_MEMORY_MB);
int cpus = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVETEZCPUVCORES) > 0
?
HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVETEZCPUVCORES) :
conf.getInt(MRJobConfig.MAP_CPU_VCORES,
MRJobConfig.DEFAULT_MAP_CPU_VCORES);
return Resource.newInstance(memory, cpus);
}
{code}
If Tez Container Size or VCores is an invalid value ( <= 0 ) then it falls back
onto the MapReduce configurations, but if the MapReduce configurations have
invalid values ( <= 0 ), they are excepted regardless and this will cause
failures down the road.
This code should also check the MapReduce values and fall back to MapReduce
default values if they are <= 0.
Also, some logging would be nice here too, reporting about where the
configuration values came from.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
