[
https://issues.apache.org/jira/browse/HIVE-21929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16873921#comment-16873921
]
Oleksiy Sayankin commented on HIVE-21929:
-----------------------------------------
*FIXED*
*ROOT CAUSE*
Implementation of
[MAPREDUCE-5785|https://issues.apache.org/jira/browse/MAPREDUCE-5785] was not
covered in Hive 3. The default value for properties
{{mapreduce.map/reduce.memory.mb}} was changed from 1024 to -1. {{JobConf}}
should parse java options (other configuration properties) look for {{-Xmx}}
option, and calculate memory for container dynamically or use the non-default
value from {{mapreduce.map/reduce.memory.mb}}.
*SOLUTION*
Hive on Tez uses own hierarchy of Java options and container resources
properties which do not fully correspond to Hadoop MR Job configuration.
So, for backward compatibility of Hive configurations and Hive on Tez vertices
configuration on Hive side and Tez side, reduce coupling of Hive on Tez with
Hadoop MR Job abstraction it was decided to fix Vertex container configuration
code without rewriting all flow of Vertices configuration properties.
{code}
int hiveTezContainerSize = HiveConf.getIntVar(conf,
HiveConf.ConfVars.HIVETEZCONTAINERSIZE);
int mapMemoryMb = conf.getInt(MRJobConfig.MAP_MEMORY_MB,
MRJobConfig.DEFAULT_MAP_MEMORY_MB);
int memory = MRJobConfig.DEFAULT_MAP_MEMORY_MB;
if (hiveTezContainerSize > 0) {
memory = hiveTezContainerSize;
} else if (mapMemoryMb > 0) {
memory = mapMemoryMb;
}
{code}
*EFFECTS*
1. Hive on Tez Vertex memory/cpu configuration.
> Hive on Tez requers explicite set of property "hive.tez.container.size"
> -----------------------------------------------------------------------
>
> Key: HIVE-21929
> URL: https://issues.apache.org/jira/browse/HIVE-21929
> Project: Hive
> Issue Type: Bug
> Reporter: Oleksiy Sayankin
> Assignee: Oleksiy Sayankin
> Priority: Major
> Attachments: HIVE-21929.1.patch
>
>
> Without the explicit setting of the property {{hive.tez.container.size}} Tez
> client submit to the YARN memory size as "-1". After that container creation
> is rejected by YARN.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
