Bikas’ comment ( and mine below ) is relevant only for task specific settings. Hive does not override any settings for the Tez AM so the tez configs for the AM memory/vcores will reflect at runtime.
I believe Hive has a proxy config - hive.tez.cpu.vcores - for (3) which may be why your setting for (3) is not taking effect. Additionally, Hive also tends to fallback to MR based values if tez specific values are not specified which might be something else you may wish to ask on the Hive user list. thanks — Hitesh > On May 4, 2016, at 10:14 PM, Bikas Saha <bi...@apache.org> wrote: > > IIRC 1) will override 2) since 2) is the tez config and 1) is the Hive config > that is a proxy for 2). > > Bikas > > Date: Mon, 25 Apr 2016 13:57:38 +0530 > Subject: Varying vcores/ram for hive queries running Tez engine > From: nk94.nitinku...@gmail.com > To: u...@hive.apache.org; user@tez.apache.org > > I was trying to benchmark some hive queries. I am using the tez execution > engine. I varied the values of the following properties: > • hive.tez.container.size > • tez.task.resource.memory.mb > • tez.task.resource.cpu.vcores > Changes in values for property 1 is reflected properly. However it seems that > hive does not respect changes in values of property 3; it always allocates > one vcore per requested container (RM is configured to use the > DominantResourceCalculator). This got me thinking about the precedence of > property values in hive and tez. > I have the following questions with respect to these configurations > • Does hive respect the set values for the properties 2 and 3 at all? > • If I set property 1 to a value say 2048 MB and property 2 is set to a > value of say 1024 MB does this mean that I am wasting about a GB of memory > for each spawned container? > • Is there a property in hive similar to property 1 that allows me to > use the 'set' command in the .hql file to specify the number of vcores to use > per container? > • Changes in value for the property tez.am.resource.cpu.vcores are > reflected at runtime. However I do not observe the same behaviour with > property 3. Are there other configurations that take precedence over it? > Your inputs and suggestions would be highly appreciated. > > Thanks! > > > PS: Tests conducted on a 5 node cluster running HDP 2.3.0
