[
https://issues.apache.org/jira/browse/HIVE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123383#comment-16123383
]
Peter Vary commented on HIVE-17292:
-----------------------------------
Yeah, I was able to identify the root cause of the problem.
When a scheduler is used then there is an additional configuration value for
the minimum allocation. See:
https://github.com/apache/hadoop/blob/branch-2.8/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java#L161
This is defined by {{yarn.scheduler.minimum-allocation-mb}}. By default this is
set to {{1024}}, so the minimum memory allocation is 1G. Also this is
documented here:
https://hadoop.apache.org/docs/r2.8.0/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
??The minimum allocation for every container request at the RM, in MBs. Memory
requests lower than this will throw a InvalidResourceRequestException.??
When I set {{yarn.scheduler.minimum-allocation-mb}} to {{512}} in
hive-site.xml, then I will get 4 reducers as expected.
As for the test results - we have to wait for HIVE-17291 to get in to have
consistent outputs.
Thanks,
Peter
> Change TestMiniSparkOnYarnCliDriver test configuration to use the configured
> cores
> ----------------------------------------------------------------------------------
>
> Key: HIVE-17292
> URL: https://issues.apache.org/jira/browse/HIVE-17292
> Project: Hive
> Issue Type: Sub-task
> Components: Spark, Test
> Affects Versions: 3.0.0
> Reporter: Peter Vary
> Assignee: Peter Vary
> Attachments: HIVE-17292.1.patch
>
>
> Currently the {{hive-site.xml}} for the {{TestMiniSparkOnYarnCliDriver}} test
> defines 2 cores, and 2 executors, but only 1 is used, because the MiniCluster
> does not allows the creation of the 3rd container.
> The FairScheduler uses 1GB increments for memory, but the containers would
> like to use only 512MB. We should change the fairscheduler configuration to
> use only the requested 512MB
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
