[
https://issues.apache.org/jira/browse/HIVE-17292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123544#comment-16123544
]
Rui Li commented on HIVE-17292:
-------------------------------
I mean we did set {{RM_SCHEDULER_MINIMUM_ALLOCATION_MB}} to 512 in the code:
{code}
public MiniSparkShim(Configuration conf, int numberOfTaskTrackers,
String nameNode, int numDir) throws IOException {
mr = new MiniSparkOnYARNCluster("sparkOnYarn");
conf.set("fs.defaultFS", nameNode);
conf.set("yarn.resourcemanager.scheduler.class",
"org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler");
// disable resource monitoring, although it should be off by default
conf.setBoolean(YarnConfiguration.YARN_MINICLUSTER_CONTROL_RESOURCE_MONITORING,
false);
conf.setInt(YarnConfiguration.YARN_MINICLUSTER_NM_PMEM_MB, 2048);
conf.setInt(YarnConfiguration.RM_SCHEDULER_MINIMUM_ALLOCATION_MB, 512);
conf.setInt(YarnConfiguration.RM_SCHEDULER_MAXIMUM_ALLOCATION_MB, 2048);
configureImpersonation(conf);
mr.init(conf);
mr.start();
this.conf = mr.getConfig();
}
{code}
Do you mean {{RM_SCHEDULER_MINIMUM_ALLOCATION_MB}} is different from
{{yarn.scheduler.minimum-allocation-mb}}, or we set it in the wrong place?
> Change TestMiniSparkOnYarnCliDriver test configuration to use the configured
> cores
> ----------------------------------------------------------------------------------
>
> Key: HIVE-17292
> URL: https://issues.apache.org/jira/browse/HIVE-17292
> Project: Hive
> Issue Type: Sub-task
> Components: Spark, Test
> Affects Versions: 3.0.0
> Reporter: Peter Vary
> Assignee: Peter Vary
> Attachments: HIVE-17292.1.patch
>
>
> Currently the {{hive-site.xml}} for the {{TestMiniSparkOnYarnCliDriver}} test
> defines 2 cores, and 2 executors, but only 1 is used, because the MiniCluster
> does not allows the creation of the 3rd container.
> The FairScheduler uses 1GB increments for memory, but the containers would
> like to use only 512MB. We should change the fairscheduler configuration to
> use only the requested 512MB
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
