Rui Li commented on HIVE-17270:

When automatically deciding numReducers, it should be no less than numCores. On 
the other side, {{spark.executor.instances}} is numContainers spark will 
request from YARN. How many containers can be really allocated is up to YARN.
So I can think of two possible reasons why we have 2 instead of 4 here.
# Only 1 container is allocated, in which case 2 is the right way to go.
# 2 containers are allocated but only 1 has started running when we get the 
executor count. This is a common case in real cluster and can make our test 
result unstable. We should find a way to fix it.

I guess we can monitor the log of the mini-yarn test to see how many cores we 
really have during execution.

> Qtest results show wrong number of executors
> --------------------------------------------
>                 Key: HIVE-17270
>                 URL: https://issues.apache.org/jira/browse/HIVE-17270
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>    Affects Versions: 3.0.0
>            Reporter: Peter Vary
>            Assignee: Peter Vary
> The hive-site.xml shows, that the TestMiniSparkOnYarnCliDriver uses 2 cores, 
> and 2 executor instances to run the queries. See: 
> https://github.com/apache/hive/blob/master/data/conf/spark/yarn-client/hive-site.xml#L233
> When reading the log files for the query tests, I see the following:
> {code}
> 2017-08-08T07:41:03,315  INFO [0381325d-2c8c-46fb-ab51-423defaddd84 main] 
> session.SparkSession: Spark cluster current has executors: 1, total cores: 2, 
> memory per executor: 512M, memoryFraction: 0.4
> {code}
> See: 
> When running the tests against a real cluster, I found that running an 
> explain query for the first time I see 1 executor, but running it for the 
> second time I see 2 executors.
> Also setting some spark configuration on the cluster resets this behavior. 
> For the first time I will see 1 executor, and for the second time I will see 
> 2 executors again.

This message was sent by Atlassian JIRA

Reply via email to