[
https://issues.apache.org/jira/browse/HIVE-17291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16126738#comment-16126738
]
Rui Li commented on HIVE-17291:
-------------------------------
[~pvary],
bq. Logic suggests, that in this case we will request new executors based on
the new settings, and we are in the middle of the query test file, so Rui Li's
magic does not applies here
I think it's because when we get a spark session, we'll set it to the
SessionState:
https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java#L127
In QTestUtil, we override the {{setSparkSession}} method, so each time we
create a new spark session, we'll wait for it to init. Currently, we have only
one executor, and we wait until numCores > 1. So we don't have any flakiness.
With your change in HIVE-17292, we'll need to wait until we have 4 cores.
[~xuefuz], your concerns about dynamic allocation are valid. I think current
implementation only uses "size per reducer" when dynamic allocation is enabled.
For static allocation, as I mentioned, the available executor/core can only
give us more reducers than needed (decided by "size per reducer"). As long as
we have a reasonable {{hive.exec.reducers.bytes.per.reducer}}, we won't
underestimate the num of reducers. So I think we're good with the current
logic. Does that make sense?
> Set the number of executors based on config if client does not provide
> information
> ----------------------------------------------------------------------------------
>
> Key: HIVE-17291
> URL: https://issues.apache.org/jira/browse/HIVE-17291
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Affects Versions: 3.0.0
> Reporter: Peter Vary
> Assignee: Peter Vary
> Attachments: HIVE-17291.1.patch
>
>
> When calculating the memory and cores and the client does not provide
> information we should try to use the one provided by default. This can happen
> on startup, when {{spark.dynamicAllocation.enabled}} is not enabled
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
