[
https://issues.apache.org/jira/browse/HIVE-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120769#comment-15120769
]
Xuefu Zhang commented on HIVE-12951:
------------------------------------
Hi [~lirui], that's an interesting idea. I can see that prewarming executors is
no longer needed if parallelism is determined by the expected executors.
However, prewarming is meant for shor-lived sessions, such as ETL job launched
by Oozie. For interactive sessions, this feature doesn't matter much.
Therefore, we probably don't want to change the way of determining parallelism
just for a less common use case.
Now let's see if using expected number of executors to determine makes sense in
the general case. In this case, we don't worry about the performance for the
first query. Whether we have static or dynamic allocation, the parallelism is
determined in the same way if we use the actual number of executors. On the
other hand, if we use the expected number of executors, it seems the expected
number of executors is undefined in case of dynamic allocation. Also, the
expected may not always match the actual. If there is a big difference, there
could be performance implications.
In conclusion, I can certainly see some benefits of using the expected number
of executors. However, there also seem some unknowns as well. I think we don't
have to make a call at this point. Maybe we can revisit once we learn more or
get more user feedback?
What do you think?
> Reduce Spark executor prewarm timeout to 5s
> -------------------------------------------
>
> Key: HIVE-12951
> URL: https://issues.apache.org/jira/browse/HIVE-12951
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Affects Versions: 1.2.0
> Reporter: Xuefu Zhang
> Assignee: Xuefu Zhang
> Attachments: HIVE-12951.patch
>
>
> Currently it's set to 30s, which tends to be longer than needed. Reduce it to
> 5s, only considering jvm startup time. (Eventually, we may want to make this
> configurable.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
