[
https://issues.apache.org/jira/browse/HIVE-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15120651#comment-15120651
]
Rui Li commented on HIVE-12951:
-------------------------------
Hi [~xuefuz], I just thought more about this. Maybe we should use the "expected
# executors" instead of "available # executors" to decide the parallelism (fall
back to what we do now if "expected" is unavailable, e.g. standalone mode).
Intuitively, users have more knowledge about their job and will set the
expected # executors accordingly. We should honor that, even if the RM cannot
grant the required amount. This way, each executor may be assigned more
reducers, but the job is less likely to fail because each reducer handles the
expected amount of data. Another benefit is our test outputs can be more
deterministic.
We can do this in separate JIRA if you think it's OK.
> Reduce Spark executor prewarm timeout to 5s
> -------------------------------------------
>
> Key: HIVE-12951
> URL: https://issues.apache.org/jira/browse/HIVE-12951
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Affects Versions: 1.2.0
> Reporter: Xuefu Zhang
> Assignee: Xuefu Zhang
> Attachments: HIVE-12951.patch
>
>
> Currently it's set to 30s, which tends to be longer than needed. Reduce it to
> 5s, only considering jvm startup time. (Eventually, we may want to make this
> configurable.)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
