[
https://issues.apache.org/jira/browse/HIVE-24485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17278522#comment-17278522
]
Gopal Vijayaraghavan commented on HIVE-24485:
---------------------------------------------
[~okumin]: can you change the patch to move the parameters into edgeProp (as in
set it there and send it in, instead of adding a conf).
{code}
public void setSlowStart(boolean slowStart) {
this.isSlowStart = slowStart;
}
{code}
Let me say that this doesn't change what it does right now, but it is easier to
tweak it by the edgeProp within planning (& the debugger is neater, because the
object has everything).
> Make the slow-start behavior tunable
> ------------------------------------
>
> Key: HIVE-24485
> URL: https://issues.apache.org/jira/browse/HIVE-24485
> Project: Hive
> Issue Type: Improvement
> Components: Hive, Tez
> Affects Versions: 3.1.2, 4.0.0
> Reporter: okumin
> Assignee: okumin
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> This ticket would enable users to configure the timing of slow-start with
> `tez.shuffle-vertex-manager.min-src-fraction` and
> `tez.shuffle-vertex-manager.max-src-fraction`.
> Hive on Tez currently doesn't honor these parameters and ShuffleVertexManager
> always uses the default value.
> We can control the timing to start vertexes the accuracy of estimated input
> size if we can tweak these ones. This is useful when a vertex has tasks that
> process a different amount of data.
>
> We can reproduce the issue with this query.
> {code:java}
> SET hive.tez.auto.reducer.parallelism=true;
> SET hive.tez.min.partition.factor=1.0; -- enforce auto-parallelism
> SET tez.shuffle-vertex-manager.min-src-fraction=0.55;
> SET tez.shuffle-vertex-manager.max-src-fraction=0.95;
> CREATE TABLE mofu (name string);
> INSERT INTO mofu (name) VALUES ('12345');
> SELECT name, count(*) FROM mofu GROUP BY name;{code}
> The fractions are ignored.
> {code:java}
> 2020-12-04 11:41:42,484 [INFO] [Dispatcher thread {Central}]
> |vertexmanager.ShuffleVertexManagerBase|: Settings minFrac: 0.25 maxFrac:
> 0.75 auto: true desiredTaskIput: 256000000
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
