Raghav Aggarwal created HIVE-30101:
--------------------------------------
Summary: Prevent SortedDynPartitionOptimizer from injecting
ReduceSink when stats are missing
Key: HIVE-30101
URL: https://issues.apache.org/jira/browse/HIVE-30101
Project: Hive
Issue Type: Bug
Reporter: Raghav Aggarwal
Assignee: Raghav Aggarwal
{{hive.optimize.sort.dynamic.partition.threshold}} is set to 0 by default
making the decision cost based. If the table is lacking stats still this config
is kicked-in and causing severe performance degradation. If 200 reducers are
getting launched only 4 - 5 of them are doing actual work. This is observed in
our production environment as well
This is a regression in a way from Hive3 as
{{hive.optimize.sort.dynamic.partition}} was by-default false and now its
enable in hive4. For external tables where stats are stale or not present (if
autogather is false then) it causes performance issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
