Aggarwal-Raghav commented on PR #6831: URL: https://github.com/apache/hive/pull/6831#issuecomment-5918824847
> For Iceberg partitioned tables without a sort order, shouldDo() is what makes the input sorted: it sets `hasCustomPartitionOrSortExpression` → `setWriteOperationIsSorted` → HiveIcebergWriterBase picks the clustered writer. With this change, an INSERT/CTAS whose source has no column stats silently falls back to the fanout writer (one open writer per partition per task, tasks × partitions files) > > The reducer-skew problem is real, but can we address it without flipping the unknown-stats default, e.g. by keeping the sort while spreading a partition across reducers, or at least excluding storage-handler/non-native tables whose writers depend on sorted input? Thanks for sharing this information. Yes, you are correct with original approach https://github.com/apache/hive/commit/9d638d963ef976ba84cce8b98dd79e90573196a1 by checking stats/cardinality, there were 2 issues: 1. For blob tests `insert_into_dynamic_partitions.q` there was data loss for unclustered/fanout writer , rest all q files were plan changes. 2. Even if user ran `ANALYZE TABLE` so that stats existed and SDPO successfully kicked in, it still routed all rows for a massive partition to exactly one reducer. It would be a bottleneck. 3. For iceberg I handled by adding additional `if` check, therefore, iceberg q files passed — tested locally. > by keeping the sort while spreading a partition across reducers, I moved to this approach in latest commit. I injected a `rand()` UDF into the distribution keys. Sharing more details here: [HIVE-30101.pdf](https://github.com/user-attachments/files/32875469/HIVE-30101.pdf) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
