hhr293 commented on PR #58424: URL: https://github.com/apache/spark/pull/58424#issuecomment-5644520952
Hi,@LuciferYang @sunchao @peter-toth I’ve updated the PR based on the latest review comments and split the changes into three focused commits: correctness/consistency fixes, the corresponding regression tests, and the `COUNT(DISTINCT)` -> `MIN/MAX` rewrite. The correctness side now fail-closes the clock-dependent STRING -> TIMESTAMP cases (including nested casts), preserves Alias metadata/qualifiers, rejects explicit join hints, tightens the repeatability/source guards, and adds coverage for idempotency, multiset parity, ANSI error behavior, fixture cleanup, and TIMESTAMP_NTZ as a positive control. I also reran the profitability experiment to check the proposed broadcast/size gate more directly. With the input fixed at ~100 MB and the rewrite-off plan remaining `BroadcastHashJoin` for every run, varying only the per-key multiplicity gives: `m=1`: 9.11s OFF / 7.18s ON `m=3`: 12.46s OFF / 6.13s ON `m=5`: 17.80s OFF / 4.52s ON `m=7`: 21.26s OFF / 3.31s ON `m=10`: 29.18s OFF / 2.95s ON So broadcastability itself does not remove the self-join pair expansion: with the same input size and the same BHJ strategy, the rewrite changes from roughly neutral at low multiplicity to a large win at high multiplicity. Because of that, I kept the rule `internal`, experimental, and disabled by default rather than adding a coarse `sizeInBytes <= autoBroadcastJoinThreshold` gate. I also updated the PR description with the benchmark matrix and explicit lifecycle/promotion criteria. Could you please take another look when you have a chance? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
