voonhous opened a new issue, #20629: URL: https://github.com/apache/hudi/issues/20629
Part of #20625. Line references are against master `db90a0da`. ### Problem When the dimension tables are MOR too, the build side of a join pays the slow per-split merge path. The date dynamic filter can then arrive just after `hudi.dynamic-filtering.wait-timeout` (default 1s), so the fact table splits are generated unpruned and timings swing between repeats. It shows locally only with a slow storage profile: `HudiMorReadBenchmark` dpp scenario with `-Dbench.s3.getMillis=60 -Dbench.s3.headMillis=40 -Dbench.s3.listMillis=150`. At the default profile the filter arrives in 0.2 to 0.4 s and pruning applies. ### Direction The estimator and per-split metadata items shrink the build-side latency; this item owns the end-to-end outcome. If those alone do not make pruning reliable, revisit the wait strategy for MOR, e.g. wait longer when the build side is selective, or wait on the partition-key filter specifically. ### Exit criteria - dpp scenario with the slow profile and `-Dbench.rounds=5`: `_rt` default reads the same fact input bytes as `_ro` default in every round, with no session overrides. - Date filter collection under 1s in that profile, or the connector demonstrably waits for it; a changed default is justified in the PR and documented in the config description. - No regression for non-DPP queries: scan wall time unchanged within noise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
