peter-toth commented on PR #58635: URL: https://github.com/apache/spark/pull/58635#issuecomment-6468432320
@pan3793 FYI, a case this PR leaves as is, in case it is in scope here. With the default `reuseBroadcastOnly`, `PartitionPruning` adds an `onlyInBroadcast` DPP filter on a key the scan offers for runtime filtering, such as a partition column. `dynamicPruningFilters` then keeps `InjectRuntimeFilter` from adding a bloom filter on that key. In a shuffle join there is no broadcast to reuse, so physical planning replaces that DPP filter with `true`. The key ends up with no runtime filter at all, and the fact side is shuffled unfiltered. Measured at this PR's head with AQE on and off, for a V1 Parquet table partitioned by the join key and for a V2 scan that offers a `bucket(N, id)` source column. Turning DPP off brings the bloom filter back. Would you like to cover it here, or should I open a separate ticket? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
