parthchandra opened a new issue, #6832: URL: https://github.com/apache/datafusion-comet/issues/6832
### Describe the bug ## What is the problem? With the broadcast-build-side auto-bridge (`spark.comet.convert.broadcastBuildSide.enabled`, added in #6786 / #6008), a fact joined to **two or more** unsupported-format dimensions via broadcast joins only runs the **innermost** join natively. The outer join(s) stay on Spark, so acceleration is lost for all but the first. ## Example A native Parquet (or Iceberg) fact broadcast-joined to two Text dimensions: ```sql SELECT /*+ BROADCAST(d1), BROADCAST(d2) */ f.v FROM fact_parquet f JOIN dim1_text d1 ON f.k1 = d1.value JOIN dim2_text d2 ON f.k2 = d2.value Planned as (fact ⋈ dim1) ⋈ dim2: - Inner join fact ⋈ dim1: probe is fact (native) → dim1 is bridged → native CometBroadcastHashJoinExec. - Outer join (…) ⋈ dim2: its probe subtree still contains dim1's unconverted text FileSourceScanExec → stays on Spark. Root cause CometExecRule.tagBroadcastBuildSideLeaves runs once, before the main transformUp, and evaluates each broadcast join independently. For the outer join, tagBuildIfProbeNative calls hasOnlyNativeScans(probe), which walks the probe subtree and returns false as soon as it sees a plain Spark FileSourceScanExec/BatchScanExec. The inner join's dim1 scan is exactly such a scan at that point — it will be bridged (it's the inner join's tagged build side), but the pre-pass hasn't converted it and doesn't account for the pending bridge. So dim2's build is never tagged and the outer join is left on Spark. (The probe-native gate is deliberately conservative — it's what prevents bridging when the join can't go native, e.g. a non-native fact — but it currently can't tell a "will-be-bridged build-side scan" apart from a genuinely non-native one.) Expected Both joins run natively: dim1 and dim2 are both bridged and the fact is naee is a native broadcast join. Possible directions - In hasOnlyNativeScans, treat a FileSourceScanExec/BatchScanExec that is nested broadcast join (i.e. will be bridged) as effectively native, ratherthan failing on it. - Or iterate the pre-pass to a fixpoint / process joins bottom-up, so an i is visible when the outer join's probe is evaluated. - Either way, keep failing for genuinely non-native probe scans (a non-native fact must still leave the join on Spark, to preserve DPP broadcast reuse — see #6786). Scope Follow-up to #6786 (Option A of #6008); intentionally left out of that PR to keep it focused. Correctness is unaffected — this is a missed optimization (the outer join just runs on Spark, as it does today without the feature). ### Steps to reproduce _No response_ ### Expected behavior _No response_ ### Additional context _No response_ ### Willingness to contribute None -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
