weiqingy opened a new pull request, #2415: URL: https://github.com/apache/auron/pull/2415
# Which issue does this PR close? Closes #2329 # Rationale for this change #1865 moved Calc-source fusion to a graph-level pass (`AuronOperatorFusionProcessor`) that counts each source's consumers and fuses only sole-consumer sources. A source feeding more than one Calc (for example `UNION ALL` over one table under Flink's default source reuse) is declined at the sole-consumer gate, and each Calc runs as a standalone native operator over the shared source stream. That multi-consumer path is correct but was under-tested and under-documented. The existing shared-source test only covered the default config, where object reuse is effectively off so Flink deep-copies each row to heap per consumer. The interesting case is object reuse ON: Flink then hands the same `AuronColumnarRowData` reference to both Calc consumers with no defensive copy in between, which is exactly where a fan-out aliasing or use-after-free bug would surface. Nothing pinned that. And the reason a multi-consumer source declines fusion was only a one-line comment. # What changes are included in this PR? A new end-to-end test `testSharedSourceUnionAllFanOutSafeWithObjectReuse` in `AuronKafkaSourceMergeITCase`: the same shared-source `UNION ALL` as the existing test, but with object reuse enabled. It asserts the row set stays correct and both Calcs remain standalone operators (the multi-consumer source is not fused). An expanded doc-comment at the sole-consumer gate in `AuronOperatorFusionProcessor` explaining why a multi-consumer source is not fused (the native source runtime is single-plan and single-output, so N per-consumer plans cannot share one source) and why declining is safe (each consumer runs a standalone native Calc that copies every column out of the shared columnar view). No production logic changes. # Are there any user-facing changes? No. # How was this patch tested? `./build/mvn test -Pspark-3.5 -Pscala-2.12 -Pflink-1.18 -pl auron-flink-extension/auron-flink-planner -am -Dtest=AuronKafkaSourceMergeITCase` with the native library built. All 4 tests pass. # Was this patch authored or co-authored using generative AI tooling? - [x] Yes - [ ] No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
