venkata91 opened a new pull request, #28148: URL: https://github.com/apache/flink/pull/28148
## What is the purpose of the change Fixes [FLINK-39669](https://issues.apache.org/jira/browse/FLINK-39669). `BatchPhysicalCorrelateRule.convertToCorrelate` and `StreamPhysicalCorrelateRule.convertToCorrelate` walk through a `FlinkLogicalCalc` between the `Correlate` and the underlying `FlinkLogicalTableFunctionScan` to extract the Calc's condition, but silently discard the Calc's projection list. The physical Correlate is then constructed with a narrower output rowType while `CorrelateCodeGenerator` concatenates the left input with the full TFS output positionally — so downstream consumers read the wrong TFS column. The bug is dormant in mainline Flink because no production rule produces the `Correlate(left, Calc(non-identity-projection) over TFS)` shape today. It surfaces when Calcite's stock `ProjectCorrelateTransposeRule` is enabled, or when Flink-specific rules push a non-identity projection above a TFS (e.g., extending FLINK-32940 to prune the right side of UNNEST's Correlate). ## Brief change log - When the Calc above the TFS has a non-identity projection (`!program.projectsOnlyIdentity()`) and the join type is `INNER` or `LEFT`, build the physical Correlate with the full combined rowType (`left ++ scan.rowType`) and apply the projection via a wrapping `BatchPhysicalCalc` / `StreamPhysicalCalc`. Identity-projection cases (today's common path: Calc with only a condition) are unchanged. SEMI/ANTI Correlates output only the left fields, so the bug cannot manifest there. - Updates two plan-test goldens (`testUnnestWithOrdinalityInSubquery`, batch + stream) that previously locked in the buggy plan folding the projection into the Correlate's claimed rowType. ## Verifying this change - 175 UNNEST tests (plan + IT, batch + stream) pass. - 198 `*Correlate*` tests (incl. `AsyncCorrelate`, `PythonCorrelate`, restore tests) pass. - The plan-test golden update for `testUnnestWithOrdinalityInSubquery` demonstrates the fix: the new plan correctly applies the projection (renaming + literal substitution from `WHERE ordinality = 1`) via a wrapping Calc rather than silently merging it into the Correlate's rowType. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API: **no** — planner-internal only. - The serializers: **no** - The runtime per-record code paths: **no** — physical operators unchanged. - Anything that affects deployment or recovery: **no** - The S3 file system connector: **no** ## Documentation - Does this pull request introduce a new feature? **no** — bug fix. ##### Was generative AI tooling used to co-author this PR? - [X] Yes (Claude Code, Anthropic Claude Opus 4.7) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
