AveryQi115 opened a new pull request, #57091: URL: https://github.com/apache/spark/pull/57091
### What changes were proposed in this pull request? This PR adds a `forceSkipInline` field to `CTERelationDef`. When set to true, `InlineCTE` never inlines the CTE relation, even in `alwaysInline` mode or when the CTE is deterministic and referenced only once. This lets a producer guarantee that a CTE is materialized rather than duplicated. The flag is self-gating: it defaults to false, so there is no behavior change unless a caller explicitly sets it. Because a force-materialized CTE is evaluated on its own, it cannot carry an outer reference across its boundary (there is no surrounding operator to resolve it against after materialization). `InlineCTE` now validates this invariant and throws an `INTERNAL_ERROR` with a descriptive message when it is violated, for both direct `OuterReference`s and subqueries with outer-scope references. `PushdownPredicatesAndPruneColumnsForCTEDef` is updated to preserve the new field via `copy` instead of reconstructing `CTERelationDef`, and `CTERelationDef.stringArgs` keeps the default string representation stable when the flag is not set. ### Why are the changes needed? Previously the default operation for CTE in spark is to inline it. However, there are many internal features requiring to generate a materialized CTE which is guaranteed to be executed only once. For example, DSv2 command rewrites (e.g. MERGE INTO) can wrap a source in a CTE to avoid duplicating it. For correctness, if such CTE contains non-deterministic operation, such a CTE must be materialized and never inlined. There is currently no way to express that requirement explicitly on `CTERelationDef`. Besides, we plan to reuse CTE results by introducing a shuffle and reuse the shuffle files. For usecases like DSv2 commands, the best CTE partitioning is already defined by the consumers. (In DSv2 case, the partitioning is the join keys for the join between source and target.) Providing a field for the suggested partitioning for the materialized CTE is necessary. ### Does this PR introduce any user-facing change? No. The new field defaults to false and is internal only. ### How was this patch tested? New unit tests in `InlineCTESuite` covering: the flag keeps a single-reference deterministic CTE materialized, the same CTE is inlined when the flag is unset, and validation fails with `INTERNAL_ERROR` when a force-materialized CTE carries an outer reference across its boundary. ### Was this patch authored or co-authored using generative AI tooling? <!-- If generative AI tooling has been used in the process of authoring this patch, please include the phrase: 'Generated-by: ' followed by the name of the tool and its version. If no, write 'No'. Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details. --> Generated-by: claude-code-Opus 4.8 (1M context) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
