LuciferYang opened a new pull request, #59217: URL: https://github.com/apache/spark/pull/59217
### What changes were proposed in this pull request? `RewriteWithExpression` pre-evaluates a common expression's definition that is referenced more than once and is not cheap in a `Project` over the first child whose output covers the definition's references. That assumes the operator evaluates its expressions over its children's rows, and a command does not. A CHECK constraint is stored as a V2 predicate built from the optimized expression, and a group-based DELETE pushes its condition down to the scan and reuses it for runtime group filtering, so after the rewrite these read a `_common_expr_N` column that only the added `Project` produces. CREATE TABLE and a DELETE on a table that deletes by filters are planned only over a `ResolvedIdentifier` or a V2 scan relation, and find the `Project` there instead. This adds a `Command` case to the rule that substitutes every deterministic definition at its references. Analysis rejects a nondeterministic expression in every command except `CreateVariable`, and a nondeterministic definition that did reach the case would keep its `With` rather than be read twice. A `With` in a conditional branch of a command is inlined as well, since a `With` translates to no source predicate. `CreateVariable` is left out: its default is evaluated rather than stored, so a definition there should be memoized, not inlined. SPARK-59954 (#59209) fixes it separately. `inlineDefsThatGainNothing` is split so the new case reuses its substitution with a different selection. ### Why are the changes needed? With default settings, NULLIF or BETWEEN over a first operand that is not cheap breaks these commands: ```sql -- InMemoryTableCatalog as testcat CREATE TABLE testcat.g (a INT, b INT GENERATED ALWAYS AS (nullif(a + 1, 3))); -- [INTERNAL_ERROR] ... No plan for CreateTable ... -- +- Project [a, b, (a + 1) AS _common_expr_0] -- +- ResolvedIdentifier ..., g, [a, b] DELETE FROM testcat.t WHERE (a + b) BETWEEN 0 AND 4; -- [INTERNAL_ERROR] Unexpected table relation: Project [a, b, (a + b) AS _common_expr_1] ... -- group-based row-level table partitioned by dep DELETE FROM rowcat.p WHERE (id + pk) BETWEEN 2 AND 4; -- [INTERNAL_ERROR_ATTRIBUTE_NOT_FOUND] Could not find _common_expr_4 in [dep] ALTER TABLE testcat.t ADD CONSTRAINT c1 CHECK (nullif(a + b, 3) > 0); -- succeeds, but the V2 predicate stored in the catalog is -- CASE WHEN _common_expr_1 = 3 THEN NULL ELSE _common_expr_1 > 0 END ``` ### Does this PR introduce _any_ user-facing change? Yes. The CREATE TABLE above succeeds and INSERT computes the generated column. The DELETE on `testcat.t` reports that the table cannot delete by `[(a + b) >= 0, (a + b) <= 4]` instead of an internal error, so a source that accepts those predicates can run it. The group-based DELETE deletes the matching rows. The stored CHECK predicate refers to the table's columns. A definition referenced twice in the expression of any command but DECLARE VARIABLE now appears at both references, as it does with `spark.sql.alwaysInlineCommonExpr=true`. ### How was this patch tested? - `RewriteWithExpressionSuite`: a DELETE condition holding a `With`, directly and in a conditional branch, comes out inlined with no `Project` under the command, and a nondeterministic definition keeps its `With`. - `DataSourceV2SQLSuiteV1Filter`: `DELETE ... WHERE (id + p) BETWEEN 0 AND 4` on a table that cannot delete by it fails with `Cannot delete from table ... where [...]` listing both comparisons over the table's columns, and deletes nothing. - `DeleteFromTableSuiteBase`: the same shape of condition deletes the matching rows on the group-based and delta-based row-level tables. - `GeneratedColumnWriteSuite`: a table with `c INT GENERATED ALWAYS AS (nullif(a + b, 3))` is created, and INSERT computes `c`. - `CheckConstraintSuite`: the stored V2 predicate of `CHECK (nullif(id + 1, 3) > 0)` refers to `id` and nothing else. All of them fail without the change except the two delta-based DELETE runs, which were not affected. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
