Stephen0421 opened a new pull request, #11122: URL: https://github.com/apache/paimon/pull/11122
### Purpose Fixes #11121 An ordinary V1 `INSERT INTO` can fail on Spark 3.2.4 when its input contains dynamic partition pruning (DPP). A query with two chained `LEFT JOIN`s over partitioned Parquet inputs reproduces an `AnalysisException: unresolved operator 'Filter dynamicpruning...` when writing to a Paimon primary-key table. The same insert succeeds with DPP disabled. The V1 callback in `PaimonWrite.toInsertableRelation` passes an already optimized input plan to `WriteIntoPaimonTable`. Writer-side Dataset transformations such as `withColumn` then trigger reanalysis of the plan containing DPP subqueries. Reuse `PaimonUtils.createNewDataFrame` at the V1 callback boundary to create an RDD-backed DataFrame from the input query execution and schema before invoking the writer. This prevents writer-side transformations from reanalyzing the optimized source plan. It applies the existing approach used by [#8052](https://github.com/apache/paimon/pull/8052) for dynamic partition overwrite to the ordinary V1 insert path. ### Tests Added `DynamicPartitionPruningWriteTestBase` with concrete suites for Spark 3.2 and 3.5. Each suite covers eight combinations of: - Dynamic and fixed buckets (`bucket=-1` and `bucket=2`). - AQE enabled and disabled. - DPP enabled and disabled. Each case checks the presence or absence of DPP in the optimized input plan, verifies the initial insert and a subsequent insert updating the same primary keys, and checks `NULL` values for unmatched right-side rows. Validation: - Reproduced the failure on the unmodified baseline with Apache Spark 3.2.4 and DPP enabled; the DPP-disabled control passed. - With the fix: Spark 3.2.4 — 8 passed; Spark 3.5.9 — 8 passed. - Maven packaging and the configured checks passed without the `fast-build` profile. `git diff --check` also passed. The Spark 3.5.9 results validate compatibility of the fix; the original failure was reproduced on Spark 3.2.4. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
