Stephen0421 opened a new pull request, #11122:
URL: https://github.com/apache/paimon/pull/11122

   ### Purpose
   
   Fixes #11121 
   
   An ordinary V1 `INSERT INTO` can fail on Spark 3.2.4 when its input contains 
dynamic partition pruning (DPP). A query with two chained `LEFT JOIN`s over 
partitioned Parquet inputs reproduces an `AnalysisException: unresolved 
operator 'Filter dynamicpruning...` when writing to a Paimon primary-key table. 
The same insert succeeds with DPP disabled.
   
   The V1 callback in `PaimonWrite.toInsertableRelation` passes an already 
optimized input plan to `WriteIntoPaimonTable`. Writer-side Dataset 
transformations such as `withColumn` then trigger reanalysis of the plan 
containing DPP subqueries.
   
   Reuse `PaimonUtils.createNewDataFrame` at the V1 callback boundary to create 
an RDD-backed DataFrame from the input query execution and schema before 
invoking the writer. This prevents writer-side transformations from reanalyzing 
the optimized source plan. It applies the existing approach used by 
[#8052](https://github.com/apache/paimon/pull/8052) for dynamic partition 
overwrite to the ordinary V1 insert path.
   
   ### Tests
   
   Added `DynamicPartitionPruningWriteTestBase` with concrete suites for Spark 
3.2 and 3.5. Each suite covers eight combinations of:
   
   - Dynamic and fixed buckets (`bucket=-1` and `bucket=2`).
   - AQE enabled and disabled.
   - DPP enabled and disabled.
   
   Each case checks the presence or absence of DPP in the optimized input plan, 
verifies the initial insert and a subsequent insert updating the same primary 
keys, and checks `NULL` values for unmatched right-side rows.
   
   Validation:
   
   - Reproduced the failure on the unmodified baseline with Apache Spark 3.2.4 
and DPP enabled; the DPP-disabled control passed.
   - With the fix: Spark 3.2.4 — 8 passed; Spark 3.5.9 — 8 passed.
   - Maven packaging and the configured checks passed without the `fast-build` 
profile. `git diff --check` also passed.
   
   The Spark 3.5.9 results validate compatibility of the fix; the original 
failure was reproduced on Spark 3.2.4.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to