Zouxxyy opened a new pull request, #11190:
URL: https://github.com/apache/paimon/pull/11190

   ### Purpose
   
   Enable `spark.paimon.write.use-v2-write` by default for supported batch 
writes and read its default directly from `SparkConnectorOptions`, removing the 
version-dependent default and blanket pre-3.4 disablement. Explicit V1 opt-out 
and existing table capability fallbacks remain available. Data source/path 
writes and streaming writes use V1.
   
   Spark 3.2/3.3 do not pass a function catalog when resolving V2 write 
distributions. Bind the equivalent distribution with Paimon's bucket expression 
before native write planning. Spark 3.2-3.4 table-as-select executors also 
bypass normal distribution preparation, so route their writes through append 
planning after table creation/staging, covering both `SparkCatalog` and 
`SparkGenericCatalog`. Preserve native creation/replacement logic and 
commit/abort callbacks; use named execution nodes whose constructors support 
Spark's reflective plan cloning.
   
   This keeps records for the same partition/bucket together. For example, a 
CTAS with one bucket and four input partitions now produces one data file 
instead of four.
   
   ### Tests
   
   - **210 tests passed** on Spark 3.2.4, 3.3.4, 3.4.4, and 3.5.9, covering 
option defaults/overrides, capability and path fallbacks, regular writes, 
CTAS/RTAS, AQE on/off, bucket file counts, and failed CTAS cleanup. Fourteen 
existing cases were skipped by their version guards.
   - **70 additional tests passed** on Spark 3.5.9: `DeltaRowLevelOpsTest`, 
`PaimonCopyOnWriteScanTest`, `MergeIntoAlignmentTest`, 
`NestedSubfieldMergeIntoTest`, and `V2WriteMergeSchemaTest`.
   - Spark 3 tests used Java 8 and the `spark3` Maven profile. Spotless 
formatting and Maven `validate` passed for the changed common, UT, and Spark 
3.2/3.3/3.4 modules; `git diff --check` also passed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to