Zouxxyy opened a new pull request, #11190: URL: https://github.com/apache/paimon/pull/11190
### Purpose Enable `spark.paimon.write.use-v2-write` by default for supported batch writes and read its default directly from `SparkConnectorOptions`, removing the version-dependent default and blanket pre-3.4 disablement. Explicit V1 opt-out and existing table capability fallbacks remain available. Data source/path writes and streaming writes use V1. Spark 3.2/3.3 do not pass a function catalog when resolving V2 write distributions. Bind the equivalent distribution with Paimon's bucket expression before native write planning. Spark 3.2-3.4 table-as-select executors also bypass normal distribution preparation, so route their writes through append planning after table creation/staging, covering both `SparkCatalog` and `SparkGenericCatalog`. Preserve native creation/replacement logic and commit/abort callbacks; use named execution nodes whose constructors support Spark's reflective plan cloning. This keeps records for the same partition/bucket together. For example, a CTAS with one bucket and four input partitions now produces one data file instead of four. ### Tests - **210 tests passed** on Spark 3.2.4, 3.3.4, 3.4.4, and 3.5.9, covering option defaults/overrides, capability and path fallbacks, regular writes, CTAS/RTAS, AQE on/off, bucket file counts, and failed CTAS cleanup. Fourteen existing cases were skipped by their version guards. - **70 additional tests passed** on Spark 3.5.9: `DeltaRowLevelOpsTest`, `PaimonCopyOnWriteScanTest`, `MergeIntoAlignmentTest`, `NestedSubfieldMergeIntoTest`, and `V2WriteMergeSchemaTest`. - Spark 3 tests used Java 8 and the `spark3` Maven profile. Spotless formatting and Maven `validate` passed for the changed common, UT, and Spark 3.2/3.3/3.4 modules; `git diff --check` also passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
