JingsongLi opened a new pull request, #11127:
URL: https://github.com/apache/paimon/pull/11127

   ### Purpose
   
   Add an experimental `paimon-spark-4.2_2.13` connector through an independent 
`spark42` Maven profile. It compiles the shared connector modules against Spark 
4.2.0 / Scala 2.13.18 while keeping the existing `spark4` baseline and Spark 
4.0/4.1 build paths.
   
   This is a smaller alternative to #9265: it does not raise the shared 
baseline for all Spark 4.x artifacts. Compatibility changes cover output 
resolution, catalog commands, function parsing/registration, CHAR/VARCHAR 
construction, staged writes, scan matching, and Spark 4.2 BinaryView spatial 
getters. A small 4.2-only rule aligns fresh UPDATE/MERGE commands and delegates 
to Spark's native row-level rewriters.
   
   The profile has its own CI lane and a documented source-build command. 
Connector artifacts from different profiles must be built separately with 
`clean`, since the shared Maven artifact names are unchanged. Snapshot/release 
publication and semantic-view execution are outside this PR.
   
   ### Tests
   
   - `mvn -pl paimon-spark/paimon-spark-4.2 -am -Pspark42 -DfailIfNoTests=false 
-Dtest=none 
-DwildcardSuites=org.apache.paimon.spark.sql.Spark42CompatibilityTest package` 
after cleaning the Spark modules: passed without `fast-build`.
   - 15 focused compatibility tests: CTAS, static/dynamic partition overwrite, 
views, partition description, scalar SQL functions, V1/V2 UPDATE/DELETE/MERGE 
on primary-key, ordinary append, row-tracking, deletion-vector and Data 
Evolution tables, subquery UPDATE, insert-only MERGE, VARIANT and spatial 
scalar/array reads.
   - Repeated the same 15 tests with the shaded connector JAR first on the 
runtime classpath: passed.
   - `python3 -m unittest discover -s tools/ci -p 'test_*.py'`: 30 passed.
   - Root Apache RAT check and `git diff --check`: passed.
   
   This initial source-build PR has focused compatibility coverage; the 
complete Spark engine test suite has not been run.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to