JustinBinber opened a new pull request, #11339:
URL: https://github.com/apache/paimon/pull/11339

   Fixes #11338
   
   ### Purpose
   
   An UPDATE + INSERT MERGE immediately after `ALTER TABLE ADD COLUMNS` can 
silently drop values for the new columns on inserted rows. The target scan must 
stay on one data snapshot for conflict detection. Its table copy was also 
replacing the current logical schema with the older snapshot schema, and the 
INSERT writer inherited that schema.
   
   Keep the current schema when pinning the data snapshot in the Spark MERGE 
command and when copying write options for the partial-column writer. This 
keeps the snapshot used for reads and conflict checks while allowing matched 
UPDATE and unmatched INSERT to write the new column.
   
   ### Tests
   
   - New regression creates a data snapshot, adds a nullable column, then uses 
one MERGE to write that column on matched and unmatched rows. It also checks an 
untouched row and runs with join reuse enabled and disabled.
   - The Spark 3.5.9 / Scala 2.12 / JDK 8 Maven reactor passed all four tests 
in `DataEvolutionUpdateSnapshotTest` and 
`DataEvolutionMergeJoinReuseConflictTest`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to