JustinBinber opened a new issue, #11338:
URL: https://github.com/apache/paimon/issues/11338

   ### Paimon version
   
   The MERGE command from Apache master at 
`ad4f6e09ef51c6b9932cbf22f0be563c6dde683f` reproduces this on a compatible 
Spark SDK. I also reproduced it in the source RowID experiment with the 
optimization both enabled and disabled.
   
   ### Compute engine
   
   Spark 3.5.9, Paimon Spark SQL extensions and catalog.
   
   ### Minimal reproduction
   
   ```sql
   CREATE TABLE paimon.default.t (id BIGINT, value STRING)
   TBLPROPERTIES (
     'bucket' = '-1',
     'row-tracking.enabled' = 'true',
     'data-evolution.enabled' = 'true',
     'data-evolution.merge-into.join-reuse' = 'false'
   );
   INSERT INTO paimon.default.t VALUES (1, 'old'), (3, 'untouched');
   ALTER TABLE paimon.default.t ADD COLUMNS (payload STRING);
   
   MERGE INTO paimon.default.t AS t
   USING (SELECT * FROM VALUES (1L, 'updated', 'matched'), (2L, 'inserted', 
'kept')
          AS v(id, value, payload)) AS s
   ON t.id = s.id
   WHEN MATCHED THEN UPDATE SET t.value = s.value
   WHEN NOT MATCHED THEN INSERT (id, value, payload) VALUES (s.id, s.value, 
s.payload);
   
   SELECT id, value, payload FROM paimon.default.t ORDER BY id;
   ```
   
   ### Expected and actual behavior
   
   The MERGE succeeds, but the new row is `(2, 'inserted', NULL)`. It should be 
`(2, 'inserted', 'kept')`. The old rows should still have NULL payloads because 
MERGE leaves that column untouched on those rows.
   
   If the MATCHED clause also sets `t.payload = s.payload`, the UPDATE writer 
needs the new column too. Preserving the current schema for the pinned scan 
alone is insufficient: the partial-column writer's options copy resolves the 
old snapshot a second time.
   
   ### Cause
   
   The command fixes the target scan to the last data snapshot. The table copy 
used for that scan also adopts the schema from the snapshot, which predates the 
ALTER. The INSERT writer then takes its schema from that copy and writes a 
Parquet file without the new field. The write completes without an error.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to