LuciferYang opened a new pull request, #10944:
URL: https://github.com/apache/paimon/pull/10944

   ### Purpose
   
   close #10942
   
   For a partitioned format table read through the engine implementation, only 
SQL `INSERT INTO` and `INSERT OVERWRITE` are rewritten to Spark's V1 file 
write. DataFrame writes and RTAS go through Spark's V2 `FileWrite`, which 
writes no partition directories, so the rows landed in the table root and were 
never read back.
   
   - `PartitionedFormatTable` now rejects V2 writes to a partitioned table with 
an `UnsupportedOperationException` that points to `format-table.implementation 
= 'paimon'`. Non-partitioned engine tables are unchanged.
   - RTAS into a partitioned engine table is rejected while planning, before 
the old table is replaced, with the check CTAS already uses. Spark 3.2 and 3.3 
plan RTAS with Spark's own strategy, so their 
`PaimonReplaceTableAsSelectStrategy` now runs the check and still returns `Nil`.
   
   ### Tests
   
   `FormatTableTestBase` "DataFrame writes to a partitioned engine table are 
rejected": `writeTo(...).append()`, `writeTo(...).overwrite(...)`, `insertInto` 
and `insertInto` in overwrite mode are rejected and leave the table unchanged; 
SQL `INSERT` still works; `CREATE OR REPLACE TABLE ... AS SELECT` and 
`saveAsTable` in overwrite mode are rejected and keep the old rows. On Spark 
3.4+ it also checks that V2 writes and RTAS on a non-partitioned engine table 
still work. On master the first write succeeds and its row is lost.
   
   ### API and Format
   
   no
   
   ### Documentation
   
   no
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to