LuciferYang opened a new issue, #10918: URL: https://github.com/apache/paimon/issues/10918
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master (b4342332ea) ### Compute Engine Spark ### Minimal reproduce step ```sql CREATE TABLE t (id INT, b STRING, c INT) TBLPROPERTIES ( 'row-tracking.enabled' = 'true', 'data-evolution.enabled' = 'true', 'write-buffer-for-append' = 'true', 'write-buffer-size' = '256 kb', 'page-size' = '64 kb' ); INSERT INTO t SELECT /*+ REPARTITION(1) */ CAST(id AS INT), CAST(id AS STRING), CAST(id AS INT) FROM range(0, 20000); CREATE TABLE s (id INT, b STRING); INSERT INTO s SELECT CAST(id AS INT), repeat('x', 100) FROM range(0, 20000); MERGE INTO t USING s ON t.id = s.id WHEN MATCHED THEN UPDATE SET t.b = s.b; ``` The small `write-buffer-size` only makes the buffer spill quickly; with the default size the same happens once the updated columns of one file exceed 256 MB. ### What doesn't meet your expectations? The `MERGE INTO` fails: ``` java.lang.NullPointerException at org.apache.paimon.disk.ExternalBuffer.spill(ExternalBuffer.java:148) at org.apache.paimon.disk.ExternalBuffer.put(ExternalBuffer.java:122) ``` `DataEvolutionTableDataWrite`, which writes the updated columns of a data evolution table for `MERGE INTO` and `UPDATE`, creates an `IOManager` but never passes it to the write. With `write-buffer-for-append = true` the append writer uses a spillable buffer (`write-buffer-spillable` is true by default), and the first spill calls `createChannel()` on a null `IOManager`. `PaimonDataWrite` and `PaimonV2DataWriter` pass their `IOManager` with `withIOManager`. ### Anything else? _No response_ ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
