hcrosse opened a new issue, #2671:
URL: https://github.com/apache/iceberg-go/issues/2671

   ### Apache Iceberg version
   
   main (development)
   
   ### Please describe the bug 🐞
   
   A partitioned `Append` can return nil and commit without the rows of a data 
file whose upload failed.
   
   I reproduced this on main and v0.7.0, with Go 1.26.10 on macOS arm64. It 
failed in 10 of 10 runs on each version.
   
   The reproduction uses the SQL catalog on SQLite and a local warehouse. One 
partition receives five batches of five rows, 200 ms apart. The target file 
size is one byte, so each batch goes to its own data file. A FileIO wrapper 
makes closing the second data file return `upload failed` and remove the file, 
as an object store upload that fails on close would:
   
   ```text
   failed uploads: 1; Append returned <nil>; rows missing: [105 106 107 108 109]
   Append returned nil but the table is missing 5 of 25 rows
   ```
   
   Expected: `Append` returns an error and commits nothing. With an 
unpartitioned table, the same setup fails with `writer is closed`.
   
   **Cause**
   
   A failed writer removes itself from the factory when its `stream` goroutine 
exits:
   
   
https://github.com/apache/iceberg-go/blob/74b684cf085355bfddba0d2c74743bacd5d2eae3/table/rolling_data_writer.go#L442-L445
   
   `closeAll` closes only the writers still in the factory:
   
   
https://github.com/apache/iceberg-go/blob/74b684cf085355bfddba0d2c74743bacd5d2eae3/table/rolling_data_writer.go#L819-L851
   
   If the failed writer leaves before another batch reaches it through `Add`, 
and before `closeAll` runs at the end of the input, nothing reports its error. 
The 200 ms pauses give it that time, as a streaming source might.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to