hcrosse opened a new issue, #2671: URL: https://github.com/apache/iceberg-go/issues/2671
### Apache Iceberg version main (development) ### Please describe the bug 🐞 A partitioned `Append` can return nil and commit without the rows of a data file whose upload failed. I reproduced this on main and v0.7.0, with Go 1.26.10 on macOS arm64. It failed in 10 of 10 runs on each version. The reproduction uses the SQL catalog on SQLite and a local warehouse. One partition receives five batches of five rows, 200 ms apart. The target file size is one byte, so each batch goes to its own data file. A FileIO wrapper makes closing the second data file return `upload failed` and remove the file, as an object store upload that fails on close would: ```text failed uploads: 1; Append returned <nil>; rows missing: [105 106 107 108 109] Append returned nil but the table is missing 5 of 25 rows ``` Expected: `Append` returns an error and commits nothing. With an unpartitioned table, the same setup fails with `writer is closed`. **Cause** A failed writer removes itself from the factory when its `stream` goroutine exits: https://github.com/apache/iceberg-go/blob/74b684cf085355bfddba0d2c74743bacd5d2eae3/table/rolling_data_writer.go#L442-L445 `closeAll` closes only the writers still in the factory: https://github.com/apache/iceberg-go/blob/74b684cf085355bfddba0d2c74743bacd5d2eae3/table/rolling_data_writer.go#L819-L851 If the failed writer leaves before another batch reaches it through `Add`, and before `closeAll` runs at the end of the input, nothing reports its error. The 200 ms pauses give it that time, as a streaming source might. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
