ghoshp83 commented on PR #4146: URL: https://github.com/apache/iceberg-python/pull/4146#issuecomment-6382179705
Thanks for looking. Fair concern, so I measured it. The setup was one Linux machine with local ext4, pyarrow 25.0.1, comparing this branch against the `open_output_stream` on main. I ran each case 200 times (20 for 10 MB) and repeated the whole run on two disks. Per file, write + close, median: | size | main | this PR | |---|---|---| | 1 KB | 0.16 ms | 3–4 ms | | 100 KB | 0.5 ms | 3.5–5 ms | | 10 MB | 17–29 ms | 38–45 ms | The first file written into a new directory pays one more directory fsync, about +3 ms. End to end, I timed `table.append()` of 1k rows through `SqlCatalog`: a data file, manifest, manifest list and metadata.json per commit, 30 commits. The median went from 127 → 161 ms and 153 → 187 ms on one disk. On the other it was 178 → 174 ms, which is within noise. So the cost is a roughly fixed ~3–5 ms per file. It only applies to `file://` warehouses, since object stores don't go through this path. A few ms is small next to a commit, but it is real for workloads that write many small files. If you'd rather not pay that by default, I'm happy to put it behind a property. That could be off by default, so local users opt in to durability, or on with an opt-out. Which would you prefer? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
