andygrove opened a new issue, #6885: URL: https://github.com/apache/datafusion-comet/issues/6885
### What is the problem the feature request solves? The native Iceberg writer reserves what its open files and partition feeds hold between batches (#6247), but not the `DataFile`s of the files it has already closed, which it keeps until the task ends to build its output manifest. Each one holds the file's per-column sizes, counts and bounds. Measured by cloning one under an allocation counter, a `DataFile` takes about 1 KB for a 3-column table and 25 KB for a 101-column one. Rolled files have always had this gap. #6773 widens it, because a fanout write the pool refuses closes partitions early and ends with about (k + 1) / 2 files per partition at a pool 1/k of what it needs. In a 101-column write of 64 partitions given a quarter of the memory it needed, the 89 files closed early kept 2.2 MB of `DataFile`s until the task ended, about a fifth of its 10 MB pool, none of it reserved. Each close still freed far more than it kept, a median of 330 KB. ### Describe the potential solution Add an estimate for each kept `DataFile` to the reservation when its file closes, from its column count and the size of its bounds. Closing partitions can't give that memory back, so `InnerWriter::reserve` would have to leave it out of what it expects a close to free. A write could then fail on a refusal if the kept files alone outgrew the pool, which can't happen today because everything a fanout write reserves belongs to a partition it can close. ### Additional context Raised by @unikdahal in the review of #6773: https://github.com/apache/datafusion-comet/pull/6773#discussion_r4243573933 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
