davseitsev commented on issue #3179: URL: https://github.com/apache/iceberg/issues/3179#issuecomment-2220507710
We have the same issue. It caused a lot of OOM problems when we had an issue with data compaction. Here is an example of the smallest heap dump we took:  As you can see, `ParallelIterator#queue` takes 21GB, it's because it contains `313 573` items. Some dumps contain more than `1M` of `org.apache.iceberg.BaseFileScanTask` items (for multimple queries):  A single call to `ParallelIterator.checkTasks()` submits 96 processing tasks and it's enough to cause back pressure and memory starvation. Maybe we can implement kind of slow-start algorithms, like `checkTasks()` can submit only 1 new task, and if the consumer is fast enough, `ParallelIterator#queue` will become empty and another `checkTasks()` call will submit additional task, etc. Maybe it worths to have separate config for the size of `ParallelIterator.taskFutures` to limit max number of producers for a single query but not reduce the worker pool size. Or maybe we can limit queue size and use `ForkJoinPool#managedBlock` not to block completely parallel flows. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
