kbendick commented on PR #4596: URL: https://github.com/apache/iceberg/pull/4596#issuecomment-1163915875
> Let me be clear. In my scene, my machine has 96 cores, when `iceberg.scan.plan-in-worker-pool` is enabled, the `iceberg.worker.num-threads` will be number of available processors in default, i.e. 96. So the input speed of queue is far greater than outpu speed of queue. Then OOM occurs. Forgive me if I missed this as I’m just coming back to it, but what happens if you lower the core count in iceberg.sc.plan-in-worker-pool via iceberg.worker.num-threads? I’ve heard of other issues with machines with very large number of cores. Maybe, possibly in addition to this, we might consider a configurable max limit and default it to something sane? The other report was from a Spark driver iirc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
