RussellSpitzer commented on issue #3703: URL: https://github.com/apache/iceberg/issues/3703#issuecomment-991101986
We have considered this but we were worried about overwhelming distributed file systems and losing control over the parallelism of the job. That said have you tried just increasing driver memory, we have folks doing some rather large remove orphan and expire snapshot jobs and they just use larger drivers when required (they also usually increase the parallelism of the driver delete process as well.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
