rdblue commented on issue #1178: URL: https://github.com/apache/iceberg/issues/1178#issuecomment-655643335
I don't think we want to use `deleteOnExit`. That just queues the delete to happen in a shutdown hook, but I don't think there are many guarantees about reliability. That would make it so we no longer retry failed operations and might affect recovery when some, but not all of them fail. It also wouldn't help with parallelism because it would happen in a single shutdown hook thread. I think the best way to parallelize this operation is to run in a thread pool passed in by the user. Then the user could configure the parallelism of that pool to ensure that it is not so high that it hits throttling exceptions. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
