rdblue commented on pull request #1362: URL: https://github.com/apache/iceberg/pull/1362#issuecomment-677960001
@RussellSpitzer, @aokolnychyi, if using the local iterator causes a job per task to be submitted to Spark, should we avoid using it? If every file to delete takes up 500 bytes in memory, then the driver can hold 4 million files in 2GB. That seems reasonable to me, so we may be over-optimizing by using the iterator instead of just collecting the data back. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
