RussellSpitzer commented on pull request #1362: URL: https://github.com/apache/iceberg/pull/1362#issuecomment-677961397
That was my original thought, but I think we wanted to air on the side of caution. I think it would be fine if we just made it configurable. Add a switch for "lazy" execution? On Thu, Aug 20, 2020 at 6:45 PM Ryan Blue <[email protected]> wrote: > @RussellSpitzer <https://github.com/RussellSpitzer>, @aokolnychyi > <https://github.com/aokolnychyi>, if using the local iterator causes a > job per task to be submitted to Spark, should we avoid using it? > > If every file to delete takes up 500 bytes in memory, then the driver can > hold 4 million files in 2GB. That seems reasonable to me, so we may be > over-optimizing by using the iterator instead of just collecting the data > back. > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > <https://github.com/apache/iceberg/pull/1362#issuecomment-677960001>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AADE2YJS5YZZKAHDXL4ZLSDSBWYQ3ANCNFSM4QGVJG7Q> > . > ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
