andygrove commented on issue #47: URL: https://github.com/apache/arrow-rs/issues/47#issuecomment-901914641
That's right. Ballista avoids this issue by limiting the number of concurrent tasks. However, Ballista has its own related issues where it will generate an excessive number of shuffle files and potentially run into inode limits, so neither solution is as scalable as we would like yet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
