Dandandan commented on pull request #972: URL: https://github.com/apache/arrow-datafusion/pull/972#issuecomment-917704920
I am not sure if this is the way we should go. I think this should be a setting which could be set on the session to **limit** the number of partitions in each execution node. This property should be used throughout the code, instead of using the number of CPU cores. It's OK to use the number of CPU cores by default in DataFusion (as is the case currently) but it should be possible to override this, e.g. from Ballista. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
