alamb commented on issue #23174: URL: https://github.com/apache/datafusion/issues/23174#issuecomment-4817077004
> My concern is that, for distributed use cases, shared-state parallelism usually isn't easily applicable, This is an excellent point -- if we want to use the same paralleization model for both local (cross core) and distributed (cross node) it would be much harder (e.g. internally the ExecutionPlan nodes would have to communicate acros nodes somehow or something) which sounds pretty complicated. On the other hand, maybe it is ok if we had two different approaches for cross core parallelism and cross node parallelism as the constraints are somewhat different -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
