gabotechs commented on issue #21650: URL: https://github.com/apache/datafusion/issues/21650#issuecomment-4255160119
In general, not baking shared state into execution plans is greatly beneficial to any system the relies on plan [de]serialization to work, for example FFI-based wrappers or distributed engines (https://github.com/apache/datafusion-ballista or https://github.com/datafusion-contrib/datafusion-distributed). Indeed, one of the main challenges for integrating dynamic filters with a distributed system is the collision with the assumption of the fact that data can simply be shared through `Arc`-based shared references. I actually gave it a try some time ago in https://github.com/gabotechs/datafusion/pull/7, but abandoned the project as a https://github.com/apache/datafusion/pull/18739 was a prerequisite but was not very well received. I can imagine how not relying on shared state inside can add some complexity if not done with care, but IMO it's something worth pursuing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
