gabotechs commented on issue #1612: URL: https://github.com/apache/datafusion-python/issues/1612#issuecomment-4892969915
Regarding `datafusion-distributed`, one thing that can be challenging is to bridge all the customization options in the Python world. TBH, I'm not sure which of the two options would make it easier: - [Customize worker resolution](https://datafusion-contrib.github.io/datafusion-distributed/user-guide/02-worker-resolver.html) (done easily in https://github.com/apache/datafusion-python/pull/1611/changes#diff-ab4c8cde88a6ba7347571f7fc56080db0c4f5a59ac737e15bb8c87d3094fa8daR146-R161, I bet this should also be relatively easy with Option 2) - [Customize worker connections](https://datafusion-contrib.github.io/datafusion-distributed/user-guide/06-channel-resolver.html) (I don't think this one can even be done in any option...) - [Propagating arbitrary headers through worker jumps](https://datafusion-contrib.github.io/datafusion-distributed/advanced/01-passthrough-headers.html) - [Propagating custom ConfigOptions to workers](https://datafusion-contrib.github.io/datafusion-distributed/advanced/02-config-extensions.html) - [User defined work unit feeds](https://datafusion-contrib.github.io/datafusion-distributed/advanced/04-work-unit-feeds.html) - [Custom worker routing for cache affinity and stuff](https://datafusion-contrib.github.io/datafusion-distributed/advanced/06-worker-routing.html) - [Users injecting network boundary nodes manually in the plan](https://datafusion-contrib.github.io/datafusion-distributed/advanced/05-custom-distributed-plans.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
