milenkovicm commented on PR #1679: URL: https://github.com/apache/datafusion-python/pull/1679#issuecomment-5375229711
Following up on comment https://github.com/apache/datafusion-ballista/pull/2252#pullrequestreview-4981918462 and follow up on this PR. `FFILogicalCodec::encode|decode_file_format` break our intent to have fully working distributed execution. I might be wrong but `encode|decode_file_format` might not be an easy fix, at least not in short term and not in datafusion 55 timeframe, hence i have a proposal to make. `DistributedExec` in ballista is nothing but a GRPC wrapper, it takes a logical plan, calls grpc endpoint and returns a stream of record batches. Would it make sense to create a `CallbackPlanner` (we might need a `CallbackExec`) in datafusion-py which would take a python closure `LogicalPlan -> Stream<RecordBatches>` (or `LogicalPlanBlob -> Stream<RecordBatches>`). Ballista would provide a closure which implements grcp logic in python code. As `CallbackPlanner` have same library marker as df python `FFILogicalCodec` will not be triggered and we should have possibility to fully integrate distributed execution. Basically we could implement query planner in python (limited but working) wdyt @timsaucer and @ntjohnson1 ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
