ryansun96 commented on issue #1354: URL: https://github.com/apache/datafusion-python/issues/1354#issuecomment-3814642156
> final use case will not have datafusion-python in the loop as a **Python** dependency at all, correct? That is correct. One detail that I did not mention in my previous post but seems to be significant based on the proposed approaches, is that in our project, we do not need to send the rust processing results back to Python, which makes Approach 2 much simpler. With that I do think Approach 3 has value outside of our use case right now. To check my understanding, approach 3 is essentially using a table provider as a "pipe" operator that embeds custom logic, rather than simply reading & returning rows? > does all the operations you want under the hood using its own session context (not the datafusion-python session context) Does it imply the rule of thumb that "If I want to use functions from different libraries in the same session context, those libraries need to be compiled together"? I.e. even if use datafusion-python as a **py** dependency, as long as we depend on source, not the artifact published to PyPI, we will be fine even if we do data transfer "naively"? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
