d80tb7 commented on issue #24965: [WIP][SPARK-27463][PYTHON] Support Dataframe Cogroup via Pandas UDFs URL: https://github.com/apache/spark/pull/24965#issuecomment-506049891 Hi @BryanCutler Yes, that seems like a valid approach. Let me see if I can produce another prototype based on that approach and see if we can compare them. I think this solution is probably more flexible in the long run, but there would obviously be a cost to defining and maintaining our own custom streaming format (even if it is largely the same as the arrow format). If we have code examples for both it'll be easier to see.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
