andygrove commented on issue #964: URL: https://github.com/apache/arrow-ballista/issues/964#issuecomment-1934303991
Yes, it could be supported, at least to some degree. I looked at the [docs](https://spark.apache.org/docs/latest/spark-connect-overview.html), and it looks like Spark Connect is essentially a gRPC service that accepts protobuf describing Spark logical plans and then executes the plans and sends results back. This seems achievable for anything that Spark can represent in a logical plan that Ballista also supports. I haven't read enough of the documentation to understand how UDFs would work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
