xinrong-meng commented on PR #39585: URL: https://github.com/apache/spark/pull/39585#issuecomment-1386654628
> Can we make the python specific stuff a binary blob instead of an actual message? That way you have more flexibility in the language specific bits, for scala for example we are planning to use a serialised JVM object instead of a proto message. My understanding is the one level higher abstraction `ScalarInlineUserDefinedFunction` may provide enough flexibility since language-specific metadata can be passed in the `PythonUDF` proto. And future updates of `PythonUDF` proto wouldn't interfere with protos of UDFS of other languages. My concern is serializing `PythonUDF` into a binary blob may hurt the debuggability of proto messages and introduce additional complexity to code. Please let me know if there are other benefits of serializing `PythonUDF` that I miss. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
