eddyxu commented on a change in pull request #31735:
URL: https://github.com/apache/spark/pull/31735#discussion_r588012018
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowPythonRunner.scala
##########
@@ -54,6 +54,9 @@ class ArrowPythonRunner(
"Pandas execution requires more than 4 bytes. Please set higher buffer. " +
s"Please change '${SQLConf.PANDAS_UDF_BUFFER_SIZE.key}'.")
+ /** This is a private key */
+ private val PANDAS_UDF_RETURN_TYPE_JSON =
"spark.sql.execution.pandas.udf.return.type.json"
Review comment:
Hi, do you suggest that we do not send schema via conf, or do not send
schema at all?
I see the schema is valuable on the worker for 2 purposes:
1). it can amortize the overhead of checking each row of a returned
pandas.{Series/DataFrame} for its schema. By detecting whether we have UDT in
the schema before running `@pandas_udf`, we can avoid to invoke the expensive
code path for the existing plain schema case.
and 2) this will be useful for passing UDT into `pandas_udf`, as wired
format is on pyarrow schema, the wire data needs to be reconstruct before
sending into the `pandas_udf`.
Also as you suggested below, this schema can be used to generate a function
that avoid type dispatch when doing UDT->struct conversion.
An alternative implementation could be:
```python
dataOut.writeInt(conf)
for ((k, v) <- conf) {
PythonRDD.writeUTF(k, dataOut)
PythonRDD.writeUTF(v, dataOut)
}
PythonRDD.writeUTF(schema_json, dataOut)
PythonUDFRunner.writeUDFs(dataOut, funcs, argOffsets)
```
Do we have concern about wire compatibility here? IIUC `worker.py` should
be deployed together with `ArrowEvalPythonExec` , it might not be an issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
