viirya commented on code in PR #58978: URL: https://github.com/apache/spark/pull/58978#discussion_r4321765862
########## dev/spark-test-image/python-312/Dockerfile: ########## @@ -66,3 +67,21 @@ RUN python3.12 -m pip install -U pip RUN python3.12 -m pip install --group ml_torch --index-url https://download.pytorch.org/whl/cpu && \ python3.12 -m pip install --group ci_classic_standard --group ci_connect_standard && \ python3.12 -m pip cache purge + +# Exercise in-process UDFs in the existing pyspark-sql job. JEP needs Python +# headers and a JDK when building its JNI library; no separate test job is needed. +# Read the JEP and CDI versions from this checkout, just as the Spark build does. +COPY --from=root pom.xml /tmp/spark-pom.xml Review Comment: Done in 8aead4b as suggested: a first `spark-versions` stage copies `pom.xml` and writes the two versions to files with `sed`, and the image copies each file right before its install, so the JEP and CDI layers change only when their versions do. ########## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/PythonUDF.scala: ########## @@ -51,13 +51,25 @@ object PythonUDF { PythonEvalType.SQL_SCALAR_PANDAS_UDF, PythonEvalType.SQL_SCALAR_PANDAS_ITER_UDF, PythonEvalType.SQL_SCALAR_ARROW_UDF, - PythonEvalType.SQL_SCALAR_ARROW_ITER_UDF + PythonEvalType.SQL_SCALAR_ARROW_ITER_UDF, + PythonEvalType.SQL_SCALAR_ARROW_INPROCESS_UDF ) def isScalarPythonUDF(e: Expression): Boolean = { e.isInstanceOf[PythonUDF] && SCALAR_TYPES.contains(e.asInstanceOf[PythonUDF].evalType) } + /** + * Whether `e` is a scalar Python UDF that runs in a Python worker, where it can be chained + * with the scalar Python UDFs around it. In-process UDFs never are: each one is evaluated by Review Comment: Fixed in 8aead4b: "a nested in-process call is extracted into a separate operator, although sibling calls may share one". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
