[
https://issues.apache.org/jira/browse/SPARK-61451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Newton updated SPARK-61451:
----------------------------------
Description:
Since Spark 4.2.0 `spark.sql.execution.pythonUDF.arrow.enabled` defaults to
`true`.
However when creating a python UDF at import time. For example if the UDF is
defined using a decorator
```
from pyspark.sql.functions import udf
@udf(returnType=...)
def some_udf()
...
```
Whether this UDF uses arrow or not, now depends on whether the spark session
was setup prior to importing this file, because of the loigic here
[https://github.com/apache/spark/blob/10e94726d7d7216850b8c91184f7a75e86a6bfdb/python/pyspark/sql/udf.py#L113-L117]
Arrow UDFs do behave a bit differently to regular UDFs e.g. arrow UDFs
dissallow a python list being converted to a struct. For us this causes
different behaviour during tests vs in real deployments, because tests create
the spark session after import time, using `pytest` fixtures, but in
deployments the spark session is created before import time by spark-submit.
was:
Since Spark 4.2.0 `spark.sql.execution.pythonUDF.arrow.enabled` defaults to
`true`.
However when creating a python UDF at import time. For example if the UDF is
defined using a decorator
```
from pyspark.sql.functions import udf
@udf(returnType=...)
def some_udf()
```
Whether this UDF uses arrow or not, now depends on whether the spark session
was setup prior to importing this file, because of the loigic here
[https://github.com/apache/spark/blob/10e94726d7d7216850b8c91184f7a75e86a6bfdb/python/pyspark/sql/udf.py#L113-L117]
Arrow UDFs do behave a bit differently to regular UDFs e.g. arrow UDFs
dissallow a python list being converted to a struct. For us this causes
different behaviour during tests vs in real deployments, because tests create
the spark session after import time, using `pytest` fixtures, but in
deployments the spark session is created before import time by spark-submit.
> Inconsistent default for whether arrow is used in UDFs
> ------------------------------------------------------
>
> Key: SPARK-61451
> URL: https://issues.apache.org/jira/browse/SPARK-61451
> Project: Spark
> Issue Type: Improvement
> Components: UDF
> Affects Versions: 4.2.0, 4.3.0, 4.2.1
> Reporter: Thomas Newton
> Priority: Minor
>
> Since Spark 4.2.0 `spark.sql.execution.pythonUDF.arrow.enabled` defaults to
> `true`.
> However when creating a python UDF at import time. For example if the UDF is
> defined using a decorator
> ```
> from pyspark.sql.functions import udf
>
> @udf(returnType=...)
> def some_udf()
> ...
> ```
> Whether this UDF uses arrow or not, now depends on whether the spark session
> was setup prior to importing this file, because of the loigic here
> [https://github.com/apache/spark/blob/10e94726d7d7216850b8c91184f7a75e86a6bfdb/python/pyspark/sql/udf.py#L113-L117]
> Arrow UDFs do behave a bit differently to regular UDFs e.g. arrow UDFs
> dissallow a python list being converted to a struct. For us this causes
> different behaviour during tests vs in real deployments, because tests create
> the spark session after import time, using `pytest` fixtures, but in
> deployments the spark session is created before import time by spark-submit.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]