[ 
https://issues.apache.org/jira/browse/SPARK-61451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Newton updated SPARK-61451:
----------------------------------
    Description: 
Since Spark 4.2.0 `spark.sql.execution.pythonUDF.arrow.enabled` defaults to 
`true`.

However when creating a python UDF at import time. For example if the UDF is 
defined using a decorator

```

from pyspark.sql.functions import udf

 

@udf(returnType=...)
def some_udf()
    ...

```

Whether this UDF uses arrow or not, now depends on whether the spark session 
was setup prior to importing this file, because of the loigic here 
[https://github.com/apache/spark/blob/10e94726d7d7216850b8c91184f7a75e86a6bfdb/python/pyspark/sql/udf.py#L113-L117]

Arrow UDFs do behave a bit differently to regular UDFs e.g. arrow UDFs 
dissallow a python list being converted to a struct. For us this causes 
different behaviour during tests vs in real deployments, because tests create 
the spark session after import time, using `pytest` fixtures, but in 
deployments the spark session is created before import time by spark-submit. 

 

  was:
Since Spark 4.2.0 `spark.sql.execution.pythonUDF.arrow.enabled` defaults to 
`true`.

However when creating a python UDF at import time. For example if the UDF is 
defined using a decorator

```

from pyspark.sql.functions import udf

 

@udf(returnType=...)
def some_udf()

```

Whether this UDF uses arrow or not, now depends on whether the spark session 
was setup prior to importing this file, because of the loigic here 
[https://github.com/apache/spark/blob/10e94726d7d7216850b8c91184f7a75e86a6bfdb/python/pyspark/sql/udf.py#L113-L117]

Arrow UDFs do behave a bit differently to regular UDFs e.g. arrow UDFs 
dissallow a python list being converted to a struct. For us this causes 
different behaviour during tests vs in real deployments, because tests create 
the spark session after import time, using `pytest` fixtures, but in 
deployments the spark session is created before import time by spark-submit. 

 


> Inconsistent default for whether arrow is used in UDFs
> ------------------------------------------------------
>
>                 Key: SPARK-61451
>                 URL: https://issues.apache.org/jira/browse/SPARK-61451
>             Project: Spark
>          Issue Type: Improvement
>          Components: UDF
>    Affects Versions: 4.2.0, 4.3.0, 4.2.1
>            Reporter: Thomas Newton
>            Priority: Minor
>
> Since Spark 4.2.0 `spark.sql.execution.pythonUDF.arrow.enabled` defaults to 
> `true`.
> However when creating a python UDF at import time. For example if the UDF is 
> defined using a decorator
> ```
> from pyspark.sql.functions import udf
>  
> @udf(returnType=...)
> def some_udf()
>     ...
> ```
> Whether this UDF uses arrow or not, now depends on whether the spark session 
> was setup prior to importing this file, because of the loigic here 
> [https://github.com/apache/spark/blob/10e94726d7d7216850b8c91184f7a75e86a6bfdb/python/pyspark/sql/udf.py#L113-L117]
> Arrow UDFs do behave a bit differently to regular UDFs e.g. arrow UDFs 
> dissallow a python list being converted to a struct. For us this causes 
> different behaviour during tests vs in real deployments, because tests create 
> the spark session after import time, using `pytest` fixtures, but in 
> deployments the spark session is created before import time by spark-submit. 
>  



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to