Hyukjin Kwon created SPARK-42115:
------------------------------------
Summary: Push down limit through Python UDFs
Key: SPARK-42115
URL: https://issues.apache.org/jira/browse/SPARK-42115
Project: Spark
Issue Type: Bug
Components: PySpark, SQL
Affects Versions: 3.4.0
Reporter: Hyukjin Kwon
{code}
from pyspark.sql.functions import udf
spark.range(10).write.mode("overwrite").parquet("/tmp/abc")
@udf(returnType='string')
def my_udf(arg):
return arg
df = spark.read.parquet("/tmp/abc")
df = df.limit(10).withColumn("prediction", my_udf(df["id"])).explain()
{code}
As an example. since Python UDFs are executed asynchronously, so pushing limits
benefit the performance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
