[
https://issues.apache.org/jira/browse/SPARK-58310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58310:
-----------------------------------
Labels: pull-request-available (was: )
> Avoid runtime Bloom-filter subqueries containing Python UDFs
> ------------------------------------------------------------
>
> Key: SPARK-58310
> URL: https://issues.apache.org/jira/browse/SPARK-58310
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Chao Sun
> Priority: Major
> Labels: pull-request-available
>
> When runtime Bloom filtering and adaptive query execution are enabled,
> InjectRuntimeFilter can build a Bloom-filter scalar subquery from a
> creation-side logical plan that contains a Python UDF. Runtime filter
> subqueries are introduced after subquery optimization, so the Python UDF in
> the newly constructed subquery is not extracted into a Python evaluation
> operator. Physical planning then reaches PythonUDF.doGenCode while executing
> the BloomFilterAggregate, causing an unevaluable-expression/code-generation
> failure.
> A nested query using a Python-normalized dimension, DISTINCT, an explicitly
> broadcast join, and a scalar aggregate reproduces the problem against Apache
> Spark master. Both standard scalar Python UDFs and Arrow-backed scalar pandas
> UDFs are affected.
> Skip only the runtime Bloom filter whose creation-side plan contains a Python
> UDF. Continue to inject safe Bloom filters when a Python UDF appears only on
> the application side. Add a regression that checks adaptive execution, exact
> query results, a Parquet round trip, standard and Arrow UDFs, and continued
> safe Bloom-filter injection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
