Yaoxuan Wu created SPARK-61734:
----------------------------------
Summary: Python UDTF with PARTITION BY gets a partition's rows in
several runs when a LEFT ANTI / LEFT SEMI join with a constant-false condition
is pushed below it
Key: SPARK-61734
URL: https://issues.apache.org/jira/browse/SPARK-61734
Project: Spark
Issue Type: Bug
Components: PySpark, SQL
Affects Versions: 4.1.1, 4.2.0, 4.0.0
Reporter: Yaoxuan Wu
{code:python}
import datetime
from pyspark.sql import SparkSession, functions as F
from pyspark.sql.functions import udtf
spark = (SparkSession.builder
.config("spark.sql.shuffle.partitions", "1")
.config("spark.sql.execution.pythonUDTF.arrow.enabled", "false")
.getOrCreate())
class CountPerKey:
def __init__(self):
self.k, self.n = None, 0
def eval(self, row):
self.k = row["k"]
self.n += 1
def terminate(self):
yield self.k, self.n
spark.udtf.register(
"count_per_key",
udtf(CountPerKey, returnType="k: date, n: int",
useArrow=False).asDeterministic())
d1, d2 = datetime.date(2004, 4, 12), datetime.date(2009, 11, 30)
spark.createDataFrame([(d1,), (d2,), (d1,), (d2,)], "k
date").createOrReplaceTempView("t")
u = spark.sql("SELECT * FROM count_per_key(TABLE(t) PARTITION BY k)")
print(sorted(u.collect()))
# [Row(k=datetime.date(2004, 4, 12), n=2), Row(k=datetime.date(2009, 11, 30),
n=2)]
r = u.select(F.col("k").alias("k2"))
v = u.join(r, F.lit(None).cast("boolean"), "left_anti") # NOT EXISTS (...
WHERE NULL): every row of u
print(sorted(v.collect()))
# expected: the same 2 rows as u
# actual: [Row(k=datetime.date(2004, 4, 12), n=1), Row(k=datetime.date(2004,
4, 12), n=1),
# Row(k=datetime.date(2009, 11, 30), n=1), Row(k=datetime.date(2009,
11, 30), n=1)]
v.explain() # no Sort [partition_by_0] below BatchEvalPythonUDTF
{code}
Same on 4.0.0, 4.1.1 and 4.2.0. The UDTF sees each partition key in two
separate runs, so {{terminate()}} is called once per run. The condition {{(u.k
= r.k2) AND false}} gives the same result. In the query where this was found,
the condition was a comparison that the optimizer folds to false only after the
first pushdown step. Without {{asDeterministic()}} the result is right, because
the pushdown requires a deterministic Generate.
Cause (master d3f4a73a):
1. {{PushDownLeftSemiAntiJoin}}, case "LeftSemi/LeftAnti over UnaryNode",
pushes the join through {{u}} when
{{PushPredicateThroughNonJoin.canPushThrough(u)}} holds, and {{canPushThrough}}
returns true for every {{Generate}}. A condition that references no column
passes the {{references.subsetOf(u.child.outputSet ++ rightOp.outputSet)}}
check. The join lands between the UDTF's {{Generate}} and its {{Project <- Sort
[partition_by_0] <- RepartitionByExpression}} input.
2. {{EliminateSorts}}, case {{Join}} with a deterministic condition, calls
{{recursiveRemoveSort}} on both children and drops that Sort.
3. The anti join with the false condition is removed later. The final plan is
{{BatchEvalPythonUDTF <- Project <- Exchange hashpartitioning(partition_by_0)}}
with no Sort, and the Python runner starts a new partition at every key change.
The Sort below a UDTF with PARTITION BY / ORDER BY is required for correctness,
so it should not be separable from the Generate. Possible fixes: do not push
semi / anti joins through a {{Generate}} of a Python UDTF that has partitioning
or ordering, or express the per-partition order as a physical
{{requiredChildOrdering}} of the UDTF exec node instead of a logical Sort.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]