Yaoxuan Wu created SPARK-61734:
----------------------------------

             Summary: Python UDTF with PARTITION BY gets a partition's rows in 
several runs when a LEFT ANTI / LEFT SEMI join with a constant-false condition 
is pushed below it
                 Key: SPARK-61734
                 URL: https://issues.apache.org/jira/browse/SPARK-61734
             Project: Spark
          Issue Type: Bug
          Components: PySpark, SQL
    Affects Versions: 4.1.1, 4.2.0, 4.0.0
            Reporter: Yaoxuan Wu


{code:python}
import datetime
from pyspark.sql import SparkSession, functions as F
from pyspark.sql.functions import udtf

spark = (SparkSession.builder
         .config("spark.sql.shuffle.partitions", "1")
         .config("spark.sql.execution.pythonUDTF.arrow.enabled", "false")
         .getOrCreate())

class CountPerKey:
    def __init__(self):
        self.k, self.n = None, 0
    def eval(self, row):
        self.k = row["k"]
        self.n += 1
    def terminate(self):
        yield self.k, self.n

spark.udtf.register(
    "count_per_key",
    udtf(CountPerKey, returnType="k: date, n: int", 
useArrow=False).asDeterministic())

d1, d2 = datetime.date(2004, 4, 12), datetime.date(2009, 11, 30)
spark.createDataFrame([(d1,), (d2,), (d1,), (d2,)], "k 
date").createOrReplaceTempView("t")

u = spark.sql("SELECT * FROM count_per_key(TABLE(t) PARTITION BY k)")
print(sorted(u.collect()))
# [Row(k=datetime.date(2004, 4, 12), n=2), Row(k=datetime.date(2009, 11, 30), 
n=2)]

r = u.select(F.col("k").alias("k2"))
v = u.join(r, F.lit(None).cast("boolean"), "left_anti")   # NOT EXISTS (... 
WHERE NULL): every row of u
print(sorted(v.collect()))
# expected: the same 2 rows as u
# actual:   [Row(k=datetime.date(2004, 4, 12), n=1), Row(k=datetime.date(2004, 
4, 12), n=1),
#            Row(k=datetime.date(2009, 11, 30), n=1), Row(k=datetime.date(2009, 
11, 30), n=1)]
v.explain()   # no Sort [partition_by_0] below BatchEvalPythonUDTF
{code}
Same on 4.0.0, 4.1.1 and 4.2.0. The UDTF sees each partition key in two 
separate runs, so {{terminate()}} is called once per run. The condition {{(u.k 
= r.k2) AND false}} gives the same result. In the query where this was found, 
the condition was a comparison that the optimizer folds to false only after the 
first pushdown step. Without {{asDeterministic()}} the result is right, because 
the pushdown requires a deterministic Generate.

Cause (master d3f4a73a):
1. {{PushDownLeftSemiAntiJoin}}, case "LeftSemi/LeftAnti over UnaryNode", 
pushes the join through {{u}} when 
{{PushPredicateThroughNonJoin.canPushThrough(u)}} holds, and {{canPushThrough}} 
returns true for every {{Generate}}. A condition that references no column 
passes the {{references.subsetOf(u.child.outputSet ++ rightOp.outputSet)}} 
check. The join lands between the UDTF's {{Generate}} and its {{Project <- Sort 
[partition_by_0] <- RepartitionByExpression}} input.
2. {{EliminateSorts}}, case {{Join}} with a deterministic condition, calls 
{{recursiveRemoveSort}} on both children and drops that Sort.
3. The anti join with the false condition is removed later. The final plan is 
{{BatchEvalPythonUDTF <- Project <- Exchange hashpartitioning(partition_by_0)}} 
with no Sort, and the Python runner starts a new partition at every key change.

The Sort below a UDTF with PARTITION BY / ORDER BY is required for correctness, 
so it should not be separable from the Generate. Possible fixes: do not push 
semi / anti joins through a {{Generate}} of a Python UDTF that has partitioning 
or ordering, or express the per-partition order as a physical 
{{requiredChildOrdering}} of the UDTF exec node instead of a logical Sort.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to