Yaoxuan Wu created SPARK-61947:
----------------------------------

             Summary: transformWithStateInPandas: with a NULL in an integral 
grouping key, the other keys of the same Arrow batch read and write the NULL 
key's state
                 Key: SPARK-61947
                 URL: https://issues.apache.org/jira/browse/SPARK-61947
             Project: Spark
          Issue Type: Bug
          Components: PySpark, Structured Streaming
    Affects Versions: 4.2.0
            Reporter: Yaoxuan Wu


In transformWithStateInPandas with an integral grouping key (TINYINT / SMALLINT 
/ INT / BIGINT) that contains NULL, the keys that arrive in the same Arrow 
batch as a NULL key read and write the NULL key's state, and handleInputRows is 
called once per NULL-key row instead of once per key. No error is raised.

{code:python}
import os, tempfile
import pandas as pd
from pyspark.sql import SparkSession
from pyspark.sql.streaming import StatefulProcessor

spark = (SparkSession.builder.config("spark.sql.shuffle.partitions", "1")
         .config("spark.sql.streaming.stateStore.providerClass",
                 
"org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider")
         .getOrCreate())
src = os.path.join(tempfile.mkdtemp(), "src")
spark.createDataFrame([(i % 3 if i % 5 else None, i) for i in range(12)], "k 
int, v int").write.parquet(src)

class Count(StatefulProcessor):
    def init(self, handle):
        self.n = handle.getValueState("n", "n long")
    def handleInputRows(self, key, rows, timerValues):
        n = (self.n.get()[0] if self.n.exists() else 0) + sum(len(pdf) for pdf 
in rows)
        self.n.update((n,))
        yield pd.DataFrame({"k": [key[0]], "n": [n]})
    def close(self):
        pass

out = {}
q = (spark.readStream.schema("k int, v int").parquet(src).groupBy("k")
     .transformWithStateInPandas(Count(), "k int, n long", "Update", "None")
     .writeStream.outputMode("update").trigger(availableNow=True)
     .option("checkpointLocation", tempfile.mkdtemp())
     .foreachBatch(lambda df, bid: out.setdefault(bid, df.collect())).start())
q.awaitTermination()
print(sorted(spark.read.parquet(src).groupBy("k").count().collect(), key=repr))
print(sorted(out[0], key=repr))
{code}

{noformat}
batch groupBy().count():  k=0: 3, k=1: 3, k=2: 3, k=NULL: 3
streaming:                k=0: 6, k=1: 9, k=2: 12, and three rows for k=NULL: 
1, 2, 3
{noformat}

* Wrong on 4.1.1 and 4.2.0; correct on 4.0.0. Master has the same code.
* transformWithState (Row API) and applyInPandasWithState return the right 
counts for the same input.
* {{spark.sql.execution.pythonUDF.pandas.preferIntExtensionDtype=true}} does 
not help: the NULL key is then {{pd.NA}}, and the regrouping fails with 
"TypeError: boolean value of NA is ambiguous".

Cause:
1. python/pyspark/worker.py (SQL_TRANSFORM_WITH_STATE_PANDAS_UDF, and the 
init-state variant) regroups the rows of each Arrow batch in Python: {{for row 
in pd.concat(data_pandas, axis=1).itertuples(index=False): batch_key = 
tuple(row[o] for o in key_offsets)}}, then {{itertools.groupby(..., key=lambda 
x: x[0])}}. The key is taken from the pandas form of the row. An integral 
column with a NULL is float64 in pandas, so the NULL key is {{nan}} and the 
batch's other keys are floats ({{2.0}}).
2. {{nan != nan}}, so {{itertools.groupby}} puts every NULL-key row in its own 
group: one handleInputRows call per row.
3. {{StatefulProcessorApiClient.set_implicit_key}} pickles the key tuple as it 
is, and the JVM rebuilds it with {{PythonSQLUtils.toJVMRow}} -> 
{{EvaluatePython.makeFromJava(IntegerType)}}, which accepts only Byte / Short / 
Int / Long and returns null for anything else. The float key {{2.0}} becomes 
NULL, so the state of k=2 is the NULL key's state.

BIGINT keys above 2^53 also lose precision on the same path. In 4.0.0 the key 
of each group came from the JVM's grouping, not from the pandas rows. Taking 
the key from the Arrow data (or converting it back to the key's Spark type 
before set_implicit_key and grouping with a NULL-aware key) would fix it.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to