Yaoxuan Wu created SPARK-61947:
----------------------------------
Summary: transformWithStateInPandas: with a NULL in an integral
grouping key, the other keys of the same Arrow batch read and write the NULL
key's state
Key: SPARK-61947
URL: https://issues.apache.org/jira/browse/SPARK-61947
Project: Spark
Issue Type: Bug
Components: PySpark, Structured Streaming
Affects Versions: 4.2.0
Reporter: Yaoxuan Wu
In transformWithStateInPandas with an integral grouping key (TINYINT / SMALLINT
/ INT / BIGINT) that contains NULL, the keys that arrive in the same Arrow
batch as a NULL key read and write the NULL key's state, and handleInputRows is
called once per NULL-key row instead of once per key. No error is raised.
{code:python}
import os, tempfile
import pandas as pd
from pyspark.sql import SparkSession
from pyspark.sql.streaming import StatefulProcessor
spark = (SparkSession.builder.config("spark.sql.shuffle.partitions", "1")
.config("spark.sql.streaming.stateStore.providerClass",
"org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider")
.getOrCreate())
src = os.path.join(tempfile.mkdtemp(), "src")
spark.createDataFrame([(i % 3 if i % 5 else None, i) for i in range(12)], "k
int, v int").write.parquet(src)
class Count(StatefulProcessor):
def init(self, handle):
self.n = handle.getValueState("n", "n long")
def handleInputRows(self, key, rows, timerValues):
n = (self.n.get()[0] if self.n.exists() else 0) + sum(len(pdf) for pdf
in rows)
self.n.update((n,))
yield pd.DataFrame({"k": [key[0]], "n": [n]})
def close(self):
pass
out = {}
q = (spark.readStream.schema("k int, v int").parquet(src).groupBy("k")
.transformWithStateInPandas(Count(), "k int, n long", "Update", "None")
.writeStream.outputMode("update").trigger(availableNow=True)
.option("checkpointLocation", tempfile.mkdtemp())
.foreachBatch(lambda df, bid: out.setdefault(bid, df.collect())).start())
q.awaitTermination()
print(sorted(spark.read.parquet(src).groupBy("k").count().collect(), key=repr))
print(sorted(out[0], key=repr))
{code}
{noformat}
batch groupBy().count(): k=0: 3, k=1: 3, k=2: 3, k=NULL: 3
streaming: k=0: 6, k=1: 9, k=2: 12, and three rows for k=NULL:
1, 2, 3
{noformat}
* Wrong on 4.1.1 and 4.2.0; correct on 4.0.0. Master has the same code.
* transformWithState (Row API) and applyInPandasWithState return the right
counts for the same input.
* {{spark.sql.execution.pythonUDF.pandas.preferIntExtensionDtype=true}} does
not help: the NULL key is then {{pd.NA}}, and the regrouping fails with
"TypeError: boolean value of NA is ambiguous".
Cause:
1. python/pyspark/worker.py (SQL_TRANSFORM_WITH_STATE_PANDAS_UDF, and the
init-state variant) regroups the rows of each Arrow batch in Python: {{for row
in pd.concat(data_pandas, axis=1).itertuples(index=False): batch_key =
tuple(row[o] for o in key_offsets)}}, then {{itertools.groupby(..., key=lambda
x: x[0])}}. The key is taken from the pandas form of the row. An integral
column with a NULL is float64 in pandas, so the NULL key is {{nan}} and the
batch's other keys are floats ({{2.0}}).
2. {{nan != nan}}, so {{itertools.groupby}} puts every NULL-key row in its own
group: one handleInputRows call per row.
3. {{StatefulProcessorApiClient.set_implicit_key}} pickles the key tuple as it
is, and the JVM rebuilds it with {{PythonSQLUtils.toJVMRow}} ->
{{EvaluatePython.makeFromJava(IntegerType)}}, which accepts only Byte / Short /
Int / Long and returns null for anything else. The float key {{2.0}} becomes
NULL, so the state of k=2 is the NULL key's state.
BIGINT keys above 2^53 also lose precision on the same path. In 4.0.0 the key
of each group came from the JVM's grouping, not from the pandas rows. Taking
the key from the Arrow data (or converting it back to the key's Spark type
before set_implicit_key and grouping with a NULL-aware key) would fix it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]