Yaoxuan Wu created SPARK-61946:
----------------------------------
Summary: transformWithStateInPandas renames input columns whose
names start with '_' (or are not identifiers) to _<position> in handleInputRows
Key: SPARK-61946
URL: https://issues.apache.org/jira/browse/SPARK-61946
Project: Spark
Issue Type: Bug
Components: PySpark, Structured Streaming
Affects Versions: 4.2.0
Reporter: Yaoxuan Wu
In transformWithStateInPandas, the pandas DataFrames passed to
StatefulProcessor.handleInputRows do not keep the input's column names: every
column whose name starts with '_', is a Python keyword or is not an identifier
(e.g. `a b`) is renamed to _<position>. Reading such a column by name raises
KeyError, or, when the new name equals another column's name, silently reads
the wrong column.
{code:python}
import os, sys, tempfile
import pandas as pd
from pyspark.sql import SparkSession, functions as F
from pyspark.sql.streaming import StatefulProcessor
spark = (SparkSession.builder.config("spark.sql.shuffle.partitions", "1")
.config("spark.sql.streaming.stateStore.providerClass",
"org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider")
.getOrCreate())
src = os.path.join(tempfile.mkdtemp(), "src")
schema = "k int, `_2` int, `_1` int"
spark.createDataFrame([(1, 4, 40), (1, 6, 60), (2, 1, 1)],
schema).write.parquet(src)
class SumOne(StatefulProcessor):
def init(self, handle):
pass
def handleInputRows(self, key, rows, timerValues):
s = 0
for pdf in rows:
print("columns seen:", list(pdf.columns), file=sys.stderr)
s += int(pdf["_1"].sum())
yield pd.DataFrame({"k": [key[0]], "s": [s]})
def close(self):
pass
out = {}
q = (spark.readStream.schema(schema).parquet(src).groupBy("k")
.transformWithStateInPandas(SumOne(), "k int, s long", "Update", "None")
.writeStream.outputMode("update").trigger(availableNow=True)
.option("checkpointLocation", tempfile.mkdtemp())
.foreachBatch(lambda df, bid: out.setdefault(bid, df.collect())).start())
q.awaitTermination()
print(sorted(spark.read.parquet(src).groupBy("k").agg(F.sum("_1").alias("s")).collect()))
print(sorted(out[0]))
{code}
{noformat}
batch: [Row(k=1, s=100), Row(k=2, s=1)]
streaming: [Row(k=1, s=10), Row(k=2, s=1)] expected the same as batch
columns seen: ['k', '_1', '_2'] (input columns are k, _2, _1)
{noformat}
* Wrong on 4.1.1 and 4.2.0; 4.0.0 returns s=100 and keeps the names. Master has
the same code.
Cause (python/pyspark/worker.py, SQL_TRANSFORM_WITH_STATE_PANDAS_UDF and the
init-state variant): the worker regroups the rows with {{pd.concat(data_pandas,
axis=1).itertuples(index=False)}} and rebuilds each chunk with
{{pd.DataFrame(rows)}}. {{itertuples}} returns namedtuples created with
{{rename=True}}, and namedtuple renames invalid identifiers, keywords,
duplicates and names starting with an underscore to {{_<index>}}.
{{pd.DataFrame}} of a list of namedtuples takes its column names from the
namedtuple fields. Here {{_2}} at position 1 becomes {{_1}}, and {{_1}} at
position 2 becomes {{_2}}.
Building the chunks with the original column names (e.g.
{{itertuples(index=False, name=None)}} and {{pd.DataFrame(rows, columns=...)}})
would avoid this. The same regrouping also takes the grouping key from the
pandas form of the row, which breaks NULL integral keys (filed separately).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]