Yaoxuan Wu created SPARK-61946:
----------------------------------

             Summary: transformWithStateInPandas renames input columns whose 
names start with '_' (or are not identifiers) to _<position> in handleInputRows
                 Key: SPARK-61946
                 URL: https://issues.apache.org/jira/browse/SPARK-61946
             Project: Spark
          Issue Type: Bug
          Components: PySpark, Structured Streaming
    Affects Versions: 4.2.0
            Reporter: Yaoxuan Wu


In transformWithStateInPandas, the pandas DataFrames passed to 
StatefulProcessor.handleInputRows do not keep the input's column names: every 
column whose name starts with '_', is a Python keyword or is not an identifier 
(e.g. `a b`) is renamed to _<position>. Reading such a column by name raises 
KeyError, or, when the new name equals another column's name, silently reads 
the wrong column.

{code:python}
import os, sys, tempfile
import pandas as pd
from pyspark.sql import SparkSession, functions as F
from pyspark.sql.streaming import StatefulProcessor

spark = (SparkSession.builder.config("spark.sql.shuffle.partitions", "1")
         .config("spark.sql.streaming.stateStore.providerClass",
                 
"org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider")
         .getOrCreate())
src = os.path.join(tempfile.mkdtemp(), "src")
schema = "k int, `_2` int, `_1` int"
spark.createDataFrame([(1, 4, 40), (1, 6, 60), (2, 1, 1)], 
schema).write.parquet(src)

class SumOne(StatefulProcessor):
    def init(self, handle):
        pass
    def handleInputRows(self, key, rows, timerValues):
        s = 0
        for pdf in rows:
            print("columns seen:", list(pdf.columns), file=sys.stderr)
            s += int(pdf["_1"].sum())
        yield pd.DataFrame({"k": [key[0]], "s": [s]})
    def close(self):
        pass

out = {}
q = (spark.readStream.schema(schema).parquet(src).groupBy("k")
     .transformWithStateInPandas(SumOne(), "k int, s long", "Update", "None")
     .writeStream.outputMode("update").trigger(availableNow=True)
     .option("checkpointLocation", tempfile.mkdtemp())
     .foreachBatch(lambda df, bid: out.setdefault(bid, df.collect())).start())
q.awaitTermination()
print(sorted(spark.read.parquet(src).groupBy("k").agg(F.sum("_1").alias("s")).collect()))
print(sorted(out[0]))
{code}

{noformat}
batch:      [Row(k=1, s=100), Row(k=2, s=1)]
streaming:  [Row(k=1, s=10), Row(k=2, s=1)]      expected the same as batch
columns seen: ['k', '_1', '_2']                   (input columns are k, _2, _1)
{noformat}

* Wrong on 4.1.1 and 4.2.0; 4.0.0 returns s=100 and keeps the names. Master has 
the same code.

Cause (python/pyspark/worker.py, SQL_TRANSFORM_WITH_STATE_PANDAS_UDF and the 
init-state variant): the worker regroups the rows with {{pd.concat(data_pandas, 
axis=1).itertuples(index=False)}} and rebuilds each chunk with 
{{pd.DataFrame(rows)}}. {{itertuples}} returns namedtuples created with 
{{rename=True}}, and namedtuple renames invalid identifiers, keywords, 
duplicates and names starting with an underscore to {{_<index>}}. 
{{pd.DataFrame}} of a list of namedtuples takes its column names from the 
namedtuple fields. Here {{_2}} at position 1 becomes {{_1}}, and {{_1}} at 
position 2 becomes {{_2}}.

Building the chunks with the original column names (e.g. 
{{itertuples(index=False, name=None)}} and {{pd.DataFrame(rows, columns=...)}}) 
would avoid this. The same regrouping also takes the grouping key from the 
pandas form of the row, which breaks NULL integral keys (filed separately).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to