Anil K created SPARK-62032:
------------------------------

             Summary: Operators that pair data with an initial state by key do 
not check that the two sides' grouping keys have one type
                 Key: SPARK-62032
                 URL: https://issues.apache.org/jira/browse/SPARK-62032
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Anil K


Operators that pair a grouped dataset with an initial state by key do not check 
that the two sides' grouping keys have one type: flatMapGroupsWithState and 
mapGroupsWithState with an initialState, transformWithState with an 
initialState (Scala, classic and Spark Connect), and transformWithStateInPandas 
/ transformWithStateInPySpark with an initialState (Python). With 
{{groupBy(cols).as[K, V]}}, or a Python {{groupBy}}, the data side and the 
initial-state side can group by keys of different types, e.g. an {{INT}} and a 
{{BIGINT}}.

Each side is then hash-partitioned on its own key type, and state is paired 
with data by comparing raw key bytes, so state can be missed or attached to the 
wrong key. Measured on master (7c8056a3365) for a batch flatMapGroupsWithState 
with an initial state, over the same key pairs as SPARK-60782:
* {{BIGINT}} data keys with {{INT}} state keys: 5 of 7 keys found their state;
* {{INT}}/{{BIGINT}}, {{INT}}/{{DOUBLE}}, struct and array keys: with AQE on, a 
task fails with {{AssertionError}} at {{CoGroupedIterator.next}};
* {{STRING}}/{{INT}}: {{NegativeArraySizeException}};
* decimals of another storage or scale, {{DATE}}/{{TIMESTAMP}}, 
{{TIMESTAMP}}/{{TIMESTAMP_NTZ}} outside UTC: some keys miss their state.

SPARK-60782 fixes the same gap for cogroups, by widening Pandas and Arrow 
cogroup keys where SQL has a common type, and otherwise failing at analysis 
with {{COGROUP_KEY_TYPE_MISMATCH}}. These operators are stateful, so the fix is 
likely different: widening the data side's key would change the state store's 
key schema, and rejecting can fail a streaming query restarted from a 
checkpoint. A fix could reuse SPARK-60782's check, rejecting keys not equal 
ignoring nullability and field names, possibly widening only the initial-state 
side to the data side's type.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to