[
https://issues.apache.org/jira/browse/SPARK-48073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842785#comment-17842785
]
Jungtaek Lim commented on SPARK-48073:
--------------------------------------
I roughly remember that Encoder.bean() had changed to address some issue;
probably generated model class of avro or protobuf or so. Looks like we are
affected by side effect.
While I agree this is a behavioral change (and maybe a breaking change) I'm
less sure what we could do. It's not something stateful operator can handle -
it's just a change of input. Do you want to propose a new SQL conf to fall back
the behavior of Encoder.bean() to 3.2?
> StateStore schema incompatibility between 3.2 and 3.4
> -----------------------------------------------------
>
> Key: SPARK-48073
> URL: https://issues.apache.org/jira/browse/SPARK-48073
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.2.4
> Reporter: L. C. Hsieh
> Priority: Major
>
> One our customer encountered some schema incompatibility problems when
> upgrading from Spark 3.2 to 3.4 with structured streaming application.
> It seems in 3.4 `Encoders.bean()` includes properties with only getter with
> or without setter, whereas in 3.2, only properties with both getter and
> setter are included.
> For example, here are schemas for an AtomicLong property/field generated by
> each version:
> 3.2:
> StructType(StructField(opaque,LongType,true),StructField(plain,LongType,true))
> 3.4:
> StructType(StructField(acquire,LongType,false),StructField(andDecrement,LongType,false),StructField(andIncrement,LongType,false),StructField(opaque,LongType,false),StructField(plain,LongType,false))
> Note that the null ability flag also changes.
> Primitive long schema has nullable=true in 3.2, but false in 3.4.
> I am not sure if the issue is aware by the community before, and if there is
> workaround for that?
> Thanks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
