[
https://issues.apache.org/jira/browse/SPARK-51092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Albrecht updated SPARK-51092:
--------------------------------------
Description:
The {{dataType}} of the {{timestampTimeoutAttribute}} is IntegerType in
FlatMapGroupsWithStateExecHelper.scala for the v1 state manager but it is
serialized and stored in memory as a Long everywhere.
This causes the {{timestampTimeout}} value to be corrupted on big endian
platforms. This is seen in unit tests relating to the v1 state manager when run
on big endian platforms.
This can't be fixed because it would be a breaking schema change so skip the
tests instead.
was:
The {{dataType}} of the {{timestampTimeoutAttribute}} is IntegerType in
FlatMapGroupsWithStateExecHelper.scala for the v1 state manager but it is
serialized and stored in memory as a Long everywhere.
This causes the {{timestampTimeout}} value to be corrupted on big endian
platforms. This is seen in unit tests relating to the v1 state manager when run
on big endian platforms.
> Skip the v1 FlatMapGroupsWithState tests with timeout on big endian platforms
> -----------------------------------------------------------------------------
>
> Key: SPARK-51092
> URL: https://issues.apache.org/jira/browse/SPARK-51092
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 3.5.4, 3.5.5, 4.1.0
> Reporter: Jonathan Albrecht
> Priority: Minor
> Labels: big-endian, pull-request-available
>
> The {{dataType}} of the {{timestampTimeoutAttribute}} is IntegerType in
> FlatMapGroupsWithStateExecHelper.scala for the v1 state manager but it is
> serialized and stored in memory as a Long everywhere.
> This causes the {{timestampTimeout}} value to be corrupted on big endian
> platforms. This is seen in unit tests relating to the v1 state manager when
> run on big endian platforms.
> This can't be fixed because it would be a breaking schema change so skip the
> tests instead.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
