HeartSaVioR commented on code in PR #49811:
URL: https://github.com/apache/spark/pull/49811#discussion_r1962873691
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSourceReadSuite.scala:
##########
@@ -794,6 +795,11 @@ abstract class StateDataSourceReadSuite extends
StateDataSourceTestBase with Ass
}
test("flatMapGroupsWithState, state ver 1") {
+ // Skip this test on big endian platforms because the
timestampTimeoutAttribute of
+ // StateManagerImplV1 is declared as IntegerType instead of LongType which
breaks
+ // serialization on big endian. This can't be fixed because it would be a
breaking
+ // schema change.
+ assume(ByteOrder.nativeOrder().equals(ByteOrder.LITTLE_ENDIAN))
Review Comment:
nit: `ByteOrder.nativeOrder().equals(ByteOrder.LITTLE_ENDIAN)`
maybe good enough to put this to Utils?
cc. @jiangxb1987 @Ngone51 WDYT?
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/FlatMapGroupsWithStateSuite.scala:
##########
@@ -458,7 +459,16 @@ class FlatMapGroupsWithStateSuite extends
StateStoreMetricsTest {
checkAnswer(df, Seq(("a", 2), ("b", 1)).toDF())
}
+ // Skip the v1 tests with timeout on big endian platforms because the
+ // timestampTimeoutAttribute of StateManagerImplV1 is declared as
IntegerType instead
+ // of LongType which breaks serialization on big endian. This can't be fixed
because it
+ // would be a breaking schema change.
+ def isStateFormatSupported(stateFormatVersion: Int): Boolean = {
+ stateFormatVersion != 1 ||
ByteOrder.nativeOrder().equals(ByteOrder.LITTLE_ENDIAN)
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
