Norio Akagi created SPARK-61501:
-----------------------------------
Summary: Flaky test: TransformWithStateAvroSuite "test that
introducing TTL after restart fails query"
Key: SPARK-61501
URL: https://issues.apache.org/jira/browse/SPARK-61501
Project: Spark
Issue Type: Test
Components: Structured Streaming
Affects Versions: 4.2.0
Reporter: Norio Akagi
`TransformWithStateAvroSuite` "test that introducing TTL after restart fails
query (encoding = Avro)" (both the "with changelog checkpointing" and "without
changelog checkpointing" variants) fails intermittently on Spark 4.2.0. Running
the test alone with the ScalaTest runner on the v4.2.0 build (no plugins), 7 of
50 runs failed (3 with, 4 without changelog checkpointing). The failure is the
exception the test expects, reported as the test failure:
`org.apache.spark.sql.execution.streaming.state.StateStoreInvalidValueSchemaEvolution:
[STATE_STORE_INVALID_VALUE_SCHEMA_EVOLUTION] Schema evolution is not possible
new
value_schema=StructType(StructField(value,StructType(StructField(value,LongType,true)),true),StructField(ttlExpirationMs,LongType,true))
and old value_schema=...`
The test code is the same in v4.0.2, v4.1.1 and v4.2.1. The second `testStream`
restarts on the same checkpoint with `RunningCountStatefulProcessorWithTTL`,
then runs `AddData`, `AdvanceManualClock(1000)`,
`ExpectFailure[StateStoreInvalidValueSchemaEvolution]`. I have not worked out
why the error is raised outside `ExpectFailure`.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]