Norio Akagi created SPARK-61501:
-----------------------------------

             Summary: Flaky test: TransformWithStateAvroSuite "test that 
introducing TTL after restart fails query"
                 Key: SPARK-61501
                 URL: https://issues.apache.org/jira/browse/SPARK-61501
             Project: Spark
          Issue Type: Test
          Components: Structured Streaming
    Affects Versions: 4.2.0
            Reporter: Norio Akagi


`TransformWithStateAvroSuite` "test that introducing TTL after restart fails 
query (encoding = Avro)" (both the "with changelog checkpointing" and "without 
changelog checkpointing" variants) fails intermittently on Spark 4.2.0. Running 
the test alone with the ScalaTest runner on the v4.2.0 build (no plugins), 7 of 
50 runs failed (3 with, 4 without changelog checkpointing). The failure is the 
exception the test expects, reported as the test failure:

`org.apache.spark.sql.execution.streaming.state.StateStoreInvalidValueSchemaEvolution:
 [STATE_STORE_INVALID_VALUE_SCHEMA_EVOLUTION] Schema evolution is not possible 
new 
value_schema=StructType(StructField(value,StructType(StructField(value,LongType,true)),true),StructField(ttlExpirationMs,LongType,true))
 and old value_schema=...`

The test code is the same in v4.0.2, v4.1.1 and v4.2.1. The second `testStream` 
restarts on the same checkpoint with `RunningCountStatefulProcessorWithTTL`, 
then runs `AddData`, `AdvanceManualClock(1000)`, 
`ExpectFailure[StateStoreInvalidValueSchemaEvolution]`. I have not worked out 
why the error is raised outside `ExpectFailure`.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to