[ 
https://issues.apache.org/jira/browse/SPARK-61501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61501:
-----------------------------------
    Labels: flaky-test pull-request-available  (was: flaky-test)

> Flaky test: TransformWithStateAvroSuite "test that introducing TTL after 
> restart fails query"
> ---------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61501
>                 URL: https://issues.apache.org/jira/browse/SPARK-61501
>             Project: Spark
>          Issue Type: Test
>          Components: Structured Streaming
>    Affects Versions: 4.2.0
>            Reporter: Norio Akagi
>            Priority: Minor
>              Labels: flaky-test, pull-request-available
>
> `TransformWithStateAvroSuite` "test that introducing TTL after restart fails 
> query (encoding = Avro)" (both the "with changelog checkpointing" and 
> "without changelog checkpointing" variants) fails intermittently on Spark 
> 4.2.0. Running the test alone with the ScalaTest runner on the v4.2.0 build 
> (no plugins), 7 of 50 runs failed (3 with, 4 without changelog 
> checkpointing). The failure is the exception the test expects, reported as 
> the test failure:
> `org.apache.spark.sql.execution.streaming.state.StateStoreInvalidValueSchemaEvolution:
>  [STATE_STORE_INVALID_VALUE_SCHEMA_EVOLUTION] Schema evolution is not 
> possible new 
> value_schema=StructType(StructField(value,StructType(StructField(value,LongType,true)),true),StructField(ttlExpirationMs,LongType,true))
>  and old value_schema=...`
> The test code is the same in v4.0.2, v4.1.1 and v4.2.1. The second 
> `testStream` restarts on the same checkpoint with 
> `RunningCountStatefulProcessorWithTTL`, then runs `AddData`, 
> `AdvanceManualClock(1000)`, 
> `ExpectFailure[StateStoreInvalidValueSchemaEvolution]`. I have not worked out 
> why the error is raised outside `ExpectFailure`.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to