[
https://issues.apache.org/jira/browse/SPARK-61501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61501:
-----------------------------------
Labels: flaky-test pull-request-available (was: flaky-test)
> Flaky test: TransformWithStateAvroSuite "test that introducing TTL after
> restart fails query"
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-61501
> URL: https://issues.apache.org/jira/browse/SPARK-61501
> Project: Spark
> Issue Type: Test
> Components: Structured Streaming
> Affects Versions: 4.2.0
> Reporter: Norio Akagi
> Priority: Minor
> Labels: flaky-test, pull-request-available
>
> `TransformWithStateAvroSuite` "test that introducing TTL after restart fails
> query (encoding = Avro)" (both the "with changelog checkpointing" and
> "without changelog checkpointing" variants) fails intermittently on Spark
> 4.2.0. Running the test alone with the ScalaTest runner on the v4.2.0 build
> (no plugins), 7 of 50 runs failed (3 with, 4 without changelog
> checkpointing). The failure is the exception the test expects, reported as
> the test failure:
> `org.apache.spark.sql.execution.streaming.state.StateStoreInvalidValueSchemaEvolution:
> [STATE_STORE_INVALID_VALUE_SCHEMA_EVOLUTION] Schema evolution is not
> possible new
> value_schema=StructType(StructField(value,StructType(StructField(value,LongType,true)),true),StructField(ttlExpirationMs,LongType,true))
> and old value_schema=...`
> The test code is the same in v4.0.2, v4.1.1 and v4.2.1. The second
> `testStream` restarts on the same checkpoint with
> `RunningCountStatefulProcessorWithTTL`, then runs `AddData`,
> `AdvanceManualClock(1000)`,
> `ExpectFailure[StateStoreInvalidValueSchemaEvolution]`. I have not worked out
> why the error is raised outside `ExpectFailure`.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]