siying commented on code in PR #49063:
URL: https://github.com/apache/spark/pull/49063#discussion_r1871841017
##########
sql/core/src/test/scala/org/apache/spark/sql/streaming/CommitLogSuite.scala:
##########
@@ -108,6 +115,57 @@ class CommitLogSuite extends SparkFunSuite with
SharedSparkSession {
assert(commitMetadata.nextBatchWatermarkMs === 233)
assert(commitMetadata.stateUniqueIds === Map.empty)
}
+
+ // Test an old version of Spark can ser-de the new version of commit log,
+ // but running under V1 (i.e. no stateUniqueIds)
+ test("v1 Serde backward compatibility") {
+ // This is the json created by a V1 commit log
+ val commitLogV1WithStateUniqueId = """v1
+
|{"nextBatchWatermarkMs":1,"stateUniqueIds":{}}""".stripMargin
Review Comment:
There should be another case where the commit log is generated by previous
releases, likely looking lik:
```
"""v1
|{"nextBatchWatermarkMs":1}"""
```
Since we are already here we should cover that too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
