HeartSaVioR commented on code in PR #38880:
URL: https://github.com/apache/spark/pull/38880#discussion_r1037968475
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBSuite.scala:
##########
@@ -116,7 +116,9 @@ class RocksDBSuite extends SparkFunSuite {
withDB(remoteDir, conf = conf) { db =>
// Generate versions without cleaning up
for (version <- 1 to 50) {
- db.put(version.toString, version.toString) // update "1" -> "1", "2"
-> "2", ...
Review Comment:
This was something we could indicate the bug earlier...
The test was incorrect. The test expects that compaction will happen, but if
we don't "overwrite" the key and just put the new keys here, compaction does
not happen, at least with Spark's default config. The reason compaction had
happened is, writebatch wasn't cleaned up during loop, hence writes are
accumulated into the same writebatch and it led to overwrites. If we fix the
code but leave this test as it is, then there are 50 SST files meaning no
compaction happens, and test fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
