Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21700#discussion_r201866279
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/StateStoreSuite.scala
---
@@ -64,6 +64,63 @@ class StateStoreSuite extends
StateStoreSuiteBase[HDFSBackedStateStoreProvider]
require(!StateStore.isMaintenanceRunning)
}
+ test("retaining only latest configured size of versions in memory") {
--- End diff --
It is fairly easy to check whether reading from cache or reading from file
with
https://github.com/apache/spark/pull/21469/commits/c9aada520889b87ace0886805910f0d56d099bd2
in #21469 since it introduces metrics for cache hit and cache miss, but not
easy to check in this PR itself.
So I just rely on checking cache to ensure the data is correctly evicted
and not available in cache as expected. Hope this is OK.
Btw, I caught a silly bug while adding tests to cover your suggestion.
Thanks!
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
