[GitHub] [spark] LuciferYang commented on a diff in pull request #39385: [SPARK-41882][CORE][SQL][UI] Add tests for `SQLAppStatusStore` with RocksDB backend and fix some bugs
LuciferYang commented on code in PR #39385: URL: https://github.com/apache/spark/pull/39385#discussion_r1062138688 ## sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala: ## @@ -1007,6 +1002,34 @@ class SQLAppStatusListenerSuite extends SharedSparkSession with JsonTestUtils } } +class SQLAppStatusListenerWithInMemoryStoreSuite extends SQLAppStatusListenerSuite { + override protected def createStatusStore(): SQLAppStatusStore = { +val conf = sparkContext.conf +kvstore = new ElementTrackingStore(new InMemoryStore, conf) +val listener = new SQLAppStatusListener(conf, kvstore, live = true) +new SQLAppStatusStore(kvstore, Some(listener)) + } +} + +class SQLAppStatusListenerWithRocksDBBackendSuite extends SQLAppStatusListenerSuite { + private val storePath = Utils.createTempDir() Review Comment: still make `storePath` as class field and override `afterAll` to delete base `storePath` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #39385: [SPARK-41882][CORE][SQL][UI] Add tests for `SQLAppStatusStore` with RocksDB backend and fix some bugs
LuciferYang commented on code in PR #39385: URL: https://github.com/apache/spark/pull/39385#discussion_r1062051944 ## sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala: ## @@ -1007,6 +1004,36 @@ class SQLAppStatusListenerSuite extends SharedSparkSession with JsonTestUtils } } +class SQLAppStatusListenerWithInMemoryStoreSuite extends SQLAppStatusListenerSuite { + override protected def createStatusStore(): SQLAppStatusStore = { +val conf = sparkContext.conf +kvstore = new ElementTrackingStore(new InMemoryStore, conf) +val listener = new SQLAppStatusListener(conf, kvstore, live = true) +new SQLAppStatusStore(kvstore, Some(listener)) + } +} Review Comment: OK, let me fixed it together after https://github.com/apache/spark/pull/39226 merged -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #39385: [SPARK-41882][CORE][SQL][UI] Add tests for `SQLAppStatusStore` with RocksDB backend and fix some bugs
LuciferYang commented on code in PR #39385: URL: https://github.com/apache/spark/pull/39385#discussion_r1062051333 ## sql/core/src/test/scala/org/apache/spark/sql/execution/ui/AllExecutionsPageSuite.scala: ## @@ -178,3 +177,35 @@ class AllExecutionsPageSuite extends SharedSparkSession with BeforeAndAfter { } } +class AllExecutionsPageWithInMemoryStoreSuite extends AllExecutionsPageSuite { + override protected def createStatusStore: SQLAppStatusStore = { +val conf = sparkContext.conf +kvstore = new ElementTrackingStore(new InMemoryStore, conf) +val listener = new SQLAppStatusListener(conf, kvstore, live = true) +new SQLAppStatusStore(kvstore, Some(listener)) + } +} + +class AllExecutionsPageWithRocksDBBackendSuite extends AllExecutionsPageSuite { + // TODO: SPARK-41882 remove this field after RocksDB can automatically cleanup Review Comment: Thanks for your review and guidance @dongjoon-hyun , let's wait https://github.com/apache/spark/pull/39226, then I think we can clear the TODO in this pr -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] LuciferYang commented on a diff in pull request #39385: [SPARK-41882][CORE][SQL][UI] Add tests for `SQLAppStatusStore` with RocksDB backend and fix some bugs
LuciferYang commented on code in PR #39385: URL: https://github.com/apache/spark/pull/39385#discussion_r1062050562 ## sql/core/src/test/scala/org/apache/spark/sql/execution/ui/SQLAppStatusListenerSuite.scala: ## @@ -1007,6 +1004,36 @@ class SQLAppStatusListenerSuite extends SharedSparkSession with JsonTestUtils } } +class SQLAppStatusListenerWithInMemoryStoreSuite extends SQLAppStatusListenerSuite { + protected def createStatusStore(): SQLAppStatusStore = { Review Comment: Thanks @gengliangwang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org