mridulm commented on code in PR #39226:
URL: https://github.com/apache/spark/pull/39226#discussion_r1059289062
##########
core/src/main/scala/org/apache/spark/status/AppStatusStore.scala:
##########
@@ -733,6 +734,15 @@ private[spark] class AppStatusStore(
def close(): Unit = {
store.close()
+ cleanUpStorePath()
+ }
+
+ private def cleanUpStorePath(): Unit = {
+ storePath.foreach { p =>
+ if (p.exists()) {
+ p.listFiles().foreach(Utils.deleteRecursively)
+ }
+ }
Review Comment:
I was specifically referring to the context of driver using the kvstore for
the live UI data.
The codepath you [referred
to](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/status/KVUtils.scala#L155)
is for use in history server (particularly after a restart/upgrade of history
server) - not relevant for driver.
It becomes relevant to driver when multiple applications end up using the
same path - primarily in client mode.
Once we have support for persisting the kvstore for history server
consumption, this could change - but until then, I would say this should be an
internal impl detail, and get managed/deleted by driver - just like how we
manage other temp files today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
