HyukjinKwon opened a new pull request, #42289: URL: https://github.com/apache/spark/pull/42289
### What changes were proposed in this pull request? This PR proposes to remove session-based directories when the isolated session is evicted from the cache. ### Why are the changes needed? SPARK-44078 added the cache for isolated sessions, and SPARK-44348 added the session-based directory for isolation. When the isolated session cache is evicted, we should remove the session-based directory so it doesn't fail when the same session is used, see also https://github.com/apache/spark/pull/41625#discussion_r1251427466 ### Does this PR introduce _any_ user-facing change? No to end users since the feature has not been released yet. ### How was this patch tested? I manually tested as described in https://github.com/apache/spark/pull/41292. Especially, I reduced the TTL to few minutes, and tested as below at the last step: ```python spark.range(10).select(plug_one("id")).show() spark.range(10).select(plug_one("id")).show() # Wait few minutes spark.range(10).select(plug_one("id")).show() ``` I verified that the same session can be added back to the cache, and creates the directory with the same name. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
