MonkeyCanCode commented on PR #14590: URL: https://github.com/apache/iceberg/pull/14590#issuecomment-6806324628
> > @MonkeyCanCode thanks for confirming that at least #18378 fixes the original issue. The reason you're seeing this particular stack trace is because we register the `FileIO` instance to be closed in `RESTSessionCatalog` but the `RESTSessionCatalog` itself is never closed, hence we're leaking `FileIO` instances there. We're still closing them but we're issuing a WARN msg to tell users that this is a resource leak. The same is true for the executor used in `RESTMetricsReporter` as its closing is tied to `RESTSessionCatalog` being closed. > > Thanks @nastra, that makes sense. To clarify the direction for the remaining part: > > 1. Is closing the catalog the preferred fix ([[SPARK-54367][Connect] Close session catalogs when a Spark Connect session is closed spark#59952](https://github.com/apache/spark/pull/59952)) plus SparkCatalog/SparkSessionCatalog passing close() through (this PR)? > 2. Separately, would you accept letting the metrics executor's idle thread exit (allowCoreThreadTimeOut, no inherited thread-locals)? I tested it on top of [Core: Share one daemon evictor across AuthSessionCache instances #18378](https://github.com/apache/iceberg/pull/18378), and released Spark Connect sessions are no longer retained even without catalog close. The FileIO warning remains in that case, as expected. @nastra mind takes a look at the above when you get a chance and let me know when we can have a conclusion on how we should proceed with this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
