lirui-apache commented on PR #6175: URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1319845848
The pool is still reused as long as the same UGI instance is in use. But I agree it might be desirable to reuse the pool across queries or sessions for the same user. Since it's difficult for iceberg to assume how UGI instances are used, I think another option is to let the engines decide what kind of cache to use. I noticed that trino overrides the FileSystem cache in their own Hadoop fork, and uses only user names in their cache key [1][2]. Maybe we can also let users specify the cache to use in `HiveCatalog`? [1] https://github.com/trinodb/trino-hadoop-apache/blob/master/src/main/java/org/apache/hadoop/fs/FileSystemManager.java#L30 [2] https://github.com/trinodb/trino/blob/c28f676525bc426201302e71f4d9a978bb76a84b/lib/trino-hdfs/src/main/java/io/trino/hdfs/TrinoFileSystemCache.java#L254 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
