szehon-ho opened a new issue, #5371:
URL: https://github.com/apache/iceberg/issues/5371
As discovered by @flyrain , sometimes hits this issue occurs in ClientPool.
```
java.lang.IllegalStateException: Cannot get a client from a closed pool
at
org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkState(Preconditions.java:502)
at org.apache.iceberg.ClientPoolImpl.get(ClientPoolImpl.java:118)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:56)
at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51)
at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76)
at
org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:203)
at
org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:95)
at
org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:78)
at
org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:44)
at org.apache.iceberg.spark.SparkCatalog.load(SparkCatalog.java:710)
at org.apache.iceberg.spark.SparkCatalog.loadTable(SparkCatalog.java:156)
at org.apache.iceberg.spark.SparkCatalog.loadTable(SparkCatalog.java:111)
at
org.apache.iceberg.spark.Spark3Util.loadIcebergTable(Spark3Util.java:648)
```
The problem comes from
[CachedClientPool](https://github.com/apache/iceberg/blob/master/hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java).
There, run() always asks the clientPoolCache (Caffeine) for new clientPool.
But clientPoolCache has been constructed with expireAfterAccess(), which calls
close() on the client pool after x time. Thus, we may hit the situation (as
described by @flyrain )
Thread A; clientPoolCache.get() creates client A
Thread B: clientPoolCache.get() comes, gets client A since A is not closed
Caffeine Thread: Close the client A by the removal listener
Thread B: Run the command, failed with above because the client since it is
closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
