Sometimes we get warnings on shutdown. I've pinned down the cause and would like your opinions before I start working on a patch.
In PhoenixDriver.close(), the underlying connectionQueryServices are closed *before* the ExecutorService instance is closed. Because of this, tasks that are running in the background (on the executorService) can fail: - connectionQueryServices closes the hconnection - the still-running task (which is doing a hbase scan) gets errors because it was still using the hconnection. To fix this I would suggest changing the order (and using ExecutorService.awaitTermination()), so we would get something like this: // See ExecutorService javadocs for implementation - timeout could be a bit lower? shutdownAndAwaitTermination(services.getExecutor()) SQLCloseables.closeAll(connectionQueryServices) There is one small window in which a user could still use connections while the executorservice is being stopped but it's an error to use a connection after closing the driver anyway. (And it's rather unlikely someone will do something like that in the most common case where the whole thing is run in a shutdown hook anyway). WDYT? Karel Example case: Note: The stacktrace below comes from Phoenix 4.1 + some patches. In versions 4.2 and higher this particular error will not happen anymore because table stats are obtained synchronously in 4.2+ (AFAICT), but I think the point remains. 15/01/29 12:04:19 WARN org.apache.hadoop.hbase.client.ScannerCallable: Ignore, probably already closed org.apache.hadoop.hbase.ipc.StoppedRpcClientException at org.apache.hadoop.hbase.ipc.RpcClient.getConnection(RpcClient.java:1519) at org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1438) at org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1657) at org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1715) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:29900) at org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:285) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:153) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:57) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90) at org.apache.hadoop.hbase.client.ClientScanner.close(ClientScanner.java:431) at org.apache.phoenix.query.StatsManagerImpl.updateStats(StatsManagerImpl.java:108) at org.apache.phoenix.query.StatsManagerImpl$1.call(StatsManagerImpl.java:151) at org.apache.phoenix.query.StatsManagerImpl$1.call(StatsManagerImpl.java:147) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
