In order to share an HBase connection pool, we create an object Object Util { val HBaseConf = HBaseConfiguration.create val Connection= HConnectionManager.createConnection(HBaseConf) }
which would be shared among tasks on the same executor. e.g. val result = rdd.map(line => { val table = Util.Connection.getTable("user") ... } However, we don’t how to close the Util.Connection. If we write Util.Connection.close() in the main function, it’ll only run on the driver, not the executor. So, How to make sure every Connection closed before exist?