George created HBASE-27057:
------------------------------
Summary: Issue with ZooKeeper configuration on CDP5
Key: HBASE-27057
URL: https://issues.apache.org/jira/browse/HBASE-27057
Project: HBase
Issue Type: Task
Environment: Cloudera 5 with Kerberos / HBase
Reporter: George
Hello everyone, I am new here and in HBase as well, so sorry for my lack of
knowledge, I tried to read but the documentation and the environment that I got
as a test are confusing me. The explanation is here, I have got Cloudera 5 with
Hbase with Kerberos, 2 master nodes, and 3 worker nodes on Ubuntu machines, we
have to connect DataStore with special application with checks krb5.conf file
and keytab after authentication goes and connect to DS and check the tables the
metadata there. When I am trying to push the application to connect the master
node zookeeper is failing, when I am trying to connect zookeper node from the
same application I can scan and do what I need, but every time the connection
goes to 2181 localhost, how to check where is the issue of configuration that
masters done will give me access to the data stores?
Here is log from master node:
[2022-05-22 12:26:31.676Z] [WARN] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper [MasterNode]- Possibly
transient ZooKeeper, quorum=10.2.0.30:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
[2022-05-22 12:26:31.676Z] [INFO] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.util.RetryCounter [MasterNode]- Sleeping 1000ms before
retry #0...
[2022-05-22 12:26:31.676Z] [WARN] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper [MasterNode]- Possibly
transient ZooKeeper, quorum=10.2.0.30:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
[2022-05-22 12:26:31.676Z] [INFO] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.util.RetryCounter [MasterNode]- Sleeping 2000ms before
retry #1...
[2022-05-22 12:26:31.676Z] [WARN] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper [MasterNode]- Possibly
transient ZooKeeper, quorum=10.2.0.30:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase/hbaseid
[2022-05-22 12:26:31.676Z] [ERROR] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper [MasterNode]- ZooKeeper
exists failed after 2 attempts
[2022-05-22 12:26:31.677Z] [WARN] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.ZKUtil [MasterNode]-
hconnection-0x46acd8780x0, quorum=10.2.0.30:2181, baseZNode=/hbase Unable to
set watcher on znode (/hbase/hbaseid)
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
= ConnectionLoss for /hbase/hbaseid
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:222)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
at
org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:833)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:623)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
at
com.bigid.scanner.hbase.HBaseScanner.getDataSourceObjects(HBaseScanner.java:60)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:108)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:105)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper.getDataSourceObjects(HadoopUGIConnectorWrapper.java:105)
at
com.bigid.scanner.connectors.generic.DataSourceConnectorAdapter.testConnection(DataSourceConnectorAdapter.java:289)
at
com.bigid.scanner.core.tasks.legacy.LegacyTestConnectionTask.execute(LegacyTestConnectionTask.java:43)
at
com.bigid.scanner.core.TestConnectionTask.execute(TestConnectionTask.java:36)
at com.bigid.scanner.core.TaskExecution.execute(TaskExecution.java:36)
at
com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:57)
at
com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:20)
at
com.bigid.scanner.scannerboot.common.http.managers.TestConnectionSyncManager.lambda$runService$0(TestConnectionSyncManager.java:55)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
[2022-05-22 12:26:31.678Z] [ERROR] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher [MasterNode]-
hconnection-0x46acd8780x0, quorum=10.2.0.30:2181, baseZNode=/hbase Received
unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
= ConnectionLoss for /hbase/hbaseid
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:222)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
at
org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:833)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:623)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
at
com.bigid.scanner.hbase.HBaseScanner.getDataSourceObjects(HBaseScanner.java:60)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:108)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:105)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper.getDataSourceObjects(HadoopUGIConnectorWrapper.java:105)
at
com.bigid.scanner.connectors.generic.DataSourceConnectorAdapter.testConnection(DataSourceConnectorAdapter.java:289)
at
com.bigid.scanner.core.tasks.legacy.LegacyTestConnectionTask.execute(LegacyTestConnectionTask.java:43)
at
com.bigid.scanner.core.TestConnectionTask.execute(TestConnectionTask.java:36)
at com.bigid.scanner.core.TaskExecution.execute(TaskExecution.java:36)
at
com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:57)
at
com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:20)
at
com.bigid.scanner.scannerboot.common.http.managers.TestConnectionSyncManager.lambda$runService$0(TestConnectionSyncManager.java:55)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
[2022-05-22 12:26:31.678Z] [WARN] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.client.ZooKeeperRegistry [MasterNode]- Can't retrieve
clusterId from Zookeeper
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
= ConnectionLoss for /hbase/hbaseid
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:222)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
at
org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:65)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:86)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:833)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.<init>(ConnectionManager.java:623)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown
Source)
at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
at
org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
at
com.bigid.scanner.hbase.HBaseScanner.getDataSourceObjects(HBaseScanner.java:60)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:108)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:105)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper.getDataSourceObjects(HadoopUGIConnectorWrapper.java:105)
at
com.bigid.scanner.connectors.generic.DataSourceConnectorAdapter.testConnection(DataSourceConnectorAdapter.java:289)
at
com.bigid.scanner.core.tasks.legacy.LegacyTestConnectionTask.execute(LegacyTestConnectionTask.java:43)
at
com.bigid.scanner.core.TestConnectionTask.execute(TestConnectionTask.java:36)
at com.bigid.scanner.core.TaskExecution.execute(TaskExecution.java:36)
at
com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:57)
at
com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:20)
at
com.bigid.scanner.scannerboot.common.http.managers.TestConnectionSyncManager.lambda$runService$0(TestConnectionSyncManager.java:55)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
[2022-05-22 12:26:36.337Z] [INFO] scanner_file -
7a342cd4-cd4f-4a08-ab9f-8214b4f62f09 [poolScheduler-2]
com.bigid.scanner.scannerboot.common.services.context.threads.ThreadPoolUpdatesService
- Fetching thread pool updates
[2022-05-22 12:26:41.675Z] [WARN] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper [MasterNode]- Possibly
transient ZooKeeper, quorum=10.2.0.30:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
[2022-05-22 12:26:41.676Z] [INFO] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.util.RetryCounter [MasterNode]- Sleeping 1000ms before
retry #0...
[2022-05-22 12:26:41.676Z] [WARN] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper [MasterNode]- Possibly
transient ZooKeeper, quorum=10.2.0.30:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
[2022-05-22 12:26:41.676Z] [INFO] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.util.RetryCounter [MasterNode]- Sleeping 2000ms before
retry #1...
[2022-05-22 12:26:41.676Z] [WARN] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper [MasterNode]- Possibly
transient ZooKeeper, quorum=10.2.0.30:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
[2022-05-22 12:26:41.676Z] [ERROR] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper [MasterNode]- ZooKeeper
exists failed after 2 attempts
[2022-05-22 12:26:41.676Z] [WARN] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.ZKUtil [MasterNode]-
hconnection-0x46acd8780x0, quorum=10.2.0.30:2181, baseZNode=/hbase Unable to
set watcher on znode (/hbase)
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
= ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:222)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:849)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:535)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1435)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1476)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1505)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1711)
at
org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3601)
at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:312)
at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:296)
at
com.bigid.scanner.hbase.HBaseScanner.getDataSourceObjects(HBaseScanner.java:62)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:108)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:105)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper.getDataSourceObjects(HadoopUGIConnectorWrapper.java:105)
at
com.bigid.scanner.connectors.generic.DataSourceConnectorAdapter.testConnection(DataSourceConnectorAdapter.java:289)
at
com.bigid.scanner.core.tasks.legacy.LegacyTestConnectionTask.execute(LegacyTestConnectionTask.java:43)
at
com.bigid.scanner.core.TestConnectionTask.execute(TestConnectionTask.java:36)
at com.bigid.scanner.core.TaskExecution.execute(TaskExecution.java:36)
at
com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:57)
at
com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:20)
at
com.bigid.scanner.scannerboot.common.http.managers.TestConnectionSyncManager.lambda$runService$0(TestConnectionSyncManager.java:55)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
[2022-05-22 12:26:41.676Z] [ERROR] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher [MasterNode]-
hconnection-0x46acd8780x0, quorum=10.2.0.30:2181, baseZNode=/hbase Received
unexpected KeeperException, re-throwing exception
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
= ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:222)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:849)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:535)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1435)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1476)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1505)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1711)
at
org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3601)
at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:312)
at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:296)
at
com.bigid.scanner.hbase.HBaseScanner.getDataSourceObjects(HBaseScanner.java:62)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:108)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:105)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper.getDataSourceObjects(HadoopUGIConnectorWrapper.java:105)
at
com.bigid.scanner.connectors.generic.DataSourceConnectorAdapter.testConnection(DataSourceConnectorAdapter.java:289)
at
com.bigid.scanner.core.tasks.legacy.LegacyTestConnectionTask.execute(LegacyTestConnectionTask.java:43)
at
com.bigid.scanner.core.TestConnectionTask.execute(TestConnectionTask.java:36)
at com.bigid.scanner.core.TaskExecution.execute(TaskExecution.java:36)
at
com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:57)
at
com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:20)
at
com.bigid.scanner.scannerboot.common.http.managers.TestConnectionSyncManager.lambda$runService$0(TestConnectionSyncManager.java:55)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
[2022-05-22 12:26:41.676Z] [ERROR] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation
[MasterNode]- Can't get connection to ZooKeeper: KeeperErrorCode =
ConnectionLoss for /hbase
[2022-05-22 12:26:41.676Z] [WARN] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper [MasterNode]- Possibly
transient ZooKeeper, quorum=10.2.0.30:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
[2022-05-22 12:26:41.676Z] [INFO] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.util.RetryCounter [MasterNode]- Sleeping 1000ms before
retry #0...
[2022-05-22 12:26:41.676Z] [INFO] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [poolScheduler-7]
com.bigid.scanner.scannerboot.common.services.context.threads.ThreadPoolUpdatesService
- Fetching thread pool updates
[2022-05-22 12:26:41.676Z] [WARN] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper [MasterNode]- Possibly
transient ZooKeeper, quorum=10.2.0.30:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
[2022-05-22 12:26:41.676Z] [INFO] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.util.RetryCounter [MasterNode]- Sleeping 2000ms before
retry #1...
[2022-05-22 12:26:41.676Z] [WARN] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper [MasterNode]- Possibly
transient ZooKeeper, quorum=10.2.0.30:2181,
exception=org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
[2022-05-22 12:26:41.676Z] [ERROR] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper [MasterNode]- ZooKeeper
exists failed after 2 attempts
[2022-05-22 12:26:41.676Z] [WARN] scanner_file -
12750b5c-c00b-4eb9-8ec0-66a710ed822d [Thread-312]
org.apache.hadoop.hbase.zookeeper.ZKUtil [MasterNode]-
hconnection-0x46acd8780x0, quorum=10.2.0.30:2181, baseZNode=/hbase Unable to
set watcher on znode (/hbase)
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
= ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.exists(RecoverableZooKeeper.java:222)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.checkExists(ZKUtil.java:481)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.checkIfBaseNodeAvailable(ConnectionManager.java:849)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.access$400(ConnectionManager.java:535)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1435)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1476)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1505)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1711)
at
org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124)
at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3601)
at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:312)
at org.apache.hadoop.hbase.client.HBaseAdmin.listTables(HBaseAdmin.java:296)
at
com.bigid.scanner.hbase.HBaseScanner.getDataSourceObjects(HBaseScanner.java:62)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:108)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper$4.run(HadoopUGIConnectorWrapper.java:105)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
at
com.bigid.scanner.hadoop.common.HadoopUGIConnectorWrapper.getDataSourceObjects(HadoopUGIConnectorWrapper.java:105)
at
com.bigid.scanner.connectors.generic.DataSourceConnectorAdapter.testConnection(DataSourceConnectorAdapter.java:289)
at
com.bigid.scanner.core.tasks.legacy.LegacyTestConnectionTask.execute(LegacyTestConnectionTask.java:43)
at
com.bigid.scanner.core.TestConnectionTask.execute(TestConnectionTask.java:36)
at com.bigid.scanner.core.TaskExecution.execute(TaskExecution.java:36)
at
com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:57)
at
com.bigid.scanner.scannerboot.common.services.TestConnectionService.doService(TestConnectionService.java:20)
at
com.bigid.scanner.scannerboot.common.http.managers.TestConnectionSyncManager.lambda$runService$0(TestConnectionSyncManager.java:55)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(Unknown
Source)
at java.base/java.lang.Thread.run(Unknown Source)
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
