I have an Amazon EMR cluster with with the following configuration: 1 x Master node (m5.xlarge) - 4 vCore, 16 GiB memory, EBS Storage:64 GiB 2 x Core nodes (m4.large) - 2 vCore, 8 GiB memory EBS Storage:32 GiB
Software configuration: Release label:emr-5.33.1, Hadoop distribution:Amazon 2.10.1, Hive 2.3.7, Pig 0.17.0, Hue 4.9.0, Spark 2.4.7, Phoenix 4.14.3, HBase 1.4.13 While running some tests, I realised that my DDL stages on HBase such as create are extremely slow and take as much as 3-4 seconds. Selects or scans are fine and work without any issues. The instance has about 314 tables and there isn't even a lot of data in the system. Is this normal for a HBase system? Is there a way to get it down to sub second? Where do I start troubleshooting? This may not matter as this is a once off job, but it annoys me that something that only takes a few milliseconds on local or other servers takes over two seconds in this case. Any pointers will be greatly appreciated. As an example: [image: image.png] And: [image: image.png] I don't see anything specific in logs except for the following logs in Zookeeper: 2022-02-04 22:59:05,397 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0x100000742700074 type:create cxid:0x2e56e zxid:0x324f txntype:-1 reqpath:n/a Error Path:/hbase/table-lock/PerfTestTable Error:KeeperErrorCode = NoNode for /hbase/table-lock/PerfTestTable 2022-02-04 22:59:33,458 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0x100000742700074 type:create cxid:0x2e587 zxid:0x325a txntype:-1 reqpath:n/a Error Path:/hbase/table-lock/PerfTestTable2 Error:KeeperErrorCode = NoNode for /hbase/table-lock/PerfTestTable2 2022-02-04 22:59:41,460 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0x100000742700074 type:create cxid:0x2e5a0 zxid:0x3265 txntype:-1 reqpath:n/a Error Path:/hbase/table-lock/PerfTestTable3 Error:KeeperErrorCode = NoNode for /hbase/table-lock/PerfTestTable3 2022-02-04 22:59:53,256 [myid:] - INFO [ProcessThread(sid:0 cport:2181)::PrepRequestProcessor@653] - Got user-level KeeperException when processing sessionid:0x100000742700074 type:create cxid:0x2e5ba zxid:0x3270 txntype:-1 reqpath:n/a Error Path:/hbase/table-lock/PerfTestTable4 Error:KeeperErrorCode = NoNode for /hbase/table-lock/PerfTestTable4 Is this normal for HBase? If not, any pointers and help to troubleshoot will be greatly appreciated. Best regards, Sanket
