Thank you. Wellington. I ran the snapshotFormatter too one of the snapshot files under zookeeper data directory ( there were a lot of them and each of them 98MB in size).
The snapshot formatter reveals data the ZNode Count is 103,000 and out of which 90,000 of them correspond to /hbase/replication/ - I suspect the next step is to find root cause for hbase replication lagging Last login: Fri Mar 15 21:47:03 2019 from cmdev.c.autofocus-1.internal [azafir@dn1 ~]$ java -cp /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/zookeeper-3.4.5-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/zookeeper/lib/* org.apache.zookeeper.server.SnapshotFormatter /home/azafir/snapshot.1007d594a7 | grep -E -i -w 'rs' | wc -l 90420 [azafir@dn1 ~]$ java -cp /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/zookeeper-3.4.5-cdh5.14.2.jar:/opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/zookeeper/lib/* org.apache.zookeeper.server.SnapshotFormatter /home/azafir/snapshot.1007d594a7 | grep -A3 -B3 count ZNode Details (*count*=103974): ---- / cZxid = 0x00000000000000 -- ephemeralOwner = 0x00000000000000 dataLength = 31 ---- /hbase/table/ cZxid = 0x00000c000403ee ctime = Thu Aug 16 16:12:25 UTC 2018 mZxid = 0x00000c0004f53e -- ephemeralOwner = 0x00000000000000 dataLength = 31 ---- /hbase/table/ cZxid = 0x00000c0003f861 ctime = Thu Aug 16 16:11:40 UTC 2018 mZxid = 0x00000c0004dc14 -- ephemeralOwner = 0x00000000000000 dataLength = 31 ---- /hbase/table/xxxx cZxid = 0x00000c00060d4d ctime = Sat Aug 18 00:19:22 UTC 2018 mZxid = 0x00000c00067730 -- ephemeralOwner = 0x00000000000000 dataLength = 31 ---- /hbase/table/xxxxx cZxid = 0x00000d266fb835 ctime = Thu Dec 06 18:47:58 UTC 2018 mZxid = 0x00000d266fcdf9 -- ephemeralOwner = 0x00000000000000 dataLength = 31 ---- /hbase/table/xxxxxx cZxid = 0x00000c00060d42 ctime = Sat Aug 18 00:19:09 UTC 2018 mZxid = 0x00000c00067763 -- ephemeralOwner = 0x00000000000000 dataLength = 0 ---- /hbase/table-lock/xxxx cZxid = 0x00000600004491 ctime = Fri Jun 29 17:24:02 UTC 2018 mZxid = 0x00000600004491 -- ephemeralOwner = 0x00000000000000 dataLength = 0 ---- /hbase/table-lock/wf_artifact*count* cZxid = 0x0000060000c927 ctime = Fri Jun 29 18:03:40 UTC 2018 mZxid = 0x0000060000c927 -- ephemeralOwner = 0x00000000000000 dataLength = 0 ---- /hbase/table-lock/wf_invalid_artifact_*count* cZxid = 0x00000c00060d4b ctime = Sat Aug 18 00:19:22 UTC 2018 mZxid = 0x00000c00060d4b -- ephemeralOwner = 0x00000000000000 dataLength = 0 ---- /hbase/table-lock/wf_artifact*count*_2018_12_05 cZxid = 0x00000d266fb531 ctime = Thu Dec 06 18:47:56 UTC 2018 mZxid = 0x00000d266fb531 -- ephemeralOwner = 0x00000000000000 dataLength = 0 ---- /hbase/table-lock/xxxx cZxid = 0x00000c00060d40 ctime = Sat Aug 18 00:19:09 UTC 2018 mZxid = 0x00000c00060d40 -- ephemeralOwner = 0x00000000000000 dataLength = 0 ---- /hbase/online-snapshot/abort/xxxxx cZxid = 0x00001003a90aad ctime = Fri Mar 08 22:36:36 UTC 2019 mZxid = 0x00001003a90aad -- ephemeralOwner = 0x00000000000000 dataLength = 509 ---- /hbase/online-snapshot/abort/xxxxx cZxid = 0x000010046faf65 ctime = Sat Mar 09 22:38:04 UTC 2019 mZxid = 0x000010046faf65 On Tue, Mar 12, 2019 at 6:04 AM Wellington Chevreuil < [email protected]> wrote: > Slow replication can lead to too many znodes in ZK. These would not be > direct children of "/hbase/replication/rs" znode, but of specific RSes > replication queue. The "108" showed on your "stat" command output is only > the number of RSes, and would not vary, but further znodes under each of > those would. To have a better picture of how large is your znode tree, you > should rather use ZK org.apache.zookeeper.server.SnapshotFormatter tool. > This will print the whole structure, from a given snapshot file as input. > Also, there may be some further hints about which znode is exceeding the > buffer limits by looking at the full stack trace from the jute buffer > error. Would you be able to share those? > > Em ter, 12 de mar de 2019 às 09:52, Asim Zafir <[email protected]> > escreveu: > >> >> Hi Weillington, >> >> Thanks for the response. I greatly appreciate. Yes we have replication >> enabled and if I stat /hbase/replication/rs I get the following >> >> >> cZxid = 0x1000001c8 >> >> ctime = Tue Jun 19 22:23:17 UTC 2018 >> >> mZxid = 0x1000001c8 >> >> mtime = Tue Jun 19 22:23:17 UTC 2018 >> >> pZxid = 0x10000cd555 >> >> cversion = 8050 >> >> dataVersion = 0 >> >> aclVersion = 0 >> >> ephemeralOwner = 0x0 >> >> dataLength = 0 >> >> numChildren = 108 >> >> how should I be able to see analyze the znode utilzation in this case and >> more specifiically how is it impacting the jute-buffer size.. I can see >> numChild nodes under /hbase/replication is 108 but how does it correspond >> to zk jute buffer reaching max value? >> >> Also it is not clear the timestamp on this znodes isnt' incrementing. I >> see the timestamp is still showing 2018 date. >> >> Thanks, >> asim >> >> >> -------------->>>>> >> >> >> >> This jute buffer len error generally means a given znode being watched/read >> had grown too large to fit into the buffer. It's not specific to number of >> watches attached, but amount of info stored in it, for example, too many >> children znode under a given znode. In order to understand what's behind >> the error, you should analyse your zookeeper znodes tree, you may have a >> hint by looking at zookeeper snapashot files. Would you have replication >> enabled on this cluster? A common cause for such errors in hbase is when >> replication is slow/stuck, and source cluster is under heavy write load, >> causing replication queue to grow much faster than it's ability to drain, >> which will imply on many znodes created under "replication" znode. >> >>
