Mike Drob commented on HBASE-18396:

I don't know the exact sizes of paths, data, how many requests come in a batch, 
etc, so this is all going to be hypothetical.

Say we have a typical ZK path of 100 bytes (and no data) and we're able to 
reduce that to 40 bytes. I don't expect to be able to go much further since I 
think there still need to be IDs or other non-colliding bits that exist, but 
say we map "replication" to "r" and so on. Theoretically, what used to take 1M 
will now only take 400K! That's a huge win!

But! You go on to say that we can use this win to fit more requests in a single 
buffer. Well, then you'll have the same jute.maxbuffer len error, since it 
doesn't matter how big you make your buffer or how small you make the items if 
you keep adding until it overflows. I think saving network is a fine goal, but 
at the end of the day it's much more valuable to address the cause of the 
buffer exceeding instead of trying to optimize away the problems.

Also, most systems I interact with have increased the maxbuffer to 4M and are 
running without issue.

> Encode ZNode names to reduce ZooKeeper jute buffer length requirements and 
> thus reduce memory usage
> ---------------------------------------------------------------------------------------------------
>                 Key: HBASE-18396
>                 URL: https://issues.apache.org/jira/browse/HBASE-18396
>             Project: HBase
>          Issue Type: Improvement
>    Affects Versions: 3.0.0
>            Reporter: Karan Mehta
> In our production environment, we hit the error {{ZooKeeper connectionLoss 
> due to jute.maxbuffer len of 1M getting exceeded}}. Usually 1 MB is a lot, 
> but in case of multi requests, it can exceed the maximum buffer length that 
> is allocated.
> This JIRA is a discussion for encoding various znode names. IMO, this will 
> reduce the path lengths, thus reducing the size of buffer required as well as 
> network packet size and also pack more requests in a single multi. As with 
> encoding, this will introduce overhead, but we need to determine how feasible 
> this idea is.

This message was sent by Atlassian JIRA

Reply via email to