[
https://issues.apache.org/jira/browse/HBASE-18396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16222848#comment-16222848
]
Andrew Purtell commented on HBASE-18396:
----------------------------------------
FWIW we have a replication backup and related issues currently producing
actually observed jute messages of ~26MB. Encoding can help relieve pressure
until we fix the underlying cause. HBase shouldn't manage replication state in
ZooKeeper, full stop. We have other JIRAs in place proposing a migration to
HBase hosted system tables. While doing that we can work on keeping queue state
tracking scalable as number of queues and WALs increase. I think that's the
long term direction. In the interim maybe we can consider half measures like
this as pain relief.
> Encode ZNode names to reduce ZooKeeper jute buffer length requirements and
> thus reduce memory usage
> ---------------------------------------------------------------------------------------------------
>
> Key: HBASE-18396
> URL: https://issues.apache.org/jira/browse/HBASE-18396
> Project: HBase
> Issue Type: Improvement
> Affects Versions: 3.0.0
> Reporter: Karan Mehta
>
> In our production environment, we hit the error {{ZooKeeper connectionLoss
> due to jute.maxbuffer len of 1M getting exceeded}}. Usually 1 MB is a lot,
> but in case of multi requests, it can exceed the maximum buffer length that
> is allocated.
> This JIRA is a discussion for encoding various znode names. IMO, this will
> reduce the path lengths, thus reducing the size of buffer required as well as
> network packet size and also pack more requests in a single multi. As with
> encoding, this will introduce overhead, but we need to determine how feasible
> this idea is.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
