[
https://issues.apache.org/jira/browse/SOLR-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17685891#comment-17685891
]
Justin Sweeney commented on SOLR-16608:
---------------------------------------
I'm planning an additional PR as part of this ticket to ensure that we have CLI
tools that will appropriately interact with compressed data to both retrieve
and store compressed state in Zookeeper in case someone needs to edit state.
The core functionality has been merged though.
As far as performance I don't yet have hard numbers. The
compression/decompression is typically very fast, on the order of a few
milliseconds. The gain in performance will depend a bit on network speeds, but
there is the added benefit of reducing strain on Zookeeper by keeping files
small.
> Enable ability to compress state.json in Zookeeper
> --------------------------------------------------
>
> Key: SOLR-16608
> URL: https://issues.apache.org/jira/browse/SOLR-16608
> Project: Solr
> Issue Type: Improvement
> Reporter: Justin Sweeney
> Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> As noted in the Solr documentation, Zookeeper is intended to hold small
> files, by default < 1MB:
> [https://solr.apache.org/guide/7_4/setting-up-an-external-zookeeper-ensemble.html#increasing-the-file-size-limit.]
> When creating collections with very large number of replicas, it is possible
> to exceeded that size in the state.json that is created, which then requires
> Solr operators to reconfigure Zookeeper to increase that limit and also has
> performance implications.
> I am proposing that we provide the ability to compress state.json using
> standard ZLib compression. This will be based on a configuration parameter in
> solr.xml which will provide a minimum size of state.json over which to use
> compression. This will give Solr operators a better mechanism to handle
> state.json for very large collections.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
