I like this idea, but we would still need to support bucketizing either way because we cannot guarantee that the compressed version will be compact enough for every use case.
What types of compression schemes are you planning to support? ---------------------------------------- > Date: Sat, 7 Mar 2015 22:30:15 -0800 > Subject: Use compression to store data in ZK > From: [email protected] > To: [email protected] > > Hi, > > Currently we have bucketing as one of the options when the number of > partitions are large. We have couple of bugs with the handling of > bucketized resources (one of them is fatal). > > One of the reasons to split the znode is because we use JSON to store the > data in ZNode. While JSON is good for debugging, its space inefficient. > > A better option before going to bucketing is to support compression of > Ideal state, current state and External View. This also gives good > performance. > > I plan to add this support and make it configurable. Feedback/suggestions > > thanks, > Kishore G
