On Thu, May 26, 2016 at 2:22 AM, Szekeres, Zoltan < [email protected]> wrote:
> Thanks for the response. > I'm trying to understand what operational stability issues would there be. > I'm also planning to do a latency test for requests at the time I'm > requesting getChildren on "/someparentNode". > > To give more detail I have a primary and secondary use-case: > My primary use-case includes having watchers on the children of > "/someparentNode" and requesting getChildren for > "/someparentNode/LotsOfChildNodesHere-N" (which only has a couple child > nodes). > My secondary use-case would be requesting the children of > "/someparentNode", which would be only occasionally for reporting purposes > (which has a lot of child nodes and probably won't be as much as 70k nodes, > but I hit the limit there). > > I'm looking for answers for the following questions: > What are the stability issues that you think might occur having lots of > nodes under one node, even if we read them rarely? > Can I reliable use the "jute.maxbuffer" system property on the client in > the future? > I don't see why not. We have it there as a safety mechanism and to corral folks towards using it "the right way". Although opinions on what "the right way" means may differ, which is probably why it's configurable. ;-) > Looking for answers whether the asymmetry of the default value on client > side and on server side is accidental or intentional. > See this jira, it's currently being discussed: https://issues.apache.org/jira/browse/ZOOKEEPER-2430 Patrick > > Any advice is much appreciated. > > Thanks, > Zoltan Szekeres > > -----Original Message----- > From: Tom Crayford [mailto:[email protected]] > Sent: Wednesday, May 25, 2016 12:06 PM > To: [email protected] > Cc: Gupta, Abhishek (IST); Hejj, Botond (Enterprise Infrastructure) > Subject: Re: Use-case with lots of child nodes > > Hi, > > I'd recommend rethinking your use case. Zookeeper isn't really great with > large amounts of data, nor does it handle high volumes of changes that well. > > It looks like setting that system property will work for now, but I'd > expect trying to use such a high volume of child nodes would have severe > consequences for operational stability. > > Thanks > > Tom Crayford > Heroku Kafka > > On Wednesday, 25 May 2016, Szekeres, Zoltan < > [email protected]> wrote: > > > Hi ZooKeeper users, > > > > I have a use-case where I need to create a very large number (~70,000) > > of child nodes under a parent. These nodes themselves contain no data > > and will only have a handful of child nodes themselves. > > e.g. > > /someparentNode/LotsOfChildNodesHere-1/ACoupleofNodesAtThisLevel > > /someparentNode/LotsOfChildNodesHere-2/ACoupleofNodesAtThisLevel > > ... > > /someparentNode/LotsOfChildNodesHere-70000/ACoupleofNodesAtThisLevel > > > > I've read > > (https://zookeeper.apache.org/doc/r3.4.8/zookeeperAdmin.html) > > there is a limit of 1 MB. But I hit the limit for the getChildren > > operation around 4 MB. I'm interested in what's causing the difference > in the limit. > > I was able to increase the 4MB limit by setting the "jute.maxbuffer" > > system property at the client. Can I reliably use this system property > > in the future to set the limit? > > > > Any advice is appreciated. > > > > Thank you, > > Zoltan Szekeres > > > > > > > > ________________________________ > > > > NOTICE: Morgan Stanley is not acting as a municipal advisor and the > > opinions or views contained herein are not intended to be, and do not > > constitute, advice within the meaning of Section 975 of the Dodd-Frank > > Wall Street Reform and Consumer Protection Act. If you have received > > this communication in error, please destroy all electronic and paper > > copies; do not disclose, use or act upon the information; and notify > > the sender immediately. Mistransmission is not intended to waive > > confidentiality or privilege. Morgan Stanley reserves the right, to > > the extent permitted under applicable law, to monitor electronic > > communications. This message is subject to terms available at the > following link: > > http://www.morganstanley.com/disclaimers If you cannot access these > > links, please notify us by reply message and we will send the contents > > to you. By messaging with Morgan Stanley you consent to the foregoing. > > > > > > -------------------------------------------------------------------------------- > > NOTICE: Morgan Stanley is not acting as a municipal advisor and the > opinions or views contained herein are not intended to be, and do not > constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall > Street Reform and Consumer Protection Act. If you have received this > communication in error, please destroy all electronic and paper copies; do > not disclose, use or act upon the information; and notify the sender > immediately. Mistransmission is not intended to waive confidentiality or > privilege. Morgan Stanley reserves the right, to the extent permitted under > applicable law, to monitor electronic communications. This message is > subject to terms available at the following link: > http://www.morganstanley.com/disclaimers. If you cannot access these > links, please notify us by reply message and we will send the contents to > you. By messaging with Morgan Stanley you consent to the foregoing.
