Camille's suggestion is good. You could also consider grouping lots of your records into single files. Doing atomic updates to these files would give you much of the same functionality as raw ZK except for ephemerals and your deletions could be hundreds of times faster. This would increase notification traffic, but that may not matter if you arrange your records cleverly or are not using notifications all that much. An example of how you could arrange things might be if you have many processes listening for updates to the same records, then grouping those records into a single file could actually decrease the notification traffic since multiple updates could happen before the next notification.
If you don't need ephemerals or change notification, you might also consider using ZK simply to adjudicate a master server for replicated high performance storage. That is what we do in the MapR system. This allows more flexible survival policies than ZK can allow and also allows vastly higher write performance since we can relax some of the guarantees that ZK provides such as strict ordering of all transactions. Finally, if your records don't need strict survival guarantees, you might take a look into putting ZK on a tmpfs or relaxing the synchronous nature of commits to disk. There was a posting recently on this that demonstrated an order of magnitude or two improvement in performance for some unspecified task. The cost here is relaxation of persistence guarantees which ranges from guaranteed loss (with tmpfs) to possible loss (with libeatmydata). Combining this with the multi operation could give you huge gains in performance. Can you say more about what you are doing with your many records? On Thu, Nov 22, 2012 at 9:37 AM, Camille Fournier <[email protected]>wrote: > Have you looked into the new multi-transaction features in 3.4? I think it > might be faster to group these into batches and do the delete that way. I > can't think of any other alternatives though, 18 hours is really rough. > > C > > > On Thu, Nov 22, 2012 at 10:13 AM, Brian Tarbox <[email protected] > >wrote: > > > We store hundreds of thousands of small records in a tree of zookeeper > > records. We sometimes need to delete an entire subtree and currently > > delete recursive takes about 18 hours to run. Is there a faster way to > > delete a subtree? > > > > Thanks. > > > > -- > > http://about.me/BrianTarbox > > >
