We're not entirely sure of the exact timeline. But here's what we think happened. We were building a new cluster, and put a common config on both boxes with default n_val = 3. We then manually changed that config on both boxes to n_val=1. We restarted riak on one of the nodes, but forgot to restart on the other so it's n_val=3 was still in effect.
That's our best guess. Dave -- Dave Lowell [email protected] On Dec 18, 2012, at 3:17 PM, Mark Phillips wrote: > Hey David, > > So the short answer is that we don't have anything in place that will > complain wildly if this happens. This, however, should not be the > case. Before I go file an issue, out of curiosity, how did this > happen? Did you set the n_val differently in the app.config of each > node before you started them up? > > Mark > > > On Fri, Dec 14, 2012 at 8:03 PM, David Lowell <[email protected]> wrote: >> I know this is really very boneheaded. But we ended up with n_val being 3 on >> one of two riak hosts in a dev cluster, and 1 on the other host. I would >> have expected Riak to howl loudly about this, but it said nothing. In fact, >> we discovered the problem while debugging app-level issues that resulted >> from this configuration. >> >> We found that on the two hosts, the same 2i query returned wildly different >> results. As we investigated we discovered that on the host with n=1, riak >> was returning each key 3 times. On the host with n=3, each key was in the 2i >> results once. >> >> Maybe this makes sense at some low level, but I sure would have preferred >> something in the logs, or even a riak crash to this behavior. >> >> Once we corrected n_val on the errant host, both hosts returned each key 3x. >> Is it possible for riak to recover from this situation and eventually return >> the correct 2i results with each key listed only once? >> >> This is a dev cluster, so this isn't critical. But I'd like to understand >> better whether and how riak can recover from this weird state. >> >> Thanks, >> >> Dave >> >> -- >> Dave Lowell >> [email protected] >> >> >> _______________________________________________ >> riak-users mailing list >> [email protected] >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
