Hey Ryan, Writing a 95MB file to Riak does sound a bit frightening, but it's hard to say for sure if that's the cause...
So things are running again? Also, can you send along your crash.log and error.log? That'll be more helpful for starters. Mark On Wed, Mar 20, 2013 at 9:40 AM, Ryan Flynn <[email protected]> wrote: > Hey guys, > > Managed to hard-crash 8 of the our 10 riak 1.2.1 nodes the other day. > In bowling I believe that would be a 7-10 split. > I was told on irc this was the place to report crashes so I've attached a > crash dump, config file and host-info from one of the affected nodes. > Our server loads up user data from riak and flushes it back periodically, > flushing all dirty data back to riak on service shutdown. > Shutting down the service flushed back ~150MB of data, consisting of a > single ~95 MB file and a smattering of other 1-2MB files. > Riak's distaste for larger files is documented, but I didn't expect > catastrophic results. > The only other unusual thing I can think of is that process doing the 95MB > flush was likely killed by its supervisor mid-flush due to a timeout > intended to be 60 seconds, but erroneously expressed as 60 milliseconds. > Since then I've corrected the supervisor timeout and begun compressing > files. > As the riak champion in my company I'd love to figure out why this happened > and make sure it doesn't again :) > > Ryan > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
