Hi Sebastian, Sorry the log file is a little opaque. Bitcask is having problems running the merge on /var/lib/riak/bitcask/276886052384956216925307478073 202056458252845056/4.bitcask.data and /var/lib/riak/bitcask/ 276886052384956216925307478073202056458252845056/29.bitcask.data. The zero length files are being created by the failed merges.
Something is corrupting the internal merge state structure. What version of bitcask and riak are you running? If you want some immediate reliefe, I would move the 4.bitcask.data, 4.bitcask.data, 29.bitcask.hint and 29.bitcask.hint files out of the bitcask directory to somewhere safe and remove all the zero-length .data and .hint files and restart the node. Read repair will fix the objects over time, or you could do more active repair by retrieving all the objects. More recent versions of Riak have a manual repair facility built in, but I'm guessing you're running something before 1.2. Jon On Thu, Oct 10, 2013 at 7:05 AM, Sebastian Gerlach < [email protected]> wrote: > Hello, > > we have a 12 node riak cluster. Since a while we have a log of messages > like this: > > "2013-10-10 00:22:04.725 [error] <0.5743.2160> Failed to merge > > ["/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056",[{async_folds,true},{vnode_vclocks,true},{included_applications,[]},{allow_strfun,false},{reduce_js_vm_count,6},{storage_backend,riak_kv_bitcask_backend},{legacy_keylisting,false},{hook_js_vm_count,2},{listkeys_backpressure,true},{mapred_name,"mapred"},{stats_urlpath,"stats"},{legacy_stats,true},{js_thread_stack,16},{add_paths,[]},{http_url_encoding,on},{map_js_vm_count,8},{mapred_2i_pipe,true},{mapred_system,pipe},{js_max_vm_mem,8},{read_write,true}],["/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/29.bitcask.data","/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/4.bitcask.data"]]: > > {{badrecord,mstate},[{bitcask,merge_files,1,[{file,"src/bitcask.erl"},{line,835}]},{bitcask,merge1,3,[{file,"src/bitcask.erl"},{line,509}]},{bitcask_merge_worker,do_merge,1,[{file,"src/bitcask_merge_worker.erl"},{line,130}]}]} > " > > The provides directory have a lot of files with a size of zero. And the > number of files is much higher (2821 files) than in other directorys (30 > files). > > Does anyone has a solution or a hint. > > Thanks a lot an best regards > Sebastian > > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > -- Jon Meredith VP, Engineering Basho Technologies, Inc. [email protected]
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
