Thanks for the version information. Do you have any log entries like "bitcask_fileops:fold: <filename>: expected 14 bytes but got only <some number> bytes, skipping\n",
That will tell you more accurately which file is causing the problem. It'll take me a little while to check if I have anything to dump the bucket/keys. On Thu, Oct 10, 2013 at 8:14 AM, Sebastian Gerlach < [email protected]> wrote: > Hi Jon, > > thanks for the quick replay and the offerred solution. we are running > the debian package version 1.2.0-1. > > Is there a way to get a list of keys witch are in this files. To trigger > a read. We have over 25.000.000 Million records in riak an it's would be > grat to receive the richt ones. Not all. > > Thanks in advanced. > > Best regards > Sebastian > > > On 10/10/2013 04:09 PM, Jon Meredith wrote: > > Hi Sebastian, > > > > Sorry the log file is a little opaque. Bitcask is having problems > > running the merge > > on > /var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/4.bitcask.data > > and > /var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/29.bitcask.data. > > The zero length files are being created by the failed merges. > > > > Something is corrupting the internal merge state structure. What > > version of bitcask and riak are you running? > > > > If you want some immediate reliefe, I would move the > > 4.bitcask.data, 4.bitcask.data, 29.bitcask.hint and 29.bitcask.hint > > files out of the bitcask directory to somewhere safe and remove all the > > zero-length .data and .hint files and restart the node. Read repair > > will fix the objects over time, or you could do more active repair by > > retrieving all the objects. More recent versions of Riak have a manual > > repair facility built in, but I'm guessing you're running something > > before 1.2. > > > > Jon > > > > > > > > > > > > On Thu, Oct 10, 2013 at 7:05 AM, Sebastian Gerlach > > <[email protected] <mailto:[email protected]>> > wrote: > > > > Hello, > > > > we have a 12 node riak cluster. Since a while we have a log of > messages > > like this: > > > > "2013-10-10 00:22:04.725 [error] <0.5743.2160> Failed to merge > > > ["/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056",[{async_folds,true},{vnode_vclocks,true},{included_applications,[]},{allow_strfun,false},{reduce_js_vm_count,6},{storage_backend,riak_kv_bitcask_backend},{legacy_keylisting,false},{hook_js_vm_count,2},{listkeys_backpressure,true},{mapred_name,"mapred"},{stats_urlpath,"stats"},{legacy_stats,true},{js_thread_stack,16},{add_paths,[]},{http_url_encoding,on},{map_js_vm_count,8},{mapred_2i_pipe,true},{mapred_system,pipe},{js_max_vm_mem,8},{read_write,true}],["/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/29.bitcask.data","/var/lib/riak/bitcask/276886052384956216925307478073202056458252845056/4.bitcask.data"]]: > > > {{badrecord,mstate},[{bitcask,merge_files,1,[{file,"src/bitcask.erl"},{line,835}]},{bitcask,merge1,3,[{file,"src/bitcask.erl"},{line,509}]},{bitcask_merge_worker,do_merge,1,[{file,"src/bitcask_merge_worker.erl"},{line,130}]}]} > > " > > > > The provides directory have a lot of files with a size of zero. And > the > > number of files is much higher (2821 files) than in other directorys > (30 > > files). > > > > Does anyone has a solution or a hint. > > > > Thanks a lot an best regards > > Sebastian > > > > _______________________________________________ > > riak-users mailing list > > [email protected] <mailto:[email protected]> > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > > > > > > -- > > Jon Meredith > > VP, Engineering > > Basho Technologies, Inc. > > [email protected] <mailto:[email protected]> > > -- Jon Meredith VP, Engineering Basho Technologies, Inc. [email protected]
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
