Hi Magnus
RIAK crashes on startup when I have trucated bitcask file
It also crashes when the AAE files are bad too I think. Example below
2017-02-13 21:18:30 =CRASH REPORT====
crasher:
initial call: riak_kv_index_hashtree:init/1
pid: <0.6037.0>
registered_name: []
exception exit: {{{badmatch,{error,{db_open,"Corruption: truncated
record at end of file"}}},[{hashtree,new_segment_
store,2,[{file,"src/hashtree.erl"},{line,675}]},{hashtree,new,2,[{file,"src/hashtree.erl"},{line,246}]},{riak_kv_index_h
ashtree,do_new_tree,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,610}]},{lists,foldl,3,[{file,"lists.erl"},{line,124
8}]},{riak_kv_index_hashtree,init_trees,3,[{file,"src/riak_kv_index_hashtree.erl"},{line,474}]},{riak_kv_index_hashtree,
init,1,[{file,"src/riak_kv_index_hashtree.erl"},{line,268}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]}
,{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line
,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
ancestors: [<0.715.0>,riak_core_vnode_sup,riak_core_sup,<0.160.0>]
messages: []
links: []
dictionary: []
trap_exit: false
status: running
heap_size: 1598
stack_size: 27
reductions: 889
neighbours:
Regards
Arun
On Tue, Feb 14, 2017 at 4:48 AM, Magnus Kessler <[email protected]> wrote:
> On 13 February 2017 at 19:56, Arun Rajagopalan <
> [email protected]> wrote:
>
>> Hello Riak Users
>>
>> We have situations where we dont or cant gracefully stop riak. When that
>> happens we occasionally get a truncated last-record in bitcask files
>>
>> If I delete those bitcask dir and the anti_entropy directory, Riak
>> rebuilds those bitcask files correctly
>>
>> Is there a way to rectify those borken bitcask files ?
>>
>> Thanks
>> Arun
>>
>>
>>
> Hi Arun,
>
> There should be no need to remove truncated bitcask files. Any objects up
> to the point of truncation should still be available to Riak. However, it
> may take longer for the affected partition to start up, as the
> corresponding hint file will not match the data file, and Riak will scan
> the latter to populate the key set it keeps in memory.
>
> By removing anti_entropy files you are forcing the AAE trees to be
> rebuilt. Once this has completed, AAE will fill back any objects missing
> due to the truncated bitcask files. You can also force a full partition
> repair by following the instructions from the documentation [0].
>
> Can you let me know why you cannot shut down the node gracefully? Unclean
> shutdowns should be a last resort and not part of normal operating
> procedures.
>
> Kind Regards,
>
> Magnus
>
>
> [0]: http://docs.basho.com/riak/kv/2.2.0/using/repair-recovery/
> repairs/#repairing-partitions.
>
> --
> Magnus Kessler
> Client Services Engineer
> Basho Technologies Limited
>
> Registered Office - 8 Lincoln’s Inn Fields London WC2A 3BP Reg 07970431
>
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
