Ok here’s an update on the issue, I’m really hoping someone can chime in. I
disabled Anti-Entropy, restarted Riak, and then tried to run a repair on
indexes. After running the command [riak_kv_vnode:repair(P) || P <-
Partitions]. Riak then returned the error
** exception exit: {timeout,{gen_server,call,
[{riak_core_vnode_manager,'[email protected]'},
{repair,riak_kv,
{riak_kv_vnode,22835963083295358096932575511191922182123945984},
{riak_kv_vnode,repair_filter}}]}}
in function gen_server:call/2 (gen_server.erl, line 180)
We then tried to use the Basho Riak Data Migrator tool to back up the most
recent data on the node, but it takes 2 minutes to export a simple KV object
(and these are less than 32kb in size). Thanks for your help.
FYI Using Riak 1.4.6
On Jan 2, 2014, at 11:31 AM, Justin Long <[email protected]> wrote:
> Hi everyone,
>
> Experiencing some weird behaviour in a cluster of 3 nodes. As of this
> morning, we noticed a massive performance lag and when checking the logs
> Active Anti-Entropy Exchange suddenly began initiating a huge series of
> transfers. Here’s what we see in the logs:
>
> 2014-01-02 19:26:34.396 [info]
> <0.1571.1800>@riak_kv_exchange_fsm:key_exchange:206 Repaired 882 keys during
> active anti-entropy exchange of
> {228359630832953580969325755111919221821239459840,3} between
> {228359630832953580969325755111919221821239459840,'[email protected]'} and
> {251195593916248939066258330623111144003363405824,'[email protected]’}
>
> When we check the logs on all 3 nodes we only had a few process crashes 10
> hours ago, nothing in between that time and now. It all came on suddenly. Any
> idea if this is normal and will fix itself or if we need to intervene?
>
> Thanks!
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
