Thanks Engel,
That approach looks very accurate, I would only suggest to have a
riak-admin cluster stop-aae and similar for start, for the dummies ;-)
Guido.
On 10/04/14 14:22, Engel Sanchez wrote:
Hey there. There are a couple of things to keep in mind when deleting
invalid AAE trees from the 1.4.3-1.4.7 series after upgrading to 1.4.8:
* If AAE is disabled, you don't have to stop the node to delete the
data in the anti_entropy directories
* If AAE is enabled, deleting the AAE data in a rolling manner may
trigger an avalanche of read repairs between nodes with the bad trees
and nodes with good trees as the data seems to diverge.
If your nodes are already up, with AAE enabled and with old incorrect
trees in the mix, there is a better way. You can dynamically disable
AAE with some console commands. At that point, without stopping the
nodes, you can delete all AAE data across the cluster. At a
convenient time, re-enable AAE. I say convenient because all trees
will start to rebuild, and that can be problematic in an overloaded
cluster. Doing this over the weekend might be a good idea unless your
cluster can take the extra load.
To dynamically disable AAE from the Riak console, you can run this
command:
> riak_core_util:rpc_every_member_ann(riak_kv_entropy_manager,
disable, [], 60000).
and enable with the similar:
> riak_core_util:rpc_every_member_ann(riak_kv_entropy_manager, enable,
[], 60000).
That last number is just a timeout for the RPC operation. I hope this
saves you some extra load on your clusters.
On Wed, Apr 9, 2014 at 11:02 AM, Luke Bakken <[email protected]
<mailto:[email protected]>> wrote:
Hi Guido,
I specifically meant riak-admin transfers however using riak-admin
wait-for-service riak_kv riak@node is a good first step before
waiting for transfers.
Thanks!
--
Luke Bakken
CSE
[email protected] <mailto:[email protected]>
On Wed, Apr 9, 2014 at 7:54 AM, Guido Medina
<[email protected] <mailto:[email protected]>> wrote:
What do you mean by "wait for handoff" to finish?
Are you referring to wait for the service to be fully started?
i.e. "riak-admin wait-for-service riak_kv riak@node"
Or do you mean to check for "riak-admin transfers" on the
started node and wait until those handoffs/transfers are gone?
Guido.
On 09/04/14 15:46, Luke Bakken wrote:
Hi Guido,
That is the correct process. Be sure to use the rolling
restart procedure when restarting nodes (i.e. wait for
handoff to finish before moving on).
--
Luke Bakken
CSE
[email protected] <mailto:[email protected]>
On Wed, Apr 9, 2014 at 6:34 AM, Guido Medina
<[email protected] <mailto:[email protected]>>
wrote:
Hi,
If nodes are already upgraded to 1.4.8 (and they went all
the way from 1.4.0 to 1.4.8 including AAE buggy versions)
Will the following command (as root) on Ubuntu Servers 12.04:
riak stop; rm -Rf /var/lib/riak/anti_entropy/*; riak
start
executed on each node be enough to rebuild AAE hashes?
Regards,
Guido.
_______________________________________________
riak-users mailing list
[email protected]
<mailto:[email protected]>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
[email protected] <mailto:[email protected]>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
[email protected] <mailto:[email protected]>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
