On Fri, Apr 20, 2012 at 9:01 AM, <[email protected]> wrote:
> Eventually this becomes the primary workload of the cluster and individual
> deletion latencies grow (more detailed measurements on the shape of this
> degradation are forthcoming if that is helpful).
>
Are you in EC2 or metal?
How are you deleting the values? One call per key?
We perform our deletes using MR and a Erlang reduce function. If you set
the pre-reduce option, this will keep the delete within the node that holds
the value.
-module(reduce_functions).
-export([delete/2]).
% Data is a list of bucket and key pairs, intermixed with the counts of
deleted
% objects. Returns a count of deleted objects.
delete(List, _None) ->
{ok, C} = riak:local_client(),
Delete = fun(Bucket, Key) ->
case C:delete(Bucket, Key, 0) of
ok -> 1;
_ -> 0
end
end,
F = fun(Elem, Acc) ->
case Elem of
{{Bucket, Key}, _KeyData} ->
Acc + Delete(Bucket, Key);
{Bucket, Key} ->
Acc + Delete(Bucket, Key);
[Bucket, Key] ->
Acc + Delete(Bucket, Key);
_ ->
Acc + Elem
end
end,
[lists:foldl(F, 0, List)].
> We are using riak 1.1 directly from
> https://github.com/basho/riak/tree/1.1with the eleveldb backend. The
> eleveldb specific configuration follows, but
> fiddling with these settings hasn't noticeably impacted behavior we've
> seen. Planning to set delete_mode to immediate and see if that helps.
>
> Here's some other info that might be helpful but feel free to ask for
> anything else.
>
> N = 3 (changing to 2) on 9 physical nodes w 32GB memory each
>
> Our leveldb config looks like this:
>
> %% eLevelDB Config
> {eleveldb, [
> {data_root, "/srv/riak/leveldb"},
> {max_open_files, 400},
> {block_size, 262144},
> {cache_size, 1932735280},
> {sync, false}
> ]},
>
>
You can also increase the leveldb write_buffer_size config option.
%% Amount of data to build up in memory (backed by an
unsorted
%% log on disk) before converting to a sorted on-disk file.
%% Larger values increase performance, especially during
bulk
%% loads. Up to two write buffers may be held in memory at
the
%% same time, so you may wish to adjust this parameter to
%% control memory usage. Also, a larger write buffer will
%% result in a longer recovery time the next time the
database
%% is opened. Default is: 4MB.
Elias
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
