Edgar:
Could you attach the AAE section of your app.config? I’d like to look into
this issue further for you. Something I think you might be running into is
https://github.com/basho/riak_core/pull/483.
The issue of concern is that the LevelDB bloom filter is not enabled properly
for the instance into which the AAE data is stored. You can mitigate this
particular issue by adding {use_bloomfilter, true} as shown below:
%% The LevelDB options used by AAE to generate the LevelDB-backed
%% on-disk hashtrees.
{anti_entropy_leveldb_opts, [{write_buffer_size, 4194304},
{max_open_files, 20}]},
Becomes:
%% The LevelDB options used by AAE to generate the LevelDB-backed
%% on-disk hashtrees.
{anti_entropy_leveldb_opts, [{write_buffer_size, 4194304},
{use_bloomfilter, true},
{max_open_files, 20}]},
This might not solve your specific problem, but it will certainly improve your
AAE performance.
Thanks,
Charlie Voiselle
On Dec 31, 2013, at 12:04 PM, Edgar Veiga <edgarmve...@gmail.com> wrote:
> Hey guys!
>
> Nothing on this one?
>
> Btw: Happy new year :)
>
>
> On 27 December 2013 22:35, Edgar Veiga <edgarmve...@gmail.com> wrote:
> This is a du -hs * of the riak folder:
>
> 44G anti_entropy
> 1.1M kv_vnode
> 252G leveldb
> 124K ring
>
> It's a 6 machine cluster, so ~1512G of levelDB.
>
> Thanks for the tip, I'll upgrade in a near future!
>
> Best regards
>
>
> On 27 December 2013 21:41, Matthew Von-Maszewski <matth...@basho.com> wrote:
> I have a query out to the developer that can better respond to your follow-up
> questions. It might be Monday before we get a reply due to the holidays.
>
> Do you happen to know how much data is in the leveldb dataset and/or one
> vnode? Not sure it will change the response, but might be nice to have that
> info available.
>
> Matthew
>
> P.S. Unrelated to your question: Riak 1.4.4 is available for download. It
> has a couple of nice bug fixes for leveldb.
>
>
> On Dec 27, 2013, at 2:08 PM, Edgar Veiga <edgarmve...@gmail.com> wrote:
>
>> Ok, thanks for confirming!
>>
>> Is it normal, that this action affects the overall state of the cluster? On
>> the 26th It started the regeneration and the the response times of the
>> cluster raised to never seen values. It was a day of heavy traffic but
>> everything was going quite ok until it started the regeneration process..
>>
>> Have you got any advices about changing those app.config values? My cluster
>> is running smoothly for the past 6 months and I don't want to start all over
>> again :)
>>
>> Best Regards
>>
>>
>> On 27 December 2013 18:56, Matthew Von-Maszewski <matth...@basho.com> wrote:
>> Yes. Confirmed.
>>
>> There are options available in app.config to control how often this occurs
>> and how many vnodes rehash at once: defaults are every 7 days and two
>> vnodes per server at a time.
>>
>> Matthew Von-Maszewski
>>
>>
>> On Dec 27, 2013, at 13:50, Edgar Veiga <edgarmve...@gmail.com> wrote:
>>
>>> Hi!
>>>
>>> I've been trying to find what may be the cause of this.
>>>
>>> Every once in a week, all the nodes in my riak cluster start to do some
>>> kind of operation that lasts at least for two days.
>>>
>>> You can watch a sample of my munin logs regarding the last week in here:
>>>
>>> https://cloudup.com/imWiBwaC6fm
>>> Take a look at the days 19 and 20, and now it has started again on the 26...
>>>
>>> I'm suspecting that this may be caused by the aae hash trees being
>>> regenerated, as you say in your documentation:
>>> For added protection, Riak periodically (default: once a week) clears and
>>> regenerates all hash trees from the on-disk K/V data.
>>> Can you confirm me that this may be the root of the "problem" and if it's
>>> normal for the action to last for two days?
>>>
>>> I'm using riak 1.4.2 on 6 machines, with centOS. The backend is levelDB.
>>>
>>> Best Regards,
>>> Edgar Veiga
>>> _______________________________________________
>>> riak-users mailing list
>>> riak-users@lists.basho.com
>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>
>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
