Jie,
4G ram is small for your ring size and node count. My first recommendation is
that you reduce your ring size to 32 {ring_creation_size, 32}. Then change the
max_open_files setting of eleveldb to 40 {max_open_files, 50} and block_size to
32768 {sst_block_size, 32768}. These three settings should quickly improve
performance … but require you start over with your cluster.
Active Anti-Entropy has a bug in 1.4.7. You might as well disable it too:
{anti_entropy, {off, []}}. The recommendation is that you upgrade to 1.4.8.
Matthew
On Apr 14, 2014, at 10:10 PM, Jie Lu <[email protected]> wrote:
>
> I also have a problem in performance test of Riak Cluster.
> ~~~~~~~~~~~~
> Riak version: 1.4.7
> OS: openSUSE 11.3
> RAM: 4G
> ring size is 64
> backend: leveldb
> Nodes in cluster: 6 nodes
>
> ~~~~~~~~~~~~~
>
> I write a key/value with value is 1K bytes, and 25 concurrent threads on one
> client nodes. The test result only 20 ops/s performance.
>
> Is there any performance benchmark to compare with?
>
>
>
>
>
>
> On Tue, Apr 15, 2014 at 6:26 AM, Luke Bakken <[email protected]> wrote:
> Hi Matthew -
>
> Some suggestions:
>
> * Upgrade to Riak 1.4.8
>
> * Test with a ring size of 64
>
> * Use staggered merge windows in your cluster
> (http://docs.basho.com/riak/latest/ops/advanced/backends/bitcask/)
>
> * Since you're on dedicated hardware RAID, use the noop scheduler for your
> Riak data volumes:
>
> cat /sys/block/sd*/queue/scheduler
> noop anticipatory deadline [cfq]
>
> * Increase +zdbbl in /etc/riak/vm.args to 96000
>
> Thanks
> --
> Luke Bakken
> CSE
> [email protected]
>
>
> On Mon, Apr 14, 2014 at 2:33 PM, Matthew MacClary
> <[email protected]> wrote:
> I have a persistent issue I am trying to diagnose. In our use of Riak we have
> multiple data creators writing into a 7 node cluster. The value size is a bit
> large at around 2MB. The behavior I am seeing is that if I delete all data
> out of bitcask, then test performance I get fast writes. As I keep doing the
> same work of writing to the cluster, then the Riak write times will start
> tailing off and getting really bad.
>
> Initial write times seen by my application: 0.5 seconds for 100MB worth of
> values (~200MB/s)
> Subsequent write times: 11 seconds for 100MB worth of values (~9MB/s)
>
> This slow down can happen over roughly 20-40 minutes of writing or about
> 200GB worth of key/value pairs written.
>
> I can reset the cluster to get fast performance again by stopping Riak and
> deleting the bitcask directories, then starting Riak again. This step is not
> feasible for production, but during testing at least the write speed goes up
> by 20x.
>
> Watching iostat I see that every few seconds the disk io jumps to ~11%. It
> doesn't seem that highly loaded from my cursory look. Watching top I see that
> beam.smp runs at around 100 for CPU% or less when heavily loaded. I am not
> sure how to tell what it is doing though :-)
>
> Thanks for any suggestions!!
>
> -Matt
>
>
>
> ================
> System Description
>
>
> avg value size = 2MB
> Riak version = 1.4.1
> n_val = 2
> client threads total = 105
> backend = bitcask
> ring_creation_size = 128
> node count = 7
> node OS = RHEL 6.2
> server RAM = 128GB
> RAID = RAID0 across 8 SAS drives
> FS = ext4
> FS options = /dev/mapper/vg0-lv0 / ext4
> rw,noatime,barrier=0,stripe=512,data=ordered 0 0
> bitcask size on one server = 133GB
> AAE = off
> interface = protobuf
> client library = riak java client
> file-max = 65536
>
> _______________________________________________
> riak-users mailing list
> [email protected]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> _______________________________________________
> riak-users mailing list
> [email protected]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
>
> --
> Best Regards.
> Lu Jie
>
>
>
> --
> Best Regards.
> Lu Jie
> _______________________________________________
> riak-users mailing list
> [email protected]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
