Hi Guys, Can anybody help here?
Also found that due to continuous put operation beam.smp taking 2.9G RAM, what could be reason of this? Thanks. Amol Rajoba On Sat, Jun 16, 2012 at 1:13 PM, Amol Rajoba <[email protected]> wrote: > Hi Guys, > I am evaluating Riak as Kay-Value storage where my requirement is to store > huge set of data(more than RAM), so Riak was setup with LevelDB as backend. > > Clients were connected using protocol buffer api. > {pb_backlog, 100000}, in app.config > > Benchmarking involved 25 Agents doing put/store on single node for 100M > records. > It runs well till 3M but then complete cluster crashes with making all > nodes down. > > Following are the System as well as Riak configurations with error & crash > logs > > Please help to find what I am missing, I need to test riak & use it in > production as soon as possible. > > Nodes: 2 (I know cluster of 5 is best but this is just test setup) > OS: Ubuntu 12.04 32bit > CPU: Core i3 > RAM: 4GB > HDD: 500GB > > app.config [changes only] > > %% eLevelDB Config > {eleveldb, [ > {data_root, "/data/riak/leveldb"}, > {block_size, 262144}, %%256k > {cache_size, 10485760}, %% 10MB - default cache size 8MB > per-partition > {write_buffer_size, 524288000}, %% 500MB in bytes > {write_buffer_size_min, 524288000}, %% 500MB in bytes > {write_buffer_size_max, 524288000}, %% 500MB in bytes > {max_open_files, 100} %% Maximum number of files open at > once per partition- Default: 20 - Minimum: 20 > ]}, > > > vm.args [changes only] > ## Enable kernel poll and a few async threads > +K true > +A 128 > > > Bucket "riaktest" properties: > > {"props":{"allow_mult":false," > > basic_quorum":false,"big_vclock":50,"chash_keyfun":{"mod":"riak_core_util","fun":"chash_std_keyfun"},"dw":"quorum","last_write_wins":true,"linkfun":{"mod":"riak_kv_wm_link_walker","fun":"mapreduce_linkfun"},"n_val":3,"name":"riaktest","notfound_ok":true,"old_vclock":86400,"postcommit":[],"pr":0,"precommit":[],"pw":0,"r":"one","rw":"one","small_vclock":50,"w":"one","young_vclock":20}} > > relatime set in /etc/fstab on all drives > > OS open files limit sysctl fs.file-max set to 800000 > > > Following are the error.log, crash.log and console.log* *files* > > error.log* > --------------- > 2012-06-15 19:09:31.777 [error] <0.20970.188> gen_server <0.20970.188> > terminated with reason: > {mem_error,[{zlib,call,3},{zlib,zip,1},{riak_kv_pb_socket,process_message,2},{riak_kv_pb_socket,handle_info,2},{gen_server2,handle_msg,7},{proc_lib,init_p_do_apply,3}]} > 2012-06-15 19:09:32.219 [error] <0.20970.188> CRASH REPORT Process > <0.20970.188> with 0 neighbours crashed with reason: > {mem_error,[{zlib,call,3},{zlib,zip,1},{riak_kv_pb_socket,process_message,2},{riak_kv_pb_socket,handle_info,2},{gen_server2,handle_msg,7},{proc_lib,init_p_do_apply,3}]} > 2012-06-15 19:09:32.320 [error] <0.284.0> Supervisor riak_kv_pb_socket_sup > had child undefined started with {riak_kv_pb_socket,start_link,undefined} > at <0.20970.188> exit with reason > {mem_error,[{zlib,call,3},{zlib,zip,1},{riak_kv_pb_socket,process_message,2},{riak_kv_pb_socket,handle_info,2},{gen_server2,handle_msg,7},{proc_lib,init_p_do_apply,3}]} > in context child_terminated > 2012-06-15 19:09:32.824 [error] <0.20974.188> gen_server <0.20974.188> > terminated with reason: > {mem_error,[{zlib,call,3},{zlib,zip,1},{riak_kv_pb_socket,process_message,2},{riak_kv_pb_socket,handle_info,2},{gen_server2,handle_msg,7},{proc_lib,init_p_do_apply,3}]} > 2012-06-15 19:09:32.972 [error] <0.20974.188> CRASH REPORT Process > <0.20974.188> with 0 neighbours crashed with reason: > {mem_error,[{zlib,call,3},{zlib,zip,1},{riak_kv_pb_socket,process_message,2},{riak_kv_pb_socket,handle_info,2},{gen_server2,handle_msg,7},{proc_lib,init_p_do_apply,3}]} > > * > > Crash.log* > -------------- > 2012-06-15 19:09:31 =ERROR REPORT==== > ** Generic server <0.20970.188> terminating > ** Last message in was > {tcp,#Port<0.6076011>,[11|<<10,6,117,114,108,99,97,116,18,39,50,53,48,50,97,98,102,49,55,97,100,102,100,48,98,55,102,48,57,48,52,99,48,99,98,101,52,48,100,100,100,55,49,55,50,48,51,51,57,34,122,10,109,34,50,53,48,50,97,98,102,49,55,97,100,102,100,48,98,55,102,48,57,48,52,99,48,99,98,101,52,48,100,100,100,55,49,55,50,48,51,51,57,58,58,32,99,97,116,101,103,111,114,121,32,49,44,32,107,101,121,119,111,114,100,32,49,44,32,99,97,116,101,103,111,114,121,32,50,44,32,99,97,116,101,103,111,114,121,32,51,44,32,107,101,121,119,111,114,100,50,44,32,107,101,121,119,111,114,100,51,34,18,9,116,101,120,116,47,106,115,111,110,40,2,48,2,56,1>>]} > ** When Server state == {state,#Port<0.6076011>,{riak_client,' > [email protected]',undefined},undefined,undefined,<<0,0,0,0>>} > ** Reason for termination == > ** > {mem_error,[{zlib,call,3},{zlib,zip,1},{riak_kv_pb_socket,process_message,2},{riak_kv_pb_socket,handle_info,2},{gen_server2,handle_msg,7},{proc_lib,init_p_do_apply,3}]} > 2012-06-15 19:09:32 =CRASH REPORT==== > crasher: > initial call: gen:init_it/6 > pid: <0.20970.188> > registered_name: [] > exception exit: > {mem_error,[{zlib,call,3},{zlib,zip,1},{riak_kv_pb_socket,process_message,2},{riak_kv_pb_socket,handle_info,2},{gen_server2,handle_msg,7},{proc_lib,init_p_do_apply,3}]} > in function gen_server2:terminate/6 > in call from proc_lib:init_p_do_apply/3 > ancestors: [riak_kv_pb_socket_sup,riak_kv_sup,<0.279.0>] > messages: [] > links: [#Port<0.6076023>,<0.284.0>,#Port<0.6076011>] > dictionary: [] > trap_exit: false > status: running > heap_size: 987 > stack_size: 24 > reductions: 974 > neighbours: > 2012-06-15 19:09:32 =SUPERVISOR REPORT==== > Supervisor: {local,riak_kv_pb_socket_sup} > Context: child_terminated > Reason: > {mem_error,[{zlib,call,3},{zlib,zip,1},{riak_kv_pb_socket,process_message,2},{riak_kv_pb_socket,handle_info,2},{gen_server2,handle_msg,7},{proc_lib,init_p_do_apply,3}]} > Offender: > [{pid,<0.20970.188>},{name,undefined},{mfargs,{riak_kv_pb_socket,start_link,undefined}},{restart_type,temporary},{shutdown,brutal_kill},{child_type,worker}] > > 2012-06-15 19:09:32 =ERROR REPORT==== > ** Generic server <0.20974.188> terminating > ** Last message in was > {tcp,#Port<0.6076015>,[11|<<10,6,117,114,108,99,97,116,18,39,50,53,48,50,97,98,102,49,55,97,100,102,100,48,98,55,102,48,57,48,52,99,48,99,98,101,52,48,100,100,100,55,49,49,50,48,51,57,57,34,122,10,109,34,50,53,48,50,97,98,102,49,55,97,100,102,100,48,98,55,102,48,57,48,52,99,48,99,98,101,52,48,100,100,100,55,49,49,50,48,51,57,57,58,58,32,99,97,116,101,103,111,114,121,32,49,44,32,107,101,121,119,111,114,100,32,49,44,32,99,97,116,101,103,111,114,121,32,50,44,32,99,97,116,101,103,111,114,121,32,51,44,32,107,101,121,119,111,114,100,50,44,32,107,101,121,119,111,114,100,51,34,18,9,116,101,120,116,47,106,115,111,110,40,2,48,2,56,1>>]} > ** When Server state == {state,#Port<0.6076015>,{riak_client,' > [email protected]',undefined},undefined,undefined,<<0,0,0,0>>} > ** Reason for termination == > ** > {mem_error,[{zlib,call,3},{zlib,zip,1},{riak_kv_pb_socket,process_message,2},{riak_kv_pb_socket,handle_info,2},{gen_server2,handle_msg,7},{proc_lib,init_p_do_apply,3}]} > 2012-06-15 19:09:33 =CRASH REPORT==== > crasher: > initial call: gen:init_it/6 > pid: <0.20974.188> > registered_name: [] > exception exit: > {mem_error,[{zlib,call,3},{zlib,zip,1},{riak_kv_pb_socket,process_message,2},{riak_kv_pb_socket,handle_info,2},{gen_server2,handle_msg,7},{proc_lib,init_p_do_apply,3}]} > in function gen_server2:terminate/6 > in call from proc_lib:init_p_do_apply/3 > ancestors: [riak_kv_pb_socket_sup,riak_kv_sup,<0.279.0>] > messages: [] > links: [#Port<0.6076029>,<0.284.0>,#Port<0.6076015>] > dictionary: [] > trap_exit: false > status: running > heap_size: 987 > stack_size: 24 > reductions: 910 > neighbours: > > > > *Console.log* > -------------------- > 2012-06-15 17:50:48.811 [info] <0.7.0> Application lager started on node ' > [email protected]' > 2012-06-15 17:50:48.970 [info] <0.7.0> Application public_key started on > node '[email protected]' > 2012-06-15 17:50:49.003 [info] <0.7.0> Application ssl started on node ' > [email protected]' > 2012-06-15 17:50:49.037 [info] <0.7.0> Application riak_core started on > node '[email protected]' > 2012-06-15 17:50:49.060 [info] <0.7.0> Application riak_control started on > node '[email protected]' > 2012-06-15 17:50:49.061 [info] <0.7.0> Application basho_metrics started > on node '[email protected]' > 2012-06-15 17:50:49.063 [info] <0.7.0> Application cluster_info started on > node '[email protected]' > 2012-06-15 17:50:49.072 [info] <0.7.0> Application merge_index started on > node '[email protected]' > 2012-06-15 17:50:49.083 [info] <0.180.0>@riak_core:wait_for_service:416 > Waiting for service riak_pipe to start (0 seconds) > 2012-06-15 17:50:49.110 [info] > <0.249.0>@riak_core:wait_for_application:396 Waiting for application > riak_pipe to start (0 seconds). > 2012-06-15 17:50:49.111 [info] <0.7.0> Application riak_pipe started on > node '[email protected]' > 2012-06-15 17:50:49.145 [info] <0.7.0> Application inets started on node ' > [email protected]' > 2012-06-15 17:50:49.151 [info] <0.7.0> Application mochiweb started on > node '[email protected]' > 2012-06-15 17:50:49.169 [info] <0.7.0> Application erlang_js started on > node '[email protected]' > 2012-06-15 17:50:49.176 [info] <0.7.0> Application luke started on node ' > [email protected]' > 2012-06-15 17:50:49.197 [info] <0.283.0>@riak_core:wait_for_service:416 > Waiting for service riak_kv to start (0 seconds) > 2012-06-15 17:50:49.212 [info] > <0.249.0>@riak_core:wait_for_application:390 Wait complete for application > riak_pipe (0 seconds) > 2012-06-15 17:50:49.285 [info] <0.180.0>@riak_core:wait_for_service:410 > Wait complete for service riak_pipe (0 seconds) > 2012-06-15 17:50:49.291 [info] <0.367.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) > host starting (<0.367.0>) > 2012-06-15 17:50:49.296 [info] <0.368.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) > host starting (<0.368.0>) > 2012-06-15 17:50:49.302 [info] <0.369.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) > host starting (<0.369.0>) > 2012-06-15 17:50:49.307 [info] <0.370.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) > host starting (<0.370.0>) > 2012-06-15 17:50:49.311 [info] <0.371.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) > host starting (<0.371.0>) > 2012-06-15 17:50:49.316 [info] <0.372.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) > host starting (<0.372.0>) > 2012-06-15 17:50:49.320 [info] <0.373.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) > host starting (<0.373.0>) > 2012-06-15 17:50:49.324 [info] <0.374.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_map) > host starting (<0.374.0>) > 2012-06-15 17:50:49.333 [info] <0.376.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: > riak_kv_js_reduce) host starting (<0.376.0>) > 2012-06-15 17:50:49.341 [info] <0.377.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: > riak_kv_js_reduce) host starting (<0.377.0>) > 2012-06-15 17:50:49.348 [info] <0.378.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: > riak_kv_js_reduce) host starting (<0.378.0>) > 2012-06-15 17:50:49.354 [info] <0.379.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: > riak_kv_js_reduce) host starting (<0.379.0>) > 2012-06-15 17:50:49.360 [info] <0.380.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: > riak_kv_js_reduce) host starting (<0.380.0>) > 2012-06-15 17:50:49.366 [info] <0.381.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: > riak_kv_js_reduce) host starting (<0.381.0>) > 2012-06-15 17:50:49.371 [info] <0.383.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_hook) > host starting (<0.383.0>) > 2012-06-15 17:50:49.375 [info] <0.384.0>@riak_kv_js_vm:init:76 > Spidermonkey VM (thread stack: 16MB, max heap: 8MB, pool: riak_kv_js_hook) > host starting (<0.384.0>) > 2012-06-15 17:50:49.395 [info] <0.7.0> Application bitcask started on node > '[email protected]' > 2012-06-15 17:50:49.567 [info] > <0.463.0>@riak_core:wait_for_application:396 Waiting for application > riak_kv to start (0 seconds). > 2012-06-15 17:50:49.571 [info] <0.7.0> Application riak_kv started on node > '[email protected]' > 2012-06-15 17:50:49.573 [info] <0.7.0> Application riak_search started on > node '[email protected]' > 2012-06-15 17:50:49.573 [info] <0.7.0> Application basho_stats started on > node '[email protected]' > 2012-06-15 17:50:49.584 [info] <0.7.0> Application runtime_tools started > on node '[email protected]' > 2012-06-15 17:50:49.669 [info] > <0.463.0>@riak_core:wait_for_application:390 Wait complete for application > riak_kv (0 seconds) > 2012-06-15 17:50:54.871 [info] <0.283.0>@riak_core:wait_for_service:410 > Wait complete for service riak_kv (4 seconds) > 2012-06-15 18:26:48.764 [info] <0.42.0> alarm_handler: > {set,{system_memory_high_watermark,[]}} > 2012-06-15 19:09:31.777 [error] <0.20970.188> gen_server <0.20970.188> > terminated with reason: > {mem_error,[{zlib,call,3},{zlib,zip,1},{riak_kv_pb_socket,process_message,2},{riak_kv_pb_socket,handle_info,2},{gen_server2,handle_msg,7},{proc_lib,init_p_do_apply,3}]} > 2012-06-15 19:09:32.219 [error] <0.20970.188> CRASH REPORT Process > <0.20970.188> with 0 neighbours crashed with reason: > {mem_error,[{zlib,call,3},{zlib,zip,1},{riak_kv_pb_socket,process_message,2},{riak_kv_pb_socket,handle_info,2},{gen_server2,handle_msg,7},{proc_lib,init_p_do_apply,3}]} > 2012-06-15 19:09:32.320 [error] <0.284.0> Supervisor riak_kv_pb_socket_sup > had child undefined started with {riak_kv_pb_socket,start_link,undefined} > at <0.20970.188> exit with reason > {mem_error,[{zlib,call,3},{zlib,zip,1},{riak_kv_pb_socket,process_message,2},{riak_kv_pb_socket,handle_info,2},{gen_server2,handle_msg,7},{proc_lib,init_p_do_apply,3}]} > in context child_terminated > 2012-06-15 19:09:32.824 [error] <0.20974.188> gen_server <0.20974.188> > terminated with reason: > {mem_error,[{zlib,call,3},{zlib,zip,1},{riak_kv_pb_socket,process_message,2},{riak_kv_pb_socket,handle_info,2},{gen_server2,handle_msg,7},{proc_lib,init_p_do_apply,3}]} > 2012-06-15 19:09:32.972 [error] <0.20974.188> CRASH REPORT Process > <0.20974.188> with 0 neighbours crashed with reason: > {mem_error,[{zlib,call,3},{zlib,zip,1},{riak_kv_pb_socket,process_message,2},{riak_kv_pb_socket,handle_info,2},{gen_server2,handle_msg,7},{proc_lib,init_p_do_apply,3}]} > > > Thanks In Advance, > Amol Rajoba >
_______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
