After upgrading my single-node instance to 1.0, I'm still seeing the "timeout
when storing" issue. Here are the changes I made based on everyone's
suggestions (much appreciated!):
- Ubuntu 11.04 (natty) 32-bit
- Python client 1.3.0
- /etc/riak/vm.args: -env ERL_MAX_PORTS 32768
- /etc/default/riak: ulimit -n 32768
Here's the /var/log/crash.log report:
2011-10-01 12:31:03 =ERROR REPORT====
** State machine <0.3452.0> terminating
** Last event in was
{'riak_vnode_req_v1',1136089163393944065322395631681798128560666312704,{fsm,undefined,<0.3451.0>},{'riak_kv_put_req_v1',{<<"nodes">>,<<"user_id-17527747-info">>},{r_object,<<"nodes">>,<<"user_id-17527747-info">>,[{r_content,{dict,4,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[[<<"content-type">>,97,112,112,108,105,99,97,116,105,111,110,47,106,115,111,110],[<<"X-Riak-VTag">>,49,88,88,75,75,51,90,88,68,117,90,122,85,53,57,85,53,101,107,89,115,110]],[[<<"index">>]],[],[[<<"X-Riak-Last-Modified">>|{1317,497463,847242}]],[],[]}}},<<"{DATA
DELETED}">>}],[],{dict,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[[clean|true]],[]}}},undefined},51456853,63484716663,[coord]}}
** When State == active
** Data ==
{state,1136089163393944065322395631681798128560666312704,riak_kv_vnode,{state,1136089163393944065322395631681798128560666312704,false,riak_kv_bitcask_backend,{state,#Ref<0.0.0.10359>,"1136089163393944065322395631681798128560666312704",[{async_folds,true},[{vnode_vclocks,true},{included_applications,[]},{add_paths,[]},{allow_strfun,false},{storage_backend,riak_kv_bitcask_backend},{legacy_keylisting,false},{reduce_js_vm_count,6},{js_thread_stack,16},{pb_ip,"0.0.0.0"},{riak_kv_stat,true},{map_js_vm_count,8},{mapred_system,pipe},{js_max_vm_mem,8},{pb_port,8087},{legacy_stats,true},{mapred_name,"mapred"},{stats_urlpath,"stats"},{http_url_encoding,on},{hook_js_vm_count,2}],{read_write,true}],1136089163393944065322395631681798128560666312704,"/var/lib/riak/bitcask"},{dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}}},<<35,9,254,249,78,135,82,106>>,3000,1000,100,100,true,false},undefined,undefined,none,undefined,<0.3454.0>,60000}
** Reason for termination =
** {bad_return_value,{error,{write_locked,emfile}}}
2011-10-01 12:31:03 =CRASH REPORT====
crasher:
initial call: riak_core_vnode:init/1
pid: <0.3452.0>
registered_name: []
exception exit: {bad_return_value,{error,{write_locked,emfile}}}
in function gen_fsm:terminate/7
in call from proc_lib:init_p_do_apply/3
ancestors: [riak_core_vnode_sup,riak_core_sup,<0.92.0>]
messages: [{'EXIT',<0.3454.0>,shutdown}]
links: [<0.96.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 6765
stack_size: 24
reductions: 160650
neighbours:
2011-10-01 12:31:03 =SUPERVISOR REPORT====
Supervisor: {local,riak_core_vnode_sup}
Context: child_terminated
Reason: {bad_return_value,{error,{write_locked,emfile}}}
Offender:
[{pid,<0.3452.0>},{name,undefined},{mfargs,{riak_core_vnode,start_link,undefined}},{restart_type,temporary},{shutdown,300000},{child_type,worker}]
2011-10-01 12:45:28 =ERROR REPORT====
Failed to merge
"/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1315770213.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1316329673.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1316330222.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1316879145.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1316995340.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1317493005.bitcask.data/var/lib/riak/bitcask/605153021707326989568713251046585937826284568576/1317495168.bitcask.data":
{{badmatch,{error,emfile}},[{bitcask,'-merge1/3-lc$^0/1-1-',1},{bitcask,'-merge1/3-lc$^0/1-1-',1},{bitcask,'merge1',3},{bitcask_merge_worker,do_merge,1}]}
-----Original Message-----
From: David Smith [mailto:[email protected]]
Sent: Fri 9/30/2011 9:56 AM
To: Jim Adler
Cc: Sean Cribbs; [email protected]
Subject: Re: Timeout when storing
IIRC, {error, emfile} indicates that the max # of ports (in the erlang
VM) is being exceeded. Try bumping up ERL_MAX_PORTS in vm.args.
D.
On Thu, Sep 29, 2011 at 10:52 PM, Jim Adler <[email protected]> wrote:
> Thanks Sean. I added the ulimit -n 10240 to /etc/default/riak, restarted
> riak, but that didn't work.
>
> Fyodor Yarochkin suggested that the bitcask files could be corrupted, but I
> wasn't sure which bitcask *.data or *.hint file to delete. Any pointers?
>
> Here's the /var/log/riak/erlang.log:
>
> =ERROR REPORT==== 29-Sep-2011::20:27:42 ===
> ** State machine <0.369.0> terminating
> ** Last event in was {riak_vnode_req_v1,
> 941983477185933521498468739836666790012612771840,
> {fsm,undefined,<0.27704.1>},
> {riak_kv_put_req_v1,
> {<<"nodes">>,<<"screen_name-psych_ic-info">>},
>
> {r_object,<<"nodes">>,<<"screen_name-psych_ic-info">>,
> [{r_content,
> {dict,3,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
> {{[],[],[],[],[],[],[],[],[],[],
> [[<<"content-type">>,97,112,112,108,105,99,97,
> 116,105,111,110,47,106,115,111,110],
>
> [<<"X-Riak-VTag">>,49,90,120,65,84,100,56,99,48,
> 80,86,99,111,122,71,79,108,90,70,97,53,87]],
> [],[],
> [[<<"X-Riak-Last-Modified">>|
> {1317,353201,695471}]],
> [],[]}}},
> <<"{DELETED DATA}">>}],
> [{<<2,65,205,48>>,{1,63484572401}}],
> {dict,1,16,16,8,80,48,
> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],[],
> [[clean|true]],
> []}}},
> undefined},
> 1174401,63484572401,
> [{returnbody,true}]}}
> ** When State == active
> ** Data == {state,941983477185933521498468739836666790012612771840,
> riak_kv_vnode,
>
> {state,941983477185933521498468739836666790012612771840,
> riak_kv_bitcask_backend,
> {#Ref<0.0.0.3952>,
>
> "/var/lib/riak/bitcask/941983477185933521498468739836666790012612771840"},
> {dict,0,16,16,8,80,48,
>
> {[],[],[],[],[],[],[],[],[],[],[],[],[],
> [],[],[]},
>
> {{[],[],[],[],[],[],[],[],[],[],[],[],[],
> [],[],[]}}},
> false},
> undefined,none,60000}
> ** Reason for termination =
> ** {{badmatch,{error,emfile}},
> [{bitcask_fileops,create_file_loop,3},
> {bitcask,put,3},
> {riak_kv_bitcask_backend,put,3},
> {riak_kv_vnode,perform_put,3},
> {riak_kv_vnode,do_put,7},
> {riak_kv_vnode,handle_command,3},
> {riak_core_vnode,vnode_command,3},
> {gen_fsm,handle_msg,7}]}
>
>
>
> -----Original Message-----
> From: Sean Cribbs [mailto:[email protected]]
> Sent: Thu 9/29/2011 3:02 PM
> To: Jim Adler
> Cc: [email protected]
> Subject: Re: Timeout when storing
>
> Your environment has too few file handles. Retry starting riak after
> setting `ulimit -n 1024` in the shell. Also see our wiki page about this
> issue: http://wiki.basho.com/Open-Files-Limit.html You may need to set this
> limit specifically for the 'riak' user.
>
> Cheers,
>
> --
> Sean Cribbs <[email protected]>
> Developer Advocate
> Basho Technologies, Inc.
> http://www.basho.com/
>
>
> _______________________________________________
> riak-users mailing list
> [email protected]
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
--
Dave Smith
Director, Engineering
Basho Technologies, Inc.
[email protected]
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
