Hi~!
I'm testing riak as a cache server with 6 server cluster, 2 hours expiration
time.
I gave traffic to cluster about 40,000 quries per second from 12hours ago.
From start point, there was no problem, but after 8hours fetch method returns
fail with "null point excpetion".
I change nothing while test operation.
Below, you can see "riak-admin status" and riak-admin top" result.
Could show me what cause this and how to solve this problem?
Thank you.
Chan.
riak-admin status
-------------------------------------------------------------------------------------------------------------------------------
riak_kv_stat_ts : 1381893196
vnode_gets : 1301374
vnode_gets_total : 1488219135
vnode_puts : 650670
vnode_puts_total : 273164708
vnode_index_reads : 0
vnode_index_reads_total : 0
vnode_index_writes : 650670
vnode_index_writes_total : 273164708
vnode_index_writes_postings : 0
vnode_index_writes_postings_total : 0
vnode_index_deletes : 0
vnode_index_deletes_total : 0
vnode_index_deletes_postings : 0
vnode_index_deletes_postings_total : 0
node_gets : 532128
node_gets_total : 605453222
node_get_fsm_siblings_mean : 0
node_get_fsm_siblings_median : 0
node_get_fsm_siblings_95 : 0
node_get_fsm_siblings_99 : 0
node_get_fsm_siblings_100 : 0
node_get_fsm_objsize_mean : 0
node_get_fsm_objsize_median : 0
node_get_fsm_objsize_95 : 0
node_get_fsm_objsize_99 : 0
node_get_fsm_objsize_100 : 0
node_get_fsm_time_mean : 476
node_get_fsm_time_median : 467
node_get_fsm_time_95 : 865
node_get_fsm_time_99 : 1119
node_get_fsm_time_100 : 3213
node_puts : 226235
node_puts_total : 95080556
node_put_fsm_time_mean : 1179
node_put_fsm_time_median : 1116
node_put_fsm_time_95 : 1806
node_put_fsm_time_99 : 2337
node_put_fsm_time_100 : 10206
read_repairs : 0
read_repairs_total : 2697556
coord_redirs_total : 58178410
executing_mappers : 0
precommit_fail : 0
postcommit_fail : 0
index_fsm_create : 0
index_fsm_create_error : 0
index_fsm_active : 0
list_fsm_create : 0
list_fsm_create_error : 0
list_fsm_active : 0
pbc_active : 29
pbc_connects : 1
pbc_connects_total : 2284
node_get_fsm_active : 22
node_get_fsm_active_60s : 527810
node_get_fsm_in_rate : 8562
node_get_fsm_out_rate : 8560
node_get_fsm_rejected : 0
node_get_fsm_rejected_60s : 0
node_get_fsm_rejected_total : 0
node_put_fsm_active : 5
node_put_fsm_active_60s : 368997
node_put_fsm_in_rate : 6130
node_put_fsm_out_rate : 6128
node_put_fsm_rejected : 0
node_put_fsm_rejected_60s : 0
node_put_fsm_rejected_total : 0
read_repairs_primary_outofdate_one : 0
read_repairs_primary_outofdate_count : 126588
read_repairs_primary_notfound_one : 0
read_repairs_primary_notfound_count : 4169558
leveldb_read_block_error : undefined
riak_pipe_stat_ts : 1381893196
pipeline_active : 0
pipeline_create_count : 0
pipeline_create_one : 0
pipeline_create_error_count : 0
pipeline_create_error_one : 0
cpu_nprocs : 361
cpu_avg1 : 2068
cpu_avg5 : 2045
cpu_avg15 : 2028
mem_total : 16799096832
mem_allocated : 4083048448
disk : [{"/",279713876,2},{"/dev/shm",8202684,0}]
nodename : '[email protected]<mailto:[email protected]>'
connected_nodes :
['[email protected]<mailto:[email protected]>','[email protected]<mailto:[email protected]>',
'[email protected]<mailto:[email protected]>','[email protected]<mailto:[email protected]>',
'[email protected]<mailto:[email protected]>']
sys_driver_version : <<"2.0">>
sys_global_heaps_size : 0
sys_heap_type : private
sys_logical_processors : 8
sys_otp_release : <<"R15B01">>
sys_process_count : 566
sys_smp_support : true
sys_system_version : <<"Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:8:8]
[async-threads:128] [kernel-poll:true]">>
sys_system_architecture : <<"x86_64-unknown-linux-gnu">>
sys_threads_enabled : true
sys_thread_pool_size : 128
sys_wordsize : 8
ring_members :
['[email protected]<mailto:[email protected]>','[email protected]<mailto:[email protected]>',
'[email protected]<mailto:[email protected]>','[email protected]<mailto:[email protected]>',
'[email protected]<mailto:[email protected]>','[email protected]<mailto:[email protected]>']
ring_num_partitions : 32
ring_ownership :
<<"[{'[email protected]<mailto:[email protected]>',5},\n
{'[email protected]<mailto:[email protected]>',5},\n
{'[email protected]<mailto:[email protected]>',5},\n
{'[email protected]<mailto:[email protected]>',5},\n
{'[email protected]<mailto:[email protected]>',6},\n
{'[email protected]<mailto:[email protected]>',6}]">>
ring_creation_size : 32
storage_backend : riak_kv_multi_backend
erlydtl_version : <<"0.7.0">>
riak_control_version : <<"1.4.2-0-g2f08324">>
cluster_info_version : <<"1.2.4">>
riak_search_version : <<"1.4.2-0-ge31368d">>
merge_index_version : <<"1.3.1">>
riak_kv_version : <<"1.4.2-0-g61ac9d8">>
sidejob_version : <<"0.2.0">>
riak_api_version : <<"1.4.2-0-ga7e00e2">>
riak_pipe_version : <<"1.4.2-0-g59d1bae">>
riak_core_version : <<"1.4.2">>
bitcask_version : <<"1.6.3">>
basho_stats_version : <<"1.0.3">>
webmachine_version : <<"1.10.4-0-gfcff795">>
mochiweb_version : <<"1.5.1p6">>
inets_version : <<"5.9">>
erlang_js_version : <<"1.2.2">>
runtime_tools_version : <<"1.8.8">>
os_mon_version : <<"2.2.9">>
riak_sysmon_version : <<"1.1.3">>
ssl_version : <<"5.0.1">>
public_key_version : <<"0.15">>
crypto_version : <<"2.1">>
sasl_version : <<"2.2.1">>
lager_version : <<"2.0.0">>
compiler_version : <<"4.8.1">>
syntax_tools_version : <<"1.6.8">>
stdlib_version : <<"1.18.1">>
kernel_version : <<"2.15.1">>
memory_total : 333983032
memory_processes : 10197917
memory_processes_used : 9925151
memory_system : 323785115
memory_atom : 504409
memory_atom_used : 485838
memory_binary : 479552
memory_code : 10516587
memory_ets : 227826744
riak-admin top
-------------------------------------------------------------------------------------------------------------------------------
Pid Name or Initial Func Time Reds Memory
MsgQ Current Function
-------------------------------------------------------------------------------------------------------------------------------
<6210.632.0> proc_lib:init_p/5 '-' 8916284 21744
1 riak_core_vnode:continue/2
<6210.631.0> proc_lib:init_p/5 '-' 8890232 21696
0 gen_fsm:loop/7
<6210.565.0> proc_lib:init_p/5 '-' 8881011 8920
0 gen_fsm:loop/7
<6210.511.0> proc_lib:init_p/5 '-' 8795742 21728
1 gen_fsm:loop/7
<6210.633.0> proc_lib:init_p/5 '-' 8791178 21696
0 gen_fsm:loop/7
<6210.337.0> riak_kv_stat_sj_8 '-' 4292821 3856
0 gen_server:loop/6
<6210.331.0> riak_kv_stat_sj_2 '-' 3583242 3856
0 gen_server:loop/6
<6210.332.0> riak_kv_stat_sj_3 '-' 3374842 5720
0 gen_server:loop/6
<6210.336.0> riak_kv_stat_sj_7 '-' 3362782 5720
0 gen_server:loop/6
<6210.334.0> riak_kv_stat_sj_5 '-' 3328089 3856
0 gen_server:loop/6
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
