Using local file tests/rpcping.
Using local file ../profile.
Total: 989 samples
321 32.5% 32.5% 321 32.5% svc_rqst_expire_cmpf
149 15.1% 47.5% 475 48.0% opr_rbtree_insert
139 14.1% 61.6% 140 14.2% __writev
56 5.7% 67.2% 66 6.7% __GI___pthread_mutex_lock
32 3.2% 70.5% 32 3.2% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/nptl/../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:371
32 3.2% 73.7% 32 3.2% futex_abstimed_wait_cancelable
23 2.3% 76.0% 23 2.3% __libc_write
21 2.1% 78.2% 23 2.3% __libc_recv
18 1.8% 80.0% 18 1.8% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/misc/../sysdeps/unix/syscall-template.S:84
17 1.7% 81.7% 17 1.7% __libc_read
15 1.5% 83.2% 15 1.5% clnt_req_xid_cmpf
14 1.4% 84.6% 14 1.4% _int_free
14 1.4% 86.0% 14 1.4% futex_wake
11 1.1% 87.2% 13 1.3% __GI_epoll_pwait
8 0.8% 88.0% 9 0.9% _int_malloc
8 0.8% 88.8% 494 49.9% svc_rqst_expire_insert
7 0.7% 89.5% 20 2.0% __libc_calloc
7 0.7% 90.2% 7 0.7% opr_rbtree_first
6 0.6% 90.8% 6 0.6% __GI___libc_free
5 0.5% 91.3% 5 0.5% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/nptl/../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
5 0.5% 91.8% 150 15.2% svc_vc_recv
5 0.5% 92.3% 470 47.5% work_pool_thread
4 0.4% 92.7% 4 0.4% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/nptl/../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:133
3 0.3% 93.0% 3 0.3% __condvar_dec_grefs
3 0.3% 93.3% 3 0.3% atomic_add_uint64_t
2 0.2% 93.5% 2 0.2% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/inet/../sysdeps/x86_64/htonl.S:29
2 0.2% 93.7% 2 0.2% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/nptl/../sysdeps/unix/sysv/linux/x86_64/cancellation.S:67
2 0.2% 93.9% 34 3.4% __pthread_mutex_unlock_usercnt
2 0.2% 94.1% 2 0.2% _seterr_reply
2 0.2% 94.3% 2 0.2% atomic_add_uint32_t@2665d
2 0.2% 94.5% 9 0.9% clnt_req_release
2 0.2% 94.7% 2 0.2% malloc_consolidate
2 0.2% 94.9% 142 14.4% svc_ioq_flushv
2 0.2% 95.1% 3 0.3% svc_ref_it@401421
2 0.2% 95.3% 255 25.8% svc_rqst_epoll_loop
2 0.2% 95.6% 12 1.2% xdr_ioq_release
2 0.2% 95.8% 6 0.6% xdr_ioq_setup
1 0.1% 95.9% 1 0.1% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/misc/../sysdeps/unix/syscall-template.S:84
1 0.1% 96.0% 1 0.1% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/nptl/../sysdeps/unix/sysv/linux/x86_64/cancellation.S:59
1 0.1% 96.1% 1 0.1% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/nptl/../sysdeps/unix/sysv/linux/x86_64/cancellation.S:60
1 0.1% 96.2% 1 0.1% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/nptl/../sysdeps/unix/sysv/linux/x86_64/cancellation.S:67
1 0.1% 96.3% 1 0.1% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/string/../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:296
1 0.1% 96.4% 1 0.1% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/string/../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:158
1 0.1% 96.5% 1 0.1% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/string/../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:189
1 0.1% 96.6% 1 0.1% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/string/../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:193
1 0.1% 96.7% 1 0.1% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/nptl/../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:138
1 0.1% 96.8% 1 0.1% 0x00007fffde7caa20
1 0.1% 96.9% 1 0.1% 0x00007fffde7caa25
1 0.1% 97.0% 1 0.1% 0x00007fffde7caaa1
1 0.1% 97.1% 1 0.1% __GI___libc_malloc
1 0.1% 97.2% 1 0.1% __condvar_get_private
1 0.1% 97.3% 1 0.1% __pthread_cond_destroy
1 0.1% 97.4% 1 0.1% __pthread_cond_signal
1 0.1% 97.5% 1 0.1% __pthread_cond_wait_common
1 0.1% 97.6% 1 0.1% _init
1 0.1% 97.7% 1 0.1% alloc_perturb
1 0.1% 97.8% 1 0.1% atomic_add_uint32_t@401377
1 0.1% 97.9% 1 0.1% atomic_dec_uint32_t@e3d8
1 0.1% 98.0% 2 0.2% atomic_inc_uint32_t@401394
1 0.1% 98.1% 4 0.4% atomic_inc_uint64_t
1 0.1% 98.2% 1 0.1% atomic_postclear_uint32_t_bits
1 0.1% 98.3% 1 0.1% atomic_postset_uint16_t_bits
1 0.1% 98.4% 1 0.1% atomic_sub_uint32_t
1 0.1% 98.5% 507 51.3% clnt_req_callback
1 0.1% 98.6% 9 0.9% clnt_req_setup
1 0.1% 98.7% 24 2.4% ev_sig
1 0.1% 98.8% 8 0.8% svc_ioq_write_submit
1 0.1% 98.9% 1 0.1% svc_release_it@268c0
1 0.1% 99.0% 1 0.1% svc_release_it@2ff36
1 0.1% 99.1% 180 18.2% svc_rqst_epoll_events
1 0.1% 99.2% 42 4.2% svc_rqst_expire_remove
1 0.1% 99.3% 257 26.0% svc_rqst_run_task
1 0.1% 99.4% 3 0.3% xdr_dplx_decode
1 0.1% 99.5% 1 0.1% xdr_getenum
1 0.1% 99.6% 1 0.1% xdr_inline_decode
1 0.1% 99.7% 18 1.8% xdr_ioq_create
1 0.1% 99.8% 18 1.8% xdr_ioq_destroy
1 0.1% 99.9% 19 1.9% xdr_ioq_destroy_internal
1 0.1% 100.0% 10 1.0% xdr_ioq_uv_release
0 0.0% 100.0% 989 100.0% ??
/usr/src/debug/glibc-2.26-137-g247c1ddd30/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:95
0 0.0% 100.0% 3 0.3% __GI___clock_gettime
0 0.0% 100.0% 1 0.1% atomic_dec_uint32_t@266b9
0 0.0% 100.0% 1 0.1% atomic_inc_uint32_t@2667a
0 0.0% 100.0% 1 0.1% clnt_req_fini
0 0.0% 100.0% 65 6.6% clnt_req_process_reply
0 0.0% 100.0% 1 0.1% clnt_req_reset
0 0.0% 100.0% 12 1.2% clnt_vc_call
0 0.0% 100.0% 17 1.7% consume_ev_sig_nb
0 0.0% 100.0% 83 8.4% decode_request
0 0.0% 100.0% 1 0.1% grow_heap
0 0.0% 100.0% 14 1.4% mem_free
0 0.0% 100.0% 10 1.0% opr_rbtree_lookup
0 0.0% 100.0% 1 0.1% rpc_dplx_rlt
0 0.0% 100.0% 989 100.0% start_thread
0 0.0% 100.0% 157 15.9% svc_ioq_write
0 0.0% 100.0% 157 15.9% svc_ioq_write_callback
0 0.0% 100.0% 1 0.1% svc_ref_it@2684d
0 0.0% 100.0% 18 1.8% svc_rqst_epoll_event
0 0.0% 100.0% 18 1.8% svc_rqst_rearm_events
0 0.0% 100.0% 1 0.1% svc_rqst_release
0 0.0% 100.0% 151 15.3% svc_rqst_xprt_task
0 0.0% 100.0% 68 6.9% svc_vc_decode
0 0.0% 100.0% 14 1.4% tirpc_calloc
0 0.0% 100.0% 18 1.8% tirpc_free
0 0.0% 100.0% 1 0.1% tirpc_malloc
0 0.0% 100.0% 16 1.6% work_pool_submit
0 0.0% 100.0% 519 52.5% worker
0 0.0% 100.0% 10 1.0% worker_cb
0 0.0% 100.0% 1 0.1% xdr_ioq_putbytes
0 0.0% 100.0% 4 0.4% xdr_ioq_uv_create
0 0.0% 100.0% 1 0.1% xdr_reply_decode
I was surprised at how much IO (the expected bottlenecks: recv, write)
are completely swamped by two problems!
Noting that the top problem is exactly my prediction by knowledge of
the code:
clnt_req_callback() opr_rbtree_insert()
The second is also exactly as expected:
svc_rqst_expire_insert() opr_rbtree_insert() svc_rqst_expire_cmpf()
These are both inserted in ascending order, sorted in ascending order,
and removed in ascending order....
QED: rb_tree is a poor data structure for this purpose.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Nfs-ganesha-devel mailing list
Nfs-ganesha-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
