Hi Mercury, VCL sessions are allocated on the heap and given that the size of one session is ~200B, 1M will eat up about 200MB of memory. How many are actually allocated in nginx? Just do “show session” in vpp and that should report how many sessions vpp is tracking.
Also, how many workers does nginx come up with? By default vcl limits the number of app workers to 16. More can be configured in vcl.conf with "max-workers <n>” although I’d first recommend running nginx with a lower number of workers with “worker_processes <n>”. Regards, Florin > On Nov 29, 2021, at 11:22 PM, mercury noah <mercury124...@gmail.com> wrote: > > Hi, > > I want to use nginx with LD_PRELOAD to act as a http web server(1 nginx > worker and 1 nginx master), > VCL_CFG=/etc/vpp/vcl.conf > LDP_PATH=/usr/lib/x86_64-linux-gnu/libvcl_ldpreload.so > LD_PRELOAD=$LDP_PATH VCL_CONFIG=$VCL_CFG /usr/sbin/nginx -c > /vsap/configs/vcl.conf > > traffic generator is ab, > test case is CPS(connections per second), > ab -n 5000000 -q -c 500 http://10.0.0.1/ <http://10.0.0.1/> > when traffic is accumulated to a certain amount, nginx worker process will > crash, > I dont know what's wrong with this environment, > > below is some env info: > [root@abc ~]# nginx -v > nginx version: nginx/1.14.2 > > vpp# show version > vpp v21.10-rc0~240-ga70b015ce built by root on debian at 2021-08-16T06:11:35 > > nginx config file is almost the same with vsap > https://github.com/FDio/vsap/blob/master/configs/nginx.conf > <https://github.com/FDio/vsap/blob/master/configs/nginx.conf> > > vcl config is > vcl { > heapsize 128M > rx-fifo-size 4000000 > tx-fifo-size 4000000 > # app-scope-local > # app-scope-global > api-socket-name /run/vpp/api.sock > } > > backtrace of nginx is: > (gdb) bt > #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 > #1 0x00007f31d244d535 in __GI_abort () at abort.c:79 > #2 0x00007f31d233ecb6 in os_panic () from > /lib/x86_64-linux-gnu/libvppinfra.so.21.10 > #3 0x00007f31d2346a35 in vec_resize_allocate_memory () from > /lib/x86_64-linux-gnu/libvppinfra.so.21.10 > #4 0x00007f31d23eb2bc in _vec_resize_inline (numa_id=<optimized out>, > data_align=<optimized out>, header_bytes=<optimized out>, > data_bytes=<optimized out>, length_increment=<optimized out>, v=<optimized > out>) at /root/cx/vpp/src/vppinfra/vec.h:172 > #5 vcl_session_alloc (wrk=<optimized out>) at > /root/cx/vpp/src/vcl/vcl_private.h:383 > #6 vppcom_epoll_create () at /root/cx/vpp/src/vcl/vppcom.c:2734 > #7 0x00007f31d23f0c14 in vcl_session_accepted_handler (ls_index=0, > mp=0x7ffe4a6f61e0, wrk=0x7f31ce1ae840) at /root/cx/vpp/src/vcl/vppcom.c:1779 > #8 vppcom_session_accept (listen_session_handle=0, > ep=ep@entry=0x7ffe4a6f6310, flags=flags@entry=2048) at > /root/cx/vpp/src/vcl/vppcom.c:1779 > #9 0x00007f31d240c97e in vls_accept (listener_vlsh=listener_vlsh@entry=0, > ep=ep@entry=0x7ffe4a6f6310, flags=flags@entry=2048) at > /root/cx/vpp/src/vcl/vcl_locked.c:368 > #10 0x00007f31d2c834d2 in ldp_accept4 (flags=2048, addr_len=0x7ffe4a6f63dc, > addr=0x7ffe4a6f63f0, listen_fd=32) at /root/cx/vpp/src/vcl/ldp.c:2140 > #11 accept4 (fd=32, addr=0x7ffe4a6f63f0, addr_len=0x7ffe4a6f63dc, flags=2048) > at /root/cx/vpp/src/vcl/ldp.c:2178 > #12 0x0000562dee1952b6 in ngx_event_accept () > #13 0x0000562dee19fab8 in ?? () > #14 0x0000562dee194a3a in ngx_process_events_and_timers () > #15 0x0000562dee19f170 in ngx_single_process_cycle () > #16 0x0000562dee17222b in main () > > I found the heapsize can influence the result, > heapsize: 256M ， successful cps: 4869103 ， > heapsize: 128M ， successful cps: 2341828 ， > heapsize: 64M ， successful cps: 1100925 ， > > it seems like a memory leak, > > when I turn nginx configuration's master_process from on to off(only 1 nginx > master), > then test for a longer time(6 min), nginx web server seem to be normal and > will not crash, > > I have tested vsap with LD_PRELOAD, things seem to be the same(nginx will > crash), > > I found the performance test result of vsap LD_PRELOAD in the following > website, > https://docs.fd.io/csit/master/report/vpp_performance_tests/hoststack_testing/vsap/index.html > > <https://docs.fd.io/csit/master/report/vpp_performance_tests/hoststack_testing/vsap/index.html> > and configurations in the following website, > https://git.fd.io/csit/tree/ <https://git.fd.io/csit/tree/> > https://docs.fd.io/csit/master/report/introduction/methodology_hoststack_testing/methodology_vsap_ab_with_nginx.html > > <https://docs.fd.io/csit/master/report/introduction/methodology_hoststack_testing/methodology_vsap_ab_with_nginx.html> > > hope someone may help me, > Thanks, > mercury, > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20560): https://lists.fd.io/g/vpp-dev/message/20560 Mute This Topic: https://lists.fd.io/mt/87398061/21656 Mute #vpp-hoststack:https://lists.fd.io/g/vpp-dev/mutehashtag/vpp-hoststack Mute #hoststack:https://lists.fd.io/g/vpp-dev/mutehashtag/hoststack Mute #vppcom:https://lists.fd.io/g/vpp-dev/mutehashtag/vppcom Mute #nginx:https://lists.fd.io/g/vpp-dev/mutehashtag/nginx Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
