Re: [vpp-dev] test performance of nginx using vpp host stack#nginx
I completely missed that, thanks Hanlin! For perf tests always use release images. Regards, Florin > On Jan 20, 2022, at 6:27 PM, weizhen9...@163.com wrote: > > Yes, I use a debug version. > Thank you. > > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20773): https://lists.fd.io/g/vpp-dev/message/20773 Mute This Topic: https://lists.fd.io/mt/88456731/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] test performance of nginx using vpp host stack#nginx
Yes, I use a debug version. Thank you. -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20772): https://lists.fd.io/g/vpp-dev/message/20772 Mute This Topic: https://lists.fd.io/mt/88456731/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] test performance of nginx using vpp host stack#nginx
Latency with vpp is very high for ~50% of the requests so either some config is
still off or something loses packets.
To check if the later is the case, try “show error” and check tcp counters for
out of order enqueues or “show tcp stats” and check retransmission counters.
To make sure we haven’t recently introduced some regression I just ran a
similar test locally and results are pretty good:
Running 10s test @ http://6.0.1.1/64B.json
30 threads and 300 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 494.20us5.33ms 243.45ms 99.76%
Req/Sec32.50k 3.47k 46.58k80.35%
9736456 requests in 10.10s, 2.63GB read
Requests/sec: 964250.62
Transfer/sec:266.68MB
Note however that the two servers that run vpp and nginx on one side and wrk on
the other are Xeon(R) Gold 6146 (Skylake) and are connected by a 40Gbps nic.
Configuration should be similar to what you’re testing, i.e., vpp 1 worker,
nginx 4 workers and nginx is serving those 64B straight from memory:
location /64B.json { return 200 '{"status":"success","result”: “fill in 64B of
characters here” }’; }
Recent CSIT results[1], obtained with ab as opposed to wrk, also look pretty
good although in those tests 2 vpp workers are used.
Regards,
Florin
[1]
https://s3-docs.fd.io/csit/master/report/vpp_performance_tests/hoststack_testing/vsap/index.html#t1c-cx556a-base-scale-rps
> On Jan 20, 2022, at 4:58 PM, weizhen9...@163.com wrote:
>
> According to config you said, the performance of nginx using vpp host stack
> is still low.
> The performance of nginx using vpp host stack:
>
> The performance of nginx using kernel host stack:
>
> What can I do?
> Thank you.
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#20770): https://lists.fd.io/g/vpp-dev/message/20770
Mute This Topic: https://lists.fd.io/mt/88456731/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Thu 13-Jan FD.io CSIT talk at LFN Developer & Testing Forum
Here pointers to materials and recordings from the session last week - Enjoy: CSIT Talk: Performance Benchmarking, Anomaly Detection, Results Tracking Session wiki page [1]. Slides lfn-ddts21-fdio-csit.pdf [2]. Recording lfn-ddts21-fdio-csit.mp4 [3]. Topics - with timestamps in recording: FD.io Overview VPP and CSIT core projects - 1m 50s FD.io CSIT Daily Trending and Release Reports - 11m 35s Anomaly Detection - 24m 30s Data Processing Infrastructure - 33m 20s Wrap-up - 39m 50s Q - 42m 0s Backup slide - jumpavg Compression Algorithm - 56m 50s End - 1h 04m 0s [1] https://wiki.lfnetworking.org/pages/viewpage.action?pageId=65538382 [2] https://wiki.lfnetworking.org/download/attachments/65538382/lfn-ddts21-fdio-csit.pdf?version=1=1642187365000=v2 [3] https://wiki.lfnetworking.org/download/attachments/65538382/lfn-ddts21-fdio-csit.mp4?version=1=1642175623000=v2 > On 11 Jan 2022, at 15:58, mkonstan wrote: > > Dear All, > > Some of us from CSIT will be giving a talk at this week LFN Developer & > Testing Forum (virtual event via Zoom, and it’s free to attend). > > Title: FD.io CSIT Performance Benchmarking, Anomaly Detection, Results > Tracking. > > Time: Thursday 13th January 2021 13:00-14:00 GMT (London) > > Abstract: > > In this talk we introduce FD.io CSIT core sub-project and give an > overview of the ways its Presentation-Analytics-Layer component > presents test results. We will focus on performance results and anomaly > detection, as a way to distinguish (probably) real performance changes > from the noisy data. And provide a sneak peak into the way test results > data is processed. Will use performance data from CI/CD automated tests > executed on Intel Xeon(skx, clx, icx), Atom, Arm servers running in > labs hosted by LFN FD.io. If you are interested in network performance > testing in general, or FD.io project specifically, this talk is for > you. > > For FD.io CSIT session details see: > >https://wiki.lfnetworking.org/pages/viewpage.action?pageId=65538382 > > Event details and how to register: > >https://wiki.lfnetworking.org/pages/viewpage.action?pageId=53609047 > > Detailed schedule (make sure to select your timezone): > >https://teamup.com/ksgw6qzqcmg9zbzsfq?date=2022-01-10=md4 > > Cheers, > Maciek > -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20768): https://lists.fd.io/g/vpp-dev/message/20768 Mute This Topic: https://lists.fd.io/mt/88351902/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] VPP 22.02 RC1 milestone is complete!
Excellent! Thanks for getting the VPP 22.02 release cycle under way :D -daw- On 1/19/2022 3:29 PM, Andrew Yourtchenko wrote: Hi all, the VPP 22.02 RC1 milestone is complete! The master branch is open for all commits, the stable/2202 is open for the bugfixes (other than the exception related to the wireguard that was discussed in the other thread today). The artifacts are available at the packagecloud.io/fdio/2202 The RC2 milestone is on the 9th of February 2022. Thanks a lot! --a /* your friendly 22.02 release manager */ -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#20767): https://lists.fd.io/g/vpp-dev/message/20767 Mute This Topic: https://lists.fd.io/mt/88543995/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
Re: [vpp-dev] Bihash is considered thread-safe but probably shouldn't
Let me resurrect this discussion as my patch is still in gerrit. Nick, any chance you can submit your proposal as discussed bellow? Thanks, — Damjan > On 04.11.2021., at 19:35, Damjan Marion via lists.fd.io > wrote: > > > Dear Nick, > > Will be great if you can support your proposal with a running code so we can > understand exactly what it means. > > — > Damjan > >> On 04.11.2021., at 19:24, Nick Zavaritsky wrote: >> >> Hi, thanks for an insightful discussion! >> >> I do understand that high performance is one of the most important goals of >> vpp, therefore certain solutions might not fly. From my POV, the version >> counter would be an improvement. It definitely decreases the probability of >> triggering the bug. >> >> Concerning isolcpus, currently this is presented as an optimisation, not a >> prerequisite. Without isolcpus, a thread could get preempted for arbitrarily >> long. Meaning that no matter how many bits we allocate for the version >> field, occasionally they won’t be enough. >> >> I’d love to have something that’s robust no matter how the threads are >> scheduled. Would it be possible to use vpp benchmarking lab to evaluate the >> performance impact of the proposed solutions? >> >> Finally, I'd like to rehash the reader lock proposal. The idea was that we >> don’t introduce any atomic operations in the reader path. A reader >> *publishes* the bucket number it is about to examine in int >> rlock[MAX_THREADS] array. Every thread uses a distinct cell in rlock >> (determined by the thread id), therefore it could be a regular write >> followed by a barrier. Eliminate false sharing with padding. >> >> Writer locks a bucket as currently implemented (CAS) and then waits until >> the bucket number disappears from rlock[]. >> >> Reader publishes the bucket number and then checks if the bucket is locked >> (regular write, barrier, regular read). Good to go if not locked, otherwise >> remove the bucket number from rlock, wait for the lock to get released, >> restart. >> >> The proposal doesn’t introduce any new atomic operations. There still might >> be a slowdown due to cache line ping-pong in the rlock array. In the worst >> case, it costs us 1 extra cache miss for the reader. Could be coalesced with >> the bucket prefetch, making it essentially free (few if any bihash users >> prefetch buckets). >> >> Best, >> >> Nick >> >> >>> On 3. Nov 2021, at 21:29, Florin Coras via lists.fd.io >>> wrote: >>> >>> >>> Agreed it’s unlikely so maybe just use the 2 bits left for the epoch >>> counter as a middle ground? The new approach should be better either way :-) >>> >>> Florin >>> >>> On Nov 3, 2021, at 11:55 AM, Damjan Marion wrote: What about the following, we shift offset by 6, as all buckets are aligned to 64, anyway, and that gives us 6 more bits so we can have 8 bit epoch counter…. ? — Damjan > On 03.11.2021., at 19:45, Damjan Marion wrote: > > > > yes, i am aware of that, it is extremelly unlikely and only way i can see > this fixed is introducing epoch on the bucket level but we dont have > enough space there…. > > — > Damjan > >> On 03.11.2021., at 19:16, Florin Coras wrote: >> >> Hi Damjan, >> >> Definitely like the scheme but the change bit might not be enough, >> unless I’m misunderstanding. For instance, two consecutive updates to a >> bucket before reader grabs b1 will hide the change. >> >> Florin >> >>> On Nov 3, 2021, at 9:36 AM, Damjan Marion via lists.fd.io >>> wrote: >>> >>> >>> Agree with Dave on atomic ops being bad on the reader side. >>> >>> What about following schema: >>> >>> As bucket is just u64 value on the reader side we grab bucket before >>> (b0) and after (b1) search operation. >>> >>> If search finds entry, we simply do 2 checks: >>> - that b0 is equal to b1 >>> - that lock bit is not set in both of them >>> If check fails, we simply retry. >>> >>> On the writer side, we have add, remove and replace operations. >>> First 2 alter refcnt which is part of bucket. >>> To deal with replace case we introduce another bit (change bit) which >>> is flipped every time data is changed in the bucket. >>> >>> Here are possible scenarios: >>> >>> - reader grabs b0 before lock and b1 after unlock >>>- add, del - refcnt and change bit will be different between b0 and >>> b1 causing retry >>>- replace - change bit will be different between b0 and b1 causing >>> retry >>> >>> - reader grabs b0 after lock and/or b1 before unlock >>>- lock bit will be set causing retry >>> >>> Of course, this to work properly we need to ensure proper memory >>> ordering (i.e. avoid bucket change to be visible to remote thread
