> On Sep 16, 2019, at 8:49 AM, Damjan Marion <[email protected]> wrote: > > > >> On 13 Sep 2019, at 18:58, Christian Hopps <[email protected]> wrote: >> >> >> >>> On Sep 13, 2019, at 12:41 PM, Christian Hopps <[email protected]> wrote: >>> >>> Signed PGP part >>> >>> >>>> On Sep 13, 2019, at 12:23 PM, Damjan Marion <[email protected]> wrote: >>>> >>>> >>>> >>>>> On 13 Sep 2019, at 18:15, Christian Hopps <[email protected]> wrote: >>>>> >>>>> Is it possible to profile vpp with gprof? >>>> >>>> What exactly do you want to profile? Can you describe your use case? >>> >>> Trying to figure out why my code is not running as fast as it should. :) >>> >>> The code is currently pretty straight-forward: >>> >>> - main thread (0) is on a core (3) >>> - send thread (1, worker 0) is on core 0 >>> - receive thread (2, worker 1) is on core 1. >>> - processing thread (3 worker 2) is on core 2, and is not doing anything >>> right now. >>> >>> Operation: >>> >>> - The send code is sending encrypted (using cryptodev mvsam) fixed sized >>> (1500b) empty packets (i.e., the data is just whatever is there when I >>> advance current_length of the buffer). >>> >>> - The receive code is decrypting (again using cryptodev mvsam) these packes >>> and doing a hand-off to the processing thread which does nothing with them. >>> The drops are detected using ESP sequence number, prior to doing the >>> hand-off -- IOW that's all my code is doing other than the hand off >>> post-decrypt. >>> >>> I'm sending on fixed intervals to achieve a throughput of 1Ghz (so 83333pps >>> spaced 12000ns apart). I'm currently seeing dropped packets (either inbound >>> or outbound as yet to be determined) and many missed send slots in my >>> polling (sending) routine. >> >> I should note, I don't mind if these packets are clumped together as long as >> the average is 83333pps, so the send routine is written to send as many as >> it should even when it misses it's send slot[s] thus using the standard >> vector of packets design of vpp. However, currently I'm even not bothering >> to catch-up when I miss sending these empty packets for missed timing slots, >> so the performance is even worse than it seems at first glance -- this not >> sending isn't a drop though it's just sending less often. > > > Are you aware that if there is no rx interfaces in polling mode (or any other > input node in polling mode) assigned to specific worker, then that worker > will end up sleeping in epoll_wait() for 10msec? > > May that be cause of your problem?
I wasn't aware of this, and I don't believe impacted my problem; however, it *is* useful information for other aspects of my design! :) I believe I may have been chasing a red-herring, I was seeing packet drops and had been assuming they were due to missed sends; however, I've now discovered that I'm seeing interface Rx errors that correlate to my application seeing the drops. I still need to verify that I'm not causing the Rx interface errors somehow. In any case, I've now changed the design to send and receive the fixed sized encrypted/decrypted tunnel packets on a single core. The to-be-tunneled input is received on a second (interface) core, and the decapped (from-the-tunnel) output is sent to a third core. That third core is output only so it could be that it might hit the epoll_wait(). I'd guess vpp only does this if there's no handoff queue frames ready? Under normally operation there should almost always be handoff frames waiting; however, it might be nice to avoid this epoll_wait in my case so that for infrequent tunnel use there isn't a 10ms latency when traffic starts. This is the unix-epoll-input node? I think I may be avoiding this currently b/c DPDK has it's cryptodev input polling node enabled and running on every worker; however, that's sub-optimal and I'd like to change it. Thanks, Chris.
signature.asc
Description: Message signed with OpenPGP
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#13991): https://lists.fd.io/g/vpp-dev/message/13991 Mute This Topic: https://lists.fd.io/mt/34128898/21656 Group Owner: [email protected] Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
