Hello,
Ø What is the “significant problem” you’re running into? The problem can be better described as: When python is spawning N instances of VPP process, all processes are from unknown reason placed with affinity 0x2 (bin 10). This can be verified by taskset –p <vpp_pid>. CFS is then placing all VPP process to the same core, making it inefficient on multicore jenkins slave container. The default vpp startup.conf is not modified thus there is no input to know where to pin the vpp threads. Simply one can said or think that this is related to python multiprocess/subprocess.popen code, which is hard-setting affinity mask to 0x2. There are multiple solutions for workaround that Juraj proposed or Maciek, but none of them is answering why is this happening. Peter Mikus Engineer – Software Cisco Systems Limited From: csit-...@lists.fd.io [mailto:csit-...@lists.fd.io] On Behalf Of Maciek Konstantynowicz (mkonstan) via Lists.Fd.Io Sent: Friday, July 27, 2018 6:53 PM To: Alec Hothan (ahothan) <ahot...@cisco.com>; Juraj Linkeš <juraj.lin...@pantheon.tech> Cc: csit-...@lists.fd.io Subject: Re: [csit-dev] [vpp-dev] Parallel test execution in VPP Test Framework Alec, This is about make test and not real packet forwarding. Per Juraj’s patch [1] Juraj, My understanding is that if you’re starting VPP without specifying core placement in startup.conf [2] cpu {..}, then Linux CFS will be placing the threads onto available cpu core resources. If you’re saying this is not the case, and indeed the wiki comment indicates this, then the way to address it is to specify different core for main.c thread per vpp instance. What is the “significant problem” you’re running into? Are tests not executing in parallel using python multiprocessing, are vpp’s having issues, else? Could you describe it a bit more? -Maciek [1] https://gerrit.fd.io/r/#/c/13491/ [2] https://git.fd.io/vpp/tree/src/vpp/conf/startup.conf On 27 Jul 2018, at 17:23, Alec Hothan (ahothan) <ahot...@cisco.com<mailto:ahot...@cisco.com>> wrote: Hi Juraj, How many instances and what level of performance are you looking at? Even if you assign different cores to each VPP instance, results can be skewed due to interference at the LLC and PCIe/NIC level (this can be somewhat mitigated by running on separate sockets) Alec From: <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>> on behalf of Juraj Linkeš <juraj.lin...@pantheon.tech<mailto:juraj.lin...@pantheon.tech>> Date: Friday, July 27, 2018 at 7:25 AM To: "Maciek Konstantynowicz (mkonstan)" <mkons...@cisco.com<mailto:mkons...@cisco.com>> Cc: "vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>" <vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>>, csit-dev <csit-...@lists.fd.io<mailto:csit-...@lists.fd.io>> Subject: Re: [vpp-dev] Parallel test execution in VPP Test Framework Hi Maciek and vpp-devs, I've run into a significant problem regarding VPP assignment to cores. All VPPs that are spawned are assigned to core 1. I looked at https://wiki.fd.io/view/VPP/Command-line_Arguments and I guess it's because that's the default behavior of VPP (dpdk coremask is not configured and Note that the "main" thread always occupies the lowest core-id specified in the DPDK [process-level] coremask."). Is my reading of the config options accurate? Obviously, all VPP instances running on the same core goes against running the tests on multiple cores. There are a couple of solutions that come to mind: • Assign VPP instances to cores manually. With possible multiple jobs running on a given host, this creates a situation where the different jobs don't know cores are already occupied (and by how many VPP instances) and thus introduces additional challenges to solve. • Add an option to override this default behavior and let the Linux CFS scheduler assign VPPs to cores or something similar where VPPs would land on different cores. Is there some other solution? Vpp-devs, what do you think about the second solution? What it be possible? Thanks, Juraj From: Maciek Konstantynowicz (mkonstan) [mailto:mkons...@cisco.com] Sent: Wednesday, July 25, 2018 1:10 PM To: Juraj Linkeš <juraj.lin...@pantheon.tech<mailto:juraj.lin...@pantheon.tech>> Cc: vpp-dev@lists.fd.io<mailto:vpp-dev@lists.fd.io>; csit-dev <csit-...@lists.fd.io<mailto:csit-...@lists.fd.io>> Subject: Re: [vpp-dev] Parallel test execution in VPP Test Framework On 19 Jul 2018, at 15:44, Juraj Linkeš <juraj.lin...@pantheon.tech<mailto:juraj.lin...@pantheon.tech>> wrote: Hi VPP devs, I'm implementing parallel test execution of tests in VPP Test Framework (the patch is here https://gerrit.fd.io/r/#/c/13491/) and the last big outstanding question is how scalable the parallelization actually is. That’s a good question. What do the tests say? :) The tests are spawning one VPP instance per each VPPTestCase class How many VPP instances are spawned and run in parallel? Cause assuming there is at least one VPPTestCase class per test_, that’s 70 VPP instances .. and the question is - how do the required compute resources per each VPP instance (cpu, ram, shm) scale and how much resources do we need with increasing number of VPP instances running in parallel (in the context of VPP Test Framework tests)? I guess this will vary between tests too. FIB scale tests will require more fib heap memory. What do the tests say? :) -Maciek The second question would be a generic "is there anything else I need to be aware of when trying to run VPPs in parallel?". Thanks, Juraj -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#9886): https://lists.fd.io/g/vpp-dev/message/9886 Mute This Topic: https://lists.fd.io/mt/23744978/675185 Group Owner: vpp-dev+ow...@lists.fd.io<mailto:vpp-dev+ow...@lists.fd.io> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [mkons...@cisco.com<mailto:mkons...@cisco.com>] -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#9954): https://lists.fd.io/g/vpp-dev/message/9954 Mute This Topic: https://lists.fd.io/mt/23834426/21656 Group Owner: vpp-dev+ow...@lists.fd.io Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
