Dear All,
My personal preference is that make test framework implements cpu assignment
code.
It should't be rocket science to parse /sys/devices/system/cpu/online and give
one cpu to each instance.
It will also help to test framework to understand how many parallel jobs it can
run...
Enforcing single cpu assignment in vpp is done intentionally, to avoid cross
numa memory allocation.
If main-core is not specified, vpp simply uses cpu core 1 (unless only 0
exists).
While adding something like "cpu { main-core any} " should be quite straight
forward, it will have broken
behaviour when dpdk is loaded and it will just confuse people. Also, we will
need to come back to the
drawing board when we decide to run multiple workers in make test, as logic
there is more complex and will likely require
rework of the thread placement code.
--
Damjan
> On 27 Jul 2018, at 20:46, Peter Mikus via Lists.Fd.Io
> <pmikus=cisco....@lists.fd.io> wrote:
>
> Hello,
>
> Ø What is the “significant problem” you’re running into?
>
> The problem can be better described as: When python is spawning N instances
> of VPP process, all processes are from unknown reason placed with affinity
> 0x2 (bin 10). This can be verified by taskset –p <vpp_pid>. CFS is then
> placing all VPP process to the same core, making it inefficient on multicore
> jenkins slave container.
> The default vpp startup.conf is not modified thus there is no input to know
> where to pin the vpp threads. Simply one can said or think that this is
> related to python multiprocess/subprocess.popen code, which is hard-setting
> affinity mask to 0x2.
>
> There are multiple solutions for workaround that Juraj proposed or Maciek,
> but none of them is answering why is this happening.
>
> Peter Mikus
> Engineer – Software
> Cisco Systems Limited
>
> From: csit-...@lists.fd.io <mailto:csit-...@lists.fd.io>
> [mailto:csit-...@lists.fd.io <mailto:csit-...@lists.fd.io>] On Behalf Of
> Maciek Konstantynowicz (mkonstan) via Lists.Fd.Io
> Sent: Friday, July 27, 2018 6:53 PM
> To: Alec Hothan (ahothan) <ahot...@cisco.com <mailto:ahot...@cisco.com>>;
> Juraj Linkeš <juraj.lin...@pantheon.tech <mailto:juraj.lin...@pantheon.tech>>
> Cc: csit-...@lists.fd.io <mailto:csit-...@lists.fd.io>
> Subject: Re: [csit-dev] [vpp-dev] Parallel test execution in VPP Test
> Framework
>
> Alec, This is about make test and not real packet forwarding. Per Juraj’s
> patch [1]
>
> Juraj, My understanding is that if you’re starting VPP without specifying
> core placement in startup.conf [2] cpu {..}, then Linux CFS will be placing
> the threads onto available cpu core resources. If you’re saying this is not
> the case, and indeed the wiki comment indicates this, then the way to address
> it is to specify different core for main.c thread per vpp instance.
>
> What is the “significant problem” you’re running into? Are tests not
> executing in parallel using python multiprocessing, are vpp’s having issues,
> else? Could you describe it a bit more?
>
> -Maciek
>
> [1] https://gerrit.fd.io/r/#/c/13491/ <https://gerrit.fd.io/r/#/c/13491/>
> [2] https://git.fd.io/vpp/tree/src/vpp/conf/startup.conf
> <https://git.fd.io/vpp/tree/src/vpp/conf/startup.conf>
>
>
> On 27 Jul 2018, at 17:23, Alec Hothan (ahothan) <ahot...@cisco.com
> <mailto:ahot...@cisco.com>> wrote:
>
> Hi Juraj,
> How many instances and what level of performance are you looking at?
> Even if you assign different cores to each VPP instance, results can be
> skewed due to interference at the LLC and PCIe/NIC level (this can be
> somewhat mitigated by running on separate sockets)
>
> Alec
>
>
> From: <vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>> on behalf of Juraj
> Linkeš <juraj.lin...@pantheon.tech <mailto:juraj.lin...@pantheon.tech>>
> Date: Friday, July 27, 2018 at 7:25 AM
> To: "Maciek Konstantynowicz (mkonstan)" <mkons...@cisco.com
> <mailto:mkons...@cisco.com>>
> Cc: "vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>" <vpp-dev@lists.fd.io
> <mailto:vpp-dev@lists.fd.io>>, csit-dev <csit-...@lists.fd.io
> <mailto:csit-...@lists.fd.io>>
> Subject: Re: [vpp-dev] Parallel test execution in VPP Test Framework
>
> Hi Maciek and vpp-devs,
>
> I've run into a significant problem regarding VPP assignment to cores. All
> VPPs that are spawned are assigned to core 1. I looked at
> https://wiki.fd.io/view/VPP/Command-line_Arguments
> <https://wiki.fd.io/view/VPP/Command-line_Arguments> and I guess it's because
> that's the default behavior of VPP (dpdk coremask is not configured and Note
> that the "main" thread always occupies the lowest core-id specified in the
> DPDK [process-level] coremask.").
>
> Is my reading of the config options accurate?
>
> Obviously, all VPP instances running on the same core goes against running
> the tests on multiple cores. There are a couple of solutions that come to
> mind:
> · Assign VPP instances to cores manually. With possible multiple jobs
> running on a given host, this creates a situation where the different jobs
> don't know cores are already occupied (and by how many VPP instances) and
> thus introduces additional challenges to solve.
> · Add an option to override this default behavior and let the Linux
> CFS scheduler assign VPPs to cores or something similar where VPPs would land
> on different cores.
>
> Is there some other solution?
>
> Vpp-devs, what do you think about the second solution? What it be possible?
>
> Thanks,
> Juraj
>
> From: Maciek Konstantynowicz (mkonstan) [mailto:mkons...@cisco.com
> <mailto:mkons...@cisco.com>]
> Sent: Wednesday, July 25, 2018 1:10 PM
> To: Juraj Linkeš <juraj.lin...@pantheon.tech
> <mailto:juraj.lin...@pantheon.tech>>
> Cc: vpp-dev@lists.fd.io <mailto:vpp-dev@lists.fd.io>; csit-dev
> <csit-...@lists.fd.io <mailto:csit-...@lists.fd.io>>
> Subject: Re: [vpp-dev] Parallel test execution in VPP Test Framework
>
>
>
>
>
>
> On 19 Jul 2018, at 15:44, Juraj Linkeš <juraj.lin...@pantheon.tech
> <mailto:juraj.lin...@pantheon.tech>> wrote:
>
> Hi VPP devs,
>
> I'm implementing parallel test execution of tests in VPP Test Framework (the
> patch is here https://gerrit.fd.io/r/#/c/13491/
> <https://gerrit.fd.io/r/#/c/13491/>) and the last big outstanding question is
> how scalable the parallelization actually is.
>
> That’s a good question. What do the tests say? :)
>
>
>
>
> The tests are spawning one VPP instance per each VPPTestCase class
>
> How many VPP instances are spawned and run in parallel? Cause assuming
> there is at least one VPPTestCase class per test_, that’s 70 VPP
> instances ..
>
>
>
>
>
> and the question is - how do the required compute resources per each VPP
> instance (cpu, ram, shm) scale and how much resources do we need with
> increasing number of VPP instances running in parallel (in the context of VPP
> Test Framework tests)?
>
> I guess this will vary between tests too. FIB scale tests will require
> more fib heap memory. What do the tests say? :)
>
> -Maciek
>
>
>
>
>
> The second question would be a generic "is there anything else I need to be
> aware of when trying to run VPPs in parallel?".
>
> Thanks,
> Juraj
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#9886): https://lists.fd.io/g/vpp-dev/message/9886
> <https://lists.fd.io/g/vpp-dev/message/9886>
> Mute This Topic: https://lists.fd.io/mt/23744978/675185
> <https://lists.fd.io/mt/23744978/675185>
> Group Owner: vpp-dev+ow...@lists.fd.io <mailto:vpp-dev+ow...@lists.fd.io>
> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub
> <https://lists.fd.io/g/vpp-dev/unsub> [mkons...@cisco.com
> <mailto:mkons...@cisco.com>]
> -=-=-=-=-=-=-=-=-=-=-=-
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
>
> View/Reply Online (#9954): https://lists.fd.io/g/vpp-dev/message/9954
> <https://lists.fd.io/g/vpp-dev/message/9954>
> Mute This Topic: https://lists.fd.io/mt/23834426/675642
> <https://lists.fd.io/mt/23834426/675642>
> Group Owner: vpp-dev+ow...@lists.fd.io <mailto:vpp-dev+ow...@lists.fd.io>
> Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub
> <https://lists.fd.io/g/vpp-dev/unsub> [dmar...@me.com
> <mailto:dmar...@me.com>]
> -=-=-=-=-=-=-=-=-=-=-=-
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#9960): https://lists.fd.io/g/vpp-dev/message/9960
Mute This Topic: https://lists.fd.io/mt/23834426/21656
Group Owner: vpp-dev+ow...@lists.fd.io
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
