Hi Gayathri,
> -Original Message-
> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of
> gayathri.manepalli at wipro.com
> Sent: Friday, May 13, 2016 7:44 AM
> To: dev at dpdk.org
> Cc: Stokes, Ian
> Subject: [dpdk-dev] [ovs-dev] Traffic scheduling by qos_sched library in
> DPDK
>
> Hi Team,
>
> I started working on implementing the QoS Shaping in OVS+DPDK by making
> use of rte_sched library provided in DPDK.
Great news, thank you!
Meanwhile to compare the
> performance, started performance test with DPDK sample scheduling
> application. Below are the configuration details of system which I am using,
>
> Server Type : Intel ATOM
>
> Huge page configuration: (Each page of 2M size)
>
> [root at ATOM qos_sched]# grep -i huge /proc/meminfo
> AnonHugePages: 90112 kB
> HugePages_Total:7168
> HugePages_Free: 7168
> HugePages_Rsvd:0
> HugePages_Surp:0
> Hugepagesize: 2048 kB
>
> Port Capacity : 1G (All four ports)
> I am able to successfully run the qos_sched with three pfc's as below,
>
> ./build/qos_sched -c 0x3e -n 1 --socket-mem 14336 -- --pfc "0,1,2,3" --pfc
> "1,0,3,2" --pfc "2,3,4,5" --cfg ./profile.cfg
>
> Issue :
>
> When I am trying to add one more profile configuration flow(4th one) , I am
> getting below error
>
> Command : ./build/qos_sched -c 0x3e -n 1 --socket-mem 14336 -- --pfc
> "0,1,2,3" --pfc "1,0,3,2" --pfc "2,3,4,5" --pfc "3,2,5,4" --cfg ./profile.cfg
>
> Error:
>
> done: Link Up - speed 1000 Mbps - full-duplex
> SCHED: Low level config for pipe profile 0:
> Token bucket: period = 1, credits per period = 1, size = 10
> Traffic classes: period = 500, credits per period = [500,
> 500,
> 500, 500]
>Traffic class 3 oversubscription: weight = 0
> WRR cost: [1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1]
> SCHED: Low level config for subport 0:
> Token bucket: period = 1, credits per period = 1, size = 10
> Traffic classes: period = 125, credits per period = [125,
> 125,
> 125, 125]
> Traffic class 3 oversubscription: wm min = 0, wm max = 0
> EAL: Error - exiting with code: 1
> Cause: Cannot init mbuf pool for socket 3
>
> Analysis:
>
> I have analyzed DPDK source code to find the root cause. I could see that in
> qos_sched, memory allocation while creating each mbug pool
> (rte_mempool_create) for corresponding RX port is as below,
>
>
> MBUF_SIZE = (1528 + sizeof(struct rte_mbuf) + RTE_PKTMBUF_HEADROOM)
>
> mp_size = 2*1024*1024
>
>
>
> From above I understood that, for each pfc/ rx port around 4G of huge pages
> are consumed. Whereas ATOM is capable of maximum 7168 huge pages of
> 2M which is 14336M in total. So I am not able to configure beyond three pfc's.
> But I would like to measure the performance with 4 port & 6 port scenario
> which requires 4-6 pfc's configured.
>
>
>
> Is there any other alternative through which I can configure more number of
> pfc's with my system configuration provided above.
>
>
Yes, you are probably running out of memory.
QoS hierarchical scheduler can be seen like a big reservoir of packets. Each
rte_sched_port object basically has thousands of packet queues internally
(number of queues is configurable). Ideally, to protect against the worst case
scenario, , the number of buffers provisioned per each output port that has the
hierarchical scheduler enabled needs to be at least equal to: number of
scheduler queues x queue size. For example, for 64K queues (4K pipes with 16
queues each) of 64 packets each, this means 4M buffers per output port, which
(assuming 2KB buffers in the mempool) leads to 8 GB of memory per output port.
In our examples/qos_sched sample application we decided to go mid-way instead
of worst case scenario, therefore we use 2M buffers per output port.
So possible workarounds for you to maximize the amount of ports with
hierarchical scheduler support given a fixed amount memory:
1. Lower the amount of buffers you use per output port, e.g. from 2M to 1M and
restest;
2. Use less pipes per output port (configurable), so number of scheduling
queues is less, which reduces the pressure on mempool size.
Note that you can use distinct mempools per output port (like in
examples/qos_sched application) or a single big global mempool shared by all
ports; this is transparent for the librte_sched library.
Regards,
Cristian
>
> Thanks & Regards,
>
> Gayathri
>
> The information contained in this electronic message and any attachments to
> this message are intended for the exclusive