On 2020-12-12 11:22, Singh, Jasvinder wrote:
On 12 Dec 2020, at 01:45, Alex Kiselev <[email protected]> wrote:On 2020-12-12 01:54, Alex Kiselev wrote:also, tc_ov_credits is set to tc_ov_wm_min only in the case of constant congestion and today I detected the problem when there was no congestion.On 2020-12-12 01:45, Alex Kiselev wrote: On 2020-12-12 01:20, Singh, Jasvinder wrote:But that is a perfectly normal situation and that's exactly the ideaOn 11 Dec 2020, at 23:37, Alex Kiselev <[email protected]> wrote: On 2020-12-11 23:55, Singh, Jasvinder wrote: On 11 Dec 2020, at 22:27, Alex Kiselev <[email protected]> wrote:On 2020-12-11 23:06, Singh, Jasvinder wrote:On 11 Dec 2020, at 21:29, Alex Kiselev <[email protected]> wrote: On 2020-12-08 14:24, Singh, Jasvinder wrote: <snip>[JS] now, returning to 1 mbps pipes situation, try reducing tcperiodfirst at subport and then at pipe level, if that help in gettingeventraffic across low bandwidth pipes.reducing subport tc from 10 to 5 period also solved the problem with 1 Mbit/s pipes. so, my second problem has been solved, but the first one with some of low bandwidth pipes stop transmitting still remains. I see, try removing "pkt_len <= pipe_tc_ov_credits" condition in thegrinder_credits_check() code for oversubscription case, instead usethis pkt_len <= pipe_tc_credits + pipe_tc_ov_credits; if I do what you suggest, I will get this code enough_credits = (pkt_len <= subport_tb_credits) && (pkt_len <= subport_tc_credits) && (pkt_len <= pipe_tb_credits) && (pkt_len <= pipe_tc_credits) && (pkt_len <= pipe_tc_credits + pipe_tc_ov_credits); And this doesn't make sense since if condition pkt_len <= pipe_tc_credits is truethen condition (pkt_len <= pipe_tc_credits + pipe_tc_ov_credits) isalso always true. [JS] my suggestion is to remove“pkt_len <= pipe_tc_credits“, “ pkt_len <= pipe_tc_ov_credits”and use only “pkt_len <= pipe_tc_credits + pipe_tc_ov_credits“ While keeping tc_ov flag on. Your suggestion just turns off TC_OV feature.I don't see your point. This new suggestion will also effectively turn off the TC_OV feature since the only effect of enabling TC_OV is adding additional condition pkt_len <= pipe_tc_ov_creditswhich doesn't allow a pipe to spend more resources than it should.And in the case of support congestion a pipe should spent less than %100 of pipe's maximum rate.And you suggest to allow pipe to spend 100% of it's rate plus someextra. I guess effect of this would even more unfair support's bandwidth distibution. Btw, a pipe might stop transmitting even when there is no congestion at a subport.Although I didn’t try this solution but the idea here is - in aparticular round, of pkt_len is less than pipe_tc_credits( which isaconstant value each time) but greater than pipe_tc_ov_credits, thenit might hit the situation when no packet will be scheduled from the pipe even though there are fixed credits greater than packet size is available.behind TC_OV. It means a pipe should wait for the next subport->tc_ov_period_id when pipe_tc_ov_credits will be reset to a new value But here it’s not guaranteed that new value of pipe_tc_ov_credits will be sufficient for low bandwidth pipe to send their packets as each time pipe_tc_ov_credits is freshly computed.pipe->tc_ov_credits = subport->tc_ov_wm * params->tc_ov_weight; which allows the pipe to continue transmitting.No that won’t happen if new tc_ov_credits value is again less than pkt_len and will hit deadlock.new tc_ov_credits can't not be less than subport->tc_ov_wm_min, and tc_ov_wm_min is equal to port->mtu.all my scheduler ports configured with mtu 1522. etherdev ports also usesthe same mtu, therefore there should be no packets bigger that 1522.so, it's highly unlikely that tc_ov_credits is always set to a valueless than pkt_size. The only scenario in which this might be the case iswhen scheduler port get a corrupted mbuf with incorrect pkt len which cause a queue deadlock.also, a defragmented ipv4 packet (multisegment mbuf) might have pkt_len much bigger then scheduler port's MTU, therefore you are right, there is absolutely no guarantee that packet will not cause queue's deadlock. and this explanation sounds very plausibleto me and I bet this is my case.But you mentioned earlier that your packet length is low, never exceeding above threshold. May be test with fixed 256/512 bytes size pkt if faces the same no transmission situation.
No, I could only say so about the test lab which I were using to test the pipe fairness and which uses a packet generator with constant pkt size. My main issue happens in a productive network. And I mentioned that it is a network to provide internet access to residential customers therefore packet sizes are up to 1522 bytes. Also, fragmented packets are valid packets in such networks. My applicationperforms ipv4 defragmentation and then sends to the scheduler, so the scheduler
might receive the multisegment packet up to 1522 * 8 size.
Maybe I should increase port's MTU? to 1540?Yes, so reducing tc period makes the case when all pipes ( high n lowAnd it could not cause a permanent pipe stop which is what I am facing.In fairness, pipe should send the as much as packets which consumes pipe_tc_credits, regardless of extra pipe_tc_ov_credits which is extra on top of pipe_tc_credits.I think it's quite the opposite. That's why after I reduced the support tc_period I got much more fairness. Since reducing subport tc_period also reduce the tc_ov_wm_max value. s->tc_ov_wm_max = rte_sched_time_ms_to_bytes(params->tc_period, port->pipe_tc3_rate_max) as a result a pipe transmits less bytes in one round. so pipe rotation inside a grinder happens much more often and a pipe can't monopolise resources. in other sos implementation this is called "quantum".bandwidth) gets lower values of tc_ov_credits values which allowlesser transmission from higher bw pipes and leave bandwidth for lowbw pipes. So, here is the thing- Either tune tc period to a valuewhich prevent high bw pipe hogging most of bw or makes changes in the code, where oversubscription add extra credits on top of guaranteed.One question, don’t your low bw pipes have higher priority traffic tc0, tc1, tc2 . Packets from those tc must be going out. Isn’t this the case ?well, it would be the case after I find out what's going on. Right now I am using a tos2tc map configured in such a way that all ipv4 packets with any TOS values goes into TC3.rcv 0 rx rate 7324160 nb pkts 5722 rcv 1 rx rate 7281920 nb pkts 5689 rcv 2 rx rate 7226880 nb pkts 5646 rcv 3 rx rate 7124480 nb pkts 5566 rcv 4 rx rate 7324160 nb pkts 5722 rcv 5 rx rate 7271680 nb pkts 5681 rcv 6 rx rate 7188480 nb pkts 5616 rcv 7 rx rate 7150080 nb pkts 5586 rcv 8 rx rate 7328000 nb pkts 5725 rcv 9 rx rate 7249920 nb pkts 5664 rcv 10 rx rate 7188480 nb pkts 5616 rcv 11 rx rate 7179520 nbpkts5609 rcv 12 rx rate 7324160 nb pkts 5722 rcv 13 rx rate7208960 nbpkts 5632 rcv 14 rx rate 7152640 nb pkts 5588 rcv 15 rx rate 7127040 nb pkts 5568 rcv 16 rx rate 7303680 nb pkts 5706 .... rcv 587 rx rate 2406400 nb pkts 1880 rcv 588 rx rate 2406400 nbpkts1880 rcv 589 rx rate 2406400 nb pkts 1880 rcv 590 rx rate2406400 nbpkts 1880 rcv 591 rx rate 2406400 nb pkts 1880 rcv 592 rx rate 2398720 nb pkts 1874 rcv 593 rx rate 2400000 nb pkts 1875 rcv594 rxrate 2400000 nb pkts 1875 rcv 595 rx rate 2400000 nb pkts 1875rcv596 rx rate 2401280 nb pkts 1876 rcv 597 rx rate 2401280 nbpkts1876 rcv 598 rx rate 2401280 nb pkts 1876 rcv 599 rx rate2402560 nbpkts 1877 rx rate sum 3156416000... despite that there is _NO_ congestion... congestion at the subport or pipe.And the subport !! doesn't use about 42 mbit/s of available bandwidth. The only difference is those test configurations is TC of generated traffic. Test 1 uses TC 1 while test 2 uses TC 3 (which is use TC_OV function). So, enabling TC_OV changes the results dramatically. ## ## test1 ## hqos add profile 7 rate 2 M size 1000000 tc period 40 # qos test port hqos add port 1 rate 10 G mtu 1522 frame overhead 24 queuesizes64 64 64 64 hqos add port 1 subport 0 rate 300 M size 1000000 tc period10hqos add port 1 subport 0 pipes 2000 profile 7 hqos add port1subport 0 pipes 200 profile 23 hqos set port 1 lcore 3 port1subport rate 300 M number of tx flows 300 generator tx rate1M TC1 ... rcv 284 rx rate 995840 nb pkts 778 rcv 285 rx rate 995840nbpkts 778 rcv 286 rx rate 995840 nb pkts 778 rcv 287 rx rate 995840 nb pkts 778 rcv 288 rx rate 995840 nb pkts 778 rcv289rx rate 995840 nb pkts 778 rcv 290 rx rate 995840 nb pkts778rcv 291 rx rate 995840 nb pkts 778 rcv 292 rx rate 995840nbpkts 778 rcv 293 rx rate 995840 nb pkts 778 rcv 294 rx rate 995840 nb pkts 778 ... sum pipe's rx rate is 298 494 720 OK. The subport rate is equally distributed to 300 pipes. ## ## test 2 ## hqos add profile 7 rate 2 M size 1000000 tc period 40 # qos test port hqos add port 1 rate 10 G mtu 1522 frame overhead 24 queuesizes64 64 64 64 hqos add port 1 subport 0 rate 300 M size 1000000 tc period10hqos add port 1 subport 0 pipes 2000 profile 7 hqos add port1subport 0 pipes 200 profile 23 hqos set port 1 lcore 3 port1subport rate 300 M number of tx flows 300 generator tx rate1M TC3 h5 ~ # rcli sh qos rcv rcv 0 rx rate 875520 nb pkts 684 rcv 1 rx rate 856320 nb pkts 669 rcv 2 rx rate 849920 nb pkts 664 rcv 3 rx rate 853760 nb pkts 667 rcv 4 rx rate 867840 nb pkts 678 rcv 5 rx rate 844800 nb pkts 660 rcv 6 rx rate 852480 nb pkts 666 rcv 7 rx rate 855040 nb pkts 668 rcv 8 rx rate 865280 nb pkts 676 rcv 9 rx rate 846080 nb pkts 661 rcv 10 rx rate 858880 nb pkts 671 rcv 11 rx rate 870400nbpkts 680 rcv 12 rx rate 864000 nb pkts 675 rcv 13 rx rate 852480 nb pkts 666 rcv 14 rx rate 855040 nb pkts 668 rcv15rx rate 857600 nb pkts 670 rcv 16 rx rate 864000 nb pkts675rcv 17 rx rate 866560 nb pkts 677 rcv 18 rx rate 865280nbpkts 676 rcv 19 rx rate 858880 nb pkts 671 rcv 20 rx rate 856320 nb pkts 669 rcv 21 rx rate 864000 nb pkts 675 rcv22rx rate 869120 nb pkts 679 rcv 23 rx rate 856320 nb pkts669rcv 24 rx rate 862720 nb pkts 674 rcv 25 rx rate 865280nbpkts 676 rcv 26 rx rate 867840 nb pkts 678 rcv 27 rx rate 870400 nb pkts 680 rcv 28 rx rate 860160 nb pkts 672 rcv29rx rate 870400 nb pkts 680 rcv 30 rx rate 869120 nb pkts679rcv 31 rx rate 870400 nb pkts 680 rcv 32 rx rate 858880nbpkts 671 rcv 33 rx rate 858880 nb pkts 671 rcv 34 rx rate 852480 nb pkts 666 rcv 35 rx rate 874240 nb pkts 683 rcv36rx rate 855040 nb pkts 668 rcv 37 rx rate 853760 nb pkts667rcv 38 rx rate 869120 nb pkts 679 rcv 39 rx rate 885760nbpkts 692 rcv 40 rx rate 861440 nb pkts 673 rcv 41 rx rate 852480 nb pkts 666 rcv 42 rx rate 871680 nb pkts 681 ... ... rcv 288 rx rate 766720 nb pkts 599 rcv 289 rx rate 766720nbpkts 599 rcv 290 rx rate 766720 nb pkts 599 rcv 291 rx rate 766720 nb pkts 599 rcv 292 rx rate 762880 nb pkts 596 rcv293rx rate 762880 nb pkts 596 rcv 294 rx rate 762880 nb pkts596rcv 295 rx rate 760320 nb pkts 594 rcv 296 rx rate 604160nbpkts 472 rcv 297 rx rate 604160 nb pkts 472 rcv 298 rx rate 604160 nb pkts 472 rcv 299 rx rate 604160 nb pkts 472 rxratesum 258839040 FAILED. The subport rate is distributed NOT equally between 300pipes.Some subport bandwith (about 42) is not being used!
