[dpdk-dev] rte_eth_tx_burst() hangs because of nofreedescriptors
Hi Helin,
I'm doing comparison tests today, (with fragmented and small traffic).
On the other hand, I checked some example code from DPDK source:
load_balancer:
app_lcore_io_tx:
n_pkts = rte_eth_tx_burst(
port,
0,
lp->tx.mbuf_out[port].array,
(uint16_t) n_mbufs);
if (unlikely(n_pkts < n_mbufs)) {
uint32_t k;
for (k = n_pkts; k < n_mbufs; k ++) {
struct rte_mbuf *pkt_to_free =
lp->tx.mbuf_out[port].array[k];
rte_pktmbuf_free(pkt_to_free);
}
}
And fragmentation example
http://dpdk.org/doc/api-2.2/ip_fragmentation_2main_8c-example.html
ret = rte_eth_tx_burst(port, queueid, m_table, n);
if (unlikely(ret < n)) {
do {
rte_pktmbuf_free(m_table[ret]);
} while (++ret < n);
}
They works basically the same as in my app, is there any other way to free
segmented packets?
The question is:
if pkt_mbuf next pointer is not none, rte_pktmbuf_free() will free all the
packet of that linklist, but due to packet re-order, I guess those packets may
not sequentially stores in the mbuf array? So how can we correctly free those
fragmented packets if rte_eth_tx_burst() didn't sent out them so we need to
manually free?
Thanks
-- Original --
From: "Zhang, Helin";<helin.zh...@intel.com>;
Date: Tue, Oct 25, 2016 09:05 AM
To: "yingzhi";
Cc: "dev at dpdk.org";
Subject: RE: RE: [dpdk-dev] rte_kni_tx_burst() hangs because of
nofreedescriptors
From: yingzhi [mailto:kai...@qq.com]
Sent: Monday, October 24, 2016 6:39 PM
To: Zhang, Helin
Cc: dev at dpdk.org
Subject: Re: RE: [dpdk-dev] rte_kni_tx_burst() hangs because of no
freedescriptors
Hi Helin,
Thanks for your response, to answer your questions:
1. we send only one packet each time calling rte_kni_tx_burst(), which means
the last argument is 1.
2. it returns 0 because the free mbuf function inside tx_burst will not free
any mbuf:
if (txq->nb_tx_free < txq->tx_free_thresh)
ixgbe_tx_free_bufs(txq);
after this operation, the txq->nb_tx_free is not increased and keeps to be "0"
eventually.
I did some tests today, I commented out this section of ixgbe_rxtx_vec_common.h
-> ixgbe_tx_free_bufs
status = txq->tx_ring[txq->tx_next_dd].wb.status;
if (!(status & IXGBE_ADVTXD_STAT_DD))
return 0;
After ignoring DD bit check, our app runs for about 6 hours without issue. I
suspect there is something wrong in my program set the DD bit somewhere. One of
the possible cause currently I suspect is as far as I know,
rte_pktmbuf_free(mbuf.array[k]) will free the mbuf of the packet and any
fragmented packets following by it. But in our application such as below code
snippet:
auto nb_tx = rte_eth_tx_burst(port, queue, mbuf.array, (uint16_t)
nb_rx);
if (unlikely(nb_tx < nb_rx)) {
for (unsigned k = nb_tx; k < nb_rx; k++) {
rte_pktmbuf_free(mbuf.array[k]);
}
}
[Zhang, Helin] it seems above code piece has memory leak, if the buffer is
chained. After all memory leaked, then the issue comes. Please try to check if
this is the root cause!
In this case if there are fragmented packets and failed transmission, may cause
a mbuf be freed multiple times.
Above is just my suspect, need to do some tests later today or tomorrow.
Thanks
-- Original --
From: "Zhang, Helin";<helin.zh...@intel.com>;
Date: Mon, Oct 24, 2016 11:33 AM
To: "yingzhi";
Cc: "dev at dpdk.org";
Subject: RE: [dpdk-dev] rte_kni_tx_burst() hangs because of no freedescriptors
Hi Yingzhi
Thank you for the reporting! The description is not so clear at least for me.
Please help to narrown down the issue by youself.
How many packets would it have for calling TX function?
Why it would return 0 after calling TX function? No memory? Or return from
else? Have you found anything?
Regards,
Helin
> -Original Message-
> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of yingzhi
> Sent: Sunday, October 23, 2016 9:30 PM
> To: users; dev at dpdk.org
> Subject: [dpdk-dev] rte_kni_tx_burst() hangs because of no free descriptors
>
> -
> Hi Experts,
>
> Background:
>
> We are using DPDK to develop a LoadBalancer following below logic: When
> a new packet is rec
[dpdk-dev] rte_kni_tx_burst() hangs because of no freedescriptors
Hi Helin,
Thanks for your response, to answer your questions:
1. we send only one packet each time calling rte_kni_tx_burst(), which means
the last argument is 1.
2. it returns 0 because the free mbuf function inside tx_burst will not free
any mbuf:
if (txq->nb_tx_free < txq->tx_free_thresh)
ixgbe_tx_free_bufs(txq);
after this operation, the txq->nb_tx_free is not increased and keeps to be "0"
eventually.
I did some tests today, I commented out this section of ixgbe_rxtx_vec_common.h
-> ixgbe_tx_free_bufs
status = txq->tx_ring[txq->tx_next_dd].wb.status;
if (!(status & IXGBE_ADVTXD_STAT_DD))
return 0;
After ignoring DD bit check, our app runs for about 6 hours without issue. I
suspect there is something wrong in my program set the DD bit somewhere. One of
the possible cause currently I suspect is as far as I know,
rte_pktmbuf_free(mbuf.array[k]) will free the mbuf of the packet and any
fragmented packets following by it. But in our application such as below code
snippet:
auto nb_tx = rte_eth_tx_burst(port, queue, mbuf.array, (uint16_t)
nb_rx);
if (unlikely(nb_tx < nb_rx)) {
for (unsigned k = nb_tx; k < nb_rx; k++) {
rte_pktmbuf_free(mbuf.array[k]);
}
}
In this case if there are fragmented packets and failed transmission, may cause
a mbuf be freed multiple times.
Above is just my suspect, need to do some tests later today or tomorrow.
Thanks
-- Original --
From: "Zhang, Helin";<helin.zh...@intel.com>;
Date: Mon, Oct 24, 2016 11:33 AM
To: "yingzhi";
Cc: "dev at dpdk.org";
Subject: RE: [dpdk-dev] rte_kni_tx_burst() hangs because of no freedescriptors
Hi Yingzhi
Thank you for the reporting! The description is not so clear at least for me.
Please help to narrown down the issue by youself.
How many packets would it have for calling TX function?
Why it would return 0 after calling TX function? No memory? Or return from
else? Have you found anything?
Regards,
Helin
> -Original Message-
> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of yingzhi
> Sent: Sunday, October 23, 2016 9:30 PM
> To: users; dev at dpdk.org
> Subject: [dpdk-dev] rte_kni_tx_burst() hangs because of no free descriptors
>
> -
> Hi Experts,
>
> Background:
>
> We are using DPDK to develop a LoadBalancer following below logic: When
> a new packet is received:
> 1. if the dst_addr is management IP, forward to KNI. 2. if the dst_addr is in
> VIP list, select backend and forward(modify dst mac address). 3. otherwise
> drop the packet.
>
> At this stage, we use one single thread for KNI forwarding and another for
> VIP forwarding(forward to eth).
>
> DPDK version: 16.07
> NIC: 82599ES 10-Gigabit SFI/SFP+ Network Connection
> Linux: 14.04.1-Ubuntu x64
>
> Promblem description:
>
> The program runs correctly for sometime(around 2 hours for 400Mb traffic).
> But it it will hang. When problem happens, rte_eth_tx_burst() will not able to
> send out any packets(always returns 0). We tracked into that function and
> noticed it is actually calling ixgbe driver's ixgbe_xmit_pkts_vec() function
> in
> our environment, because we use default tx queue configuration, after
> printing some info, we found if the free function works fine:
> tx_rs_thresh: 32, tx_free_thresh: 32, nb_tx_free: 31
>
> it will trigger free and make 32 more free descriptors:
> tx_rs_thresh: 32, tx_free_thresh: 32, nb_tx_free: 62
>
> but when something going wrong, it will no longer free anything:
> tx_rs_thresh: 32, tx_free_thresh: 32, nb_tx_free: 0 tx_rs_thresh: 32,
> tx_free_thresh: 32, nb_tx_free: 0
>
> It may related with the DD flag of the descriptor but we are not quite sure.
>
> Our program logic:
>
> create two mbuf pools on socket 0, one for rx_queue and one for kni. (all
> lcore threads runs on socket0)
>
> init kni interface with rte_kni_alloc()
>
>
> init one NIC interface with
> rte_eth_dev_configure(); rte_eth_rx_queue_setup();
> rte_eth_tx_queue_setup(); rte_eth_dev_start();
>
>
>
> in the eth main loop: (code is simplified)
> while(1) { n = rte_eth_rx_burst(packets); for (i = 0; i < n; ++i)
> { if
> (SEND_TO_KNI) { m = rte_kni_tx_burst(packets[i]); if
> (m != 1))
> { rte_pktmbuf_free(packets[i]); } }
> if (SEND_TO_ETH)
> { // after modify the packet m =
> rte_eth_tx_burst(packets[i]);
> if (m != 1)) { rte_pktmbuf_free(packets[i]); }
> } //
> otherwise drop the packet rte_pktmbuf_free(packets[i]); } }
>
>
> Please advise if I'm using DPDK in a wrong way. Sorry if I missed something
> basic, I'm new to DPDK.
>
> Thanks in advance
> Best regards
[dpdk-dev] rte_kni_tx_burst() hangs because of no free descriptors
?
Hi Experts,
Background:
We are using DPDK to develop a LoadBalancer following below logic: When a new
packet is received:
1. if the dst_addr is management IP, forward to KNI. 2. if the dst_addr is in
VIP list, select backend and forward(modify dst mac address). 3. otherwise drop
the packet.
At this stage, we use one single thread for KNI forwarding and another for VIP
forwarding(forward to eth).
DPDK version: 16.07
NIC: 82599ES 10-Gigabit SFI/SFP+ Network Connection
Linux: 14.04.1-Ubuntu x64
Promblem description:
The program runs correctly for sometime(around 2 hours for 400Mb traffic). But
it it will hang. When problem happens, rte_eth_tx_burst() will not able to send
out any packets(always returns 0). We tracked into that function and noticed it
is actually calling ixgbe driver's ixgbe_xmit_pkts_vec() function in our
environment, because we use default tx queue configuration, after printing some
info, we found if the free function works fine:
tx_rs_thresh: 32, tx_free_thresh: 32, nb_tx_free: 31
it will trigger free and make 32 more free descriptors:
tx_rs_thresh: 32, tx_free_thresh: 32, nb_tx_free: 62
but when something going wrong, it will no longer free anything:
tx_rs_thresh: 32, tx_free_thresh: 32, nb_tx_free: 0 tx_rs_thresh: 32,
tx_free_thresh: 32, nb_tx_free: 0
It may related with the DD flag of the descriptor but we are not quite sure.
Our program logic:
create two mbuf pools on socket 0, one for rx_queue and one for kni. (all lcore
threads runs on socket0)
init kni interface with rte_kni_alloc()
init one NIC interface with
rte_eth_dev_configure(); rte_eth_rx_queue_setup(); rte_eth_tx_queue_setup();
rte_eth_dev_start();
in the eth main loop: (code is simplified)
while(1) { n = rte_eth_rx_burst(packets); for (i = 0; i < n; ++i)
{ if (SEND_TO_KNI) { m = rte_kni_tx_burst(packets[i]);
if (m != 1)) { rte_pktmbuf_free(packets[i]);
} } if (SEND_TO_ETH) { // after modify the packet
m = rte_eth_tx_burst(packets[i]); if (m != 1)) {
rte_pktmbuf_free(packets[i]); } } //
otherwise drop the packet rte_pktmbuf_free(packets[i]); } }
Please advise if I'm using DPDK in a wrong way. Sorry if I missed something
basic, I'm new to DPDK.
Thanks in advance
Best regards
[dpdk-dev] [dpdk-users] How to printout PMD logs to console
Hi All, I'm doing app debug and would like to see device PMD logs, e.g. dpdk/drivers/net/ixgbe/ixgbe_rxtx.c:1703 PMD_RX_LOG(...) Currently I can only see RTE logs from console. Any comments is appreciated Thanks
