On 12/03/2018 12:08 PM, Tariq Toukan wrote:
On 09/03/2018 10:56 PM, Jesper Dangaard Brouer wrote:
This patch shows how it is possible to have both the driver local page
cache, which uses elevated refcnt for "catching"/avoiding SKB
put_page. And at the same time, have pages getting returned to the
page_pool from ndp_xdp_xmit DMA completion.
Performance is surprisingly good. Tested DMA-TX completion on ixgbe,
that calls "xdp_return_frame", which call page_pool_put_page().
Stats show DMA-TX-completion runs on CPU#9 and mlx5 RX runs on CPU#5.
(Internally page_pool uses ptr_ring, which is what gives the good
cross CPU performance).
Show adapter(s) (ixgbe2 mlx5p2) statistics (ONLY that changed!)
Ethtool(ixgbe2 ) stat: 732863573 ( 732,863,573) <= tx_bytes /sec
Ethtool(ixgbe2 ) stat: 781724427 ( 781,724,427) <= tx_bytes_nic
/sec
Ethtool(ixgbe2 ) stat: 12214393 ( 12,214,393) <= tx_packets /sec
Ethtool(ixgbe2 ) stat: 12214435 ( 12,214,435) <= tx_pkts_nic
/sec
Ethtool(mlx5p2 ) stat: 12211786 ( 12,211,786) <=
rx3_cache_empty /sec
Ethtool(mlx5p2 ) stat: 36506736 ( 36,506,736) <=
rx_64_bytes_phy /sec
Ethtool(mlx5p2 ) stat: 2336430575 ( 2,336,430,575) <= rx_bytes_phy
/sec
Ethtool(mlx5p2 ) stat: 12211786 ( 12,211,786) <=
rx_cache_empty /sec
Ethtool(mlx5p2 ) stat: 22823073 ( 22,823,073) <=
rx_discards_phy /sec
Ethtool(mlx5p2 ) stat: 1471860 ( 1,471,860) <=
rx_out_of_buffer /sec
Ethtool(mlx5p2 ) stat: 36506715 ( 36,506,715) <=
rx_packets_phy /sec
Ethtool(mlx5p2 ) stat: 2336542282 ( 2,336,542,282) <=
rx_prio0_bytes /sec
Ethtool(mlx5p2 ) stat: 13683921 ( 13,683,921) <=
rx_prio0_packets /sec
Ethtool(mlx5p2 ) stat: 821015537 ( 821,015,537) <=
rx_vport_unicast_bytes /sec
Ethtool(mlx5p2 ) stat: 13683608 ( 13,683,608) <=
rx_vport_unicast_packets /sec
Before this patch: single flow performance was 6Mpps, and if I started
two flows the collective performance drop to 4Mpps, because we hit the
page allocator lock (further negative scaling occurs).
V2: Adjustments requested by Tariq
- Changed page_pool_create return codes not return NULL, only
ERR_PTR, as this simplifies err handling in drivers.
- Save a branch in mlx5e_page_release
- Correct page_pool size calc for MLX5_WQ_TYPE_LINKED_LIST_STRIDING_RQ
Signed-off-by: Jesper Dangaard Brouer <bro...@redhat.com>
---
I am running perf tests with your series. I sense a drastic degradation
in regular TCP flows, I'm double checking the numbers now...
Well, there's a huge performance degradation indeed, whenever the
regular flows (non-XDP) use the new page pool. Cannot merge before
fixing this.
If I disable the local page-cache, numbers get as low as 100's of Mbps
in TCP stream tests.
