[Edited Message Follows]
Dear Ben & all,
I did some more debug for this issue.
This time I use 1G huge page, and the topology is also vpp(virtio-user) <-->
(Vhost-user)testpmd. Both are based on dpdk1908.
startup.conf
>
>
>
> heapsize 2G
>
>
>
> unix {
>
>
>
> nodaemon
>
>
>
> interactive
>
>
>
> cli-listen /run/vpp/cli.sock
>
>
>
> exec init.conf
>
>
>
> }
>
>
>
> dpdk {
>
>
>
> dev 0000:0a:00.0
>
>
>
> vdev virtio_user0,path=/tmp/sock0,mac=00:11:22:33:44:10
>
>
>
> single-file-segments
>
>
>
> }
>
>
init.conf
>
>
>
> set interface ip address VirtioUser1 172.20.224.1/24
>
>
>
> set interface state VirtioUser1 up
>
>
>
> set interface ip address GigabitEtherneta/0/0 10.255.255.254/24
>
>
>
> set interface state GigabitEtherneta/0/0 up
>
>
>
> ip neighbor VirtioUser1 172.20.224.10 56:48:4F:53:54:00
>
>
The VPP have placed the packets in the share memory queue of virtio/vhost.
There is something wrong when testpmd Vhost receives packets from the share
memory.
Here is the difference log between testpmd(vhost) and VPP.
testpmd(Vhost-user) log when testpmd(virtio-user) connected
>
>
>
> VHOST_CONFIG: new vhost user connection is 31
>
>
>
> VHOST_CONFIG: new device, handle is 0
>
>
>
> VHOST_CONFIG: read message VHOST_USER_SET_OWNER
>
>
>
> VHOST_CONFIG: read message VHOST_USER_GET_FEATURES
>
>
>
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
>
>
>
> VHOST_CONFIG: vring call idx:0 file:32
>
>
>
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_CALL
>
>
>
> VHOST_CONFIG: vring call idx:1 file:33
>
>
>
> VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
>
>
>
> VHOST_CONFIG: negotiated Virtio features: 0x910008000
>
>
>
> VHOST_CONFIG: read message VHOST_USER_SET_MEM_TABLE
>
>
>
> VHOST_CONFIG: guest memory region 0, size: 0x40000000
>
>
>
> guest physical addr: 0x140000000
>
>
>
> guest virtual addr: 0x140000000
>
>
>
> host virtual addr: 0x7fff80000000
>
>
>
> mmap addr : 0x7fff80000000
>
>
>
> mmap size : 0x40000000
>
>
>
> mmap align: 0x40000000
>
>
>
> mmap off : 0x0
>
>
>
> VHOST_CONFIG: guest memory region 1, size: 0x40000000
>
>
>
> guest physical addr: 0x2240000000
>
>
>
> guest virtual addr: 0x2240000000
>
>
>
> host virtual addr: 0x7fff40000000
>
>
>
> mmap addr : 0x7fff40000000
>
>
>
> mmap size : 0x40000000
>
>
>
> mmap align: 0x40000000
>
>
>
> mmap off : 0x0
>
>
>
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
>
>
>
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
>
>
>
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
>
>
>
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
>
>
>
> VHOST_CONFIG: vring kick idx:0 file:36
>
>
testpmd(Vhost-user) log when VPP connected
>
>
>
> Port 0: queue state event
>
>
>
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_NUM
>
>
>
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_BASE
>
>
>
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ADDR
>
>
>
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_KICK
>
>
>
> VHOST_CONFIG: vring kick idx:1 file:37
>
>
>
>
>
>
>
> Port 0: queue state event
>
>
>
> VHOST_CONFIG: virtio is now ready for processing.
>
>
>
>
>
>
>
> Port 0: link state change event
>
>
>
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
>
>
>
> VHOST_CONFIG: set queue enable: 1 to qp idx: 0
>
>
>
> VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
>
>
>
> VHOST_CONFIG: set queue enable: 1 to qp idx: 1
>
>
After started, send packet to GigabitEtherneta/0/0, VPP send packets
successfully, here is the log:
>
>
>
> Packet 1
>
>
>
>
>
>
>
> 00:00:07:954282: dpdk-input
>
>
>
> GigabitEtherneta/0/0 rx queue 0
>
>
>
> buffer 0x1ffb5f8: current data 0, length 98, buffer-pool 1, ref-count 1,
> totlen-nifb 0, trace handle 0x0
>
>
>
> ext-hdr-valid
>
>
>
> l4-cksum-computed l4-cksum-correct
>
>
>
> PKT MBUF: port 0, nb_segs 1, pkt_len 98
>
>
>
> buf_len 2176, data_len 98, ol_flags 0x180, data_off 128, phys_addr
> 0x7fed7e80
>
>
>
> packet_type 0x11 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
>
>
>
> rss 0x0 fdir.hi 0x0 fdir.lo 0x0
>
>
>
> Packet Offload Flags
>
>
>
> PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
>
>
>
> PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
>
>
>
> Packet Types
>
>
>
> RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
>
>
>
> RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers
>
>
>
> IP4: 94:57:a5:53:d5:cf -> 24:6e:96:07:00:d4
>
>
>
> ICMP: 10.255.255.253 -> 172.20.224.10
>
>
>
> tos 0x00, ttl 64, length 84, checksum 0xcb21 dscp CS0 ecn NON_ECN
>
>
>
> fragment id 0xd86b, flags DONT_FRAGMENT
>
>
>
> ICMP echo_request checksum 0x56f0
>
>
>
> 00:00:07:954291: ethernet-input
>
>
>
> frame: flags 0x3, hw-if-index 1, sw-if-index 1
>
>
>
> IP4: 94:57:a5:53:d5:cf -> 24:6e:96:07:00:d4
>
>
>
> 00:00:07:954297: ip4-input-no-checksum
>
>
>
> ICMP: 10.255.255.253 -> 172.20.224.10
>
>
>
> tos 0x00, ttl 64, length 84, checksum 0xcb21 dscp CS0 ecn NON_ECN
>
>
>
> fragment id 0xd86b, flags DONT_FRAGMENT
>
>
>
> ICMP echo_request checksum 0x56f0
>
>
>
> 00:00:07:954303: ip4-lookup
>
>
>
> fib 0 dpo-idx 2 flow hash: 0x00000000
>
>
>
> ICMP: 10.255.255.253 -> 172.20.224.10
>
>
>
> tos 0x00, ttl 64, length 84, checksum 0xcb21 dscp CS0 ecn NON_ECN
>
>
>
> fragment id 0xd86b, flags DONT_FRAGMENT
>
>
>
> ICMP echo_request checksum 0x56f0
>
>
>
> 00:00:07:954309: ip4-rewrite
>
>
>
> tx_sw_if_index 2 dpo-idx 2 : ipv4 via 172.20.224.10 VirtioUser1: mtu:9000
> next:3 56484f5354000011223344100800 flow hash: 0x00000000
>
>
>
> 00000000: 56484f535400001122334410080045000054d86b40003f01cc210afffffdac14
>
>
>
>
> 00000020: e00a080056f0a0b203ab0a6df15e000000003f130300000000001011
>
>
>
> 00:00:07:954311: VirtioUser1-output
>
>
>
> VirtioUser1
>
>
>
> IP4: 00:11:22:33:44:10 -> 56:48:4f:53:54:00
>
>
>
> ICMP: 10.255.255.253 -> 172.20.224.10
>
>
>
> tos 0x00, ttl 63, length 84, checksum 0xcc21 dscp CS0 ecn NON_ECN
>
>
>
> fragment id 0xd86b, flags DONT_FRAGMENT
>
>
>
> ICMP echo_request checksum 0x56f0
>
>
>
> 00:00:07:954317: VirtioUser1-tx
>
>
>
> VirtioUser1 tx queue 0
>
>
>
> buffer 0x1ffb5f8: current data 0, length 98, buffer-pool 1, ref-count 1,
> totlen-nifb 0, trace handle 0x0
>
>
>
> ext-hdr-valid
>
>
>
> l4-cksum-computed l4-cksum-correct l2-hdr-offset 0 l3-hdr-offset 14
>
>
>
> PKT MBUF: port 0, nb_segs 1, pkt_len 98
>
>
>
> buf_len 2176, data_len 98, ol_flags 0x180, data_off 128, phys_addr
> 0x7fed7e80
>
>
>
> packet_type 0x11 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
>
>
>
> rss 0x0 fdir.hi 0x0 fdir.lo 0x0
>
>
>
> Packet Offload Flags
>
>
>
> PKT_RX_IP_CKSUM_GOOD (0x0080) IP cksum of RX pkt. is valid
>
>
>
> PKT_RX_L4_CKSUM_GOOD (0x0100) L4 cksum of RX pkt. is valid
>
>
>
> Packet Types
>
>
>
> RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet
>
>
>
> RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension headers
>
>
>
> IP4: 00:11:22:33:44:10 -> 56:48:4f:53:54:00
>
>
>
> ICMP: 10.255.255.253 -> 172.20.224.10
>
>
>
> tos 0x00, ttl 63, length 84, checksum 0xcc21 dscp CS0 ecn NON_ECN
>
>
>
> fragment id 0xd86b, flags DONT_FRAGMENT
>
>
>
> ICMP echo_request checksum 0x56f0
>
>
In testpmd side, the rte_eth_rx_burst goes wrong. Here is the gdb log:
(gdb) bt
>
>
>
> #0 rte_vhost_va_from_guest_pa (len=0x7ffff5e06060, gpa=70865760180,
> mem=0x16981b2c0) at
> /home/ubuntu/chan/dpdk/x86_64-native-linux-gcc/include/rte_vhost.h:330
>
>
>
> #1 vhost_iova_to_vva (perm=1 '\001', len=0x7ffff5e06060,
> iova=70865760180, vq=0x2269d170c0, dev=0x2269d438c0) at
> /home/ubuntu/chan/dpdk/lib/librte_vhost/vhost.h:674
>
>
>
> #2 map_one_desc (perm=1 '\001', desc_len=110, desc_iova=70865760180,
> vec_idx=0x7ffff5e0384e, buf_vec=0x7ffff5e08710, vq=0x2269d170c0,
> dev=0x2269d438c0) at
> /home/ubuntu/chan/dpdk/lib/librte_vhost/virtio_net.c:471
>
>
>
> #3 fill_vec_buf_split (perm=1 '\001', desc_chain_len=0x7ffff5e038b0,
> desc_chain_head=0x7ffff5e0384a, buf_vec=0x7ffff5e08710,
> vec_idx=0x7ffff5e0384c, avail_idx=0, vq=0x2269d170c0, dev=0x2269d438c0)
>
>
>
> at /home/ubuntu/chan/dpdk/lib/librte_vhost/virtio_net.c:550
>
>
>
> #4 virtio_dev_tx_split (dev=0x2269d438c0, vq=0x2269d170c0,
> mbuf_pool=0x17f99ea40, pkts=0x7ffff5e0a0f0, count=32) at
> /home/ubuntu/chan/dpdk/lib/librte_vhost/virtio_net.c:1728
>
>
>
> #5 0x00005555559302a9 in rte_vhost_dequeue_burst (vid=0, queue_id=1,
> mbuf_pool=0x17f99ea40, pkts=0x7ffff5e0a0f0, count=32) at
> /home/ubuntu/chan/dpdk/lib/librte_vhost/virtio_net.c:2284
>
>
>
> #6 0x000055555664305c in eth_vhost_rx (q=0x16988b380,
> bufs=0x7ffff5e0a0f0, nb_bufs=32) at
> /home/ubuntu/chan/dpdk/drivers/net/vhost/rte_eth_vhost.c:400
>
>
>
> #7 0x00005555556f1888 in rte_eth_rx_burst (port_id=0, queue_id=0,
> rx_pkts=0x7ffff5e0a0f0, nb_pkts=32) at
> /home/ubuntu/chan/dpdk/x86_64-native-linux-gcc/include/rte_ethdev.h:4405
>
>
>
> #8 0x00005555556f1e82 in pkt_burst_io_forward (fs=0x16988c900) at
> /home/ubuntu/chan/dpdk/app/test-pmd/iofwd.c:67
>
>
>
> #9 0x000055555569027b in run_pkt_fwd_on_lcore (fc=0x17fb20c80,
> pkt_fwd=0x5555556f1e2e <pkt_burst_io_forward>) at
> /home/ubuntu/chan/dpdk/app/test-pmd/testpmd.c:2070
>
>
>
> #10 0x0000555555690374 in start_pkt_forward_on_core (fwd_arg=0x17fb20c80)
> at /home/ubuntu/chan/dpdk/app/test-pmd/testpmd.c:2096
>
>
>
> #11 0x0000555555a126bb in eal_thread_loop (arg=0x0) at
> /home/ubuntu/chan/dpdk/lib/librte_eal/linux/eal_thread.c:161
>
>
>
> #12 0x00007ffff72086db in start_thread (arg=0x7ffff5e0d700) at
> pthread_create.c:463
>
>
>
> #13 0x00007ffff6f3188f in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
>
>
>
>
>
> (gdb) p /x mem->regions[0]
>
>
>
> $28 = {guest_phys_addr = 0x140000000, guest_user_addr = 0x140000000,
> host_user_addr = 0x7fff80000000, size = 0x40000000, mmap_addr =
> 0x7fff80000000, mmap_size = 0x40000000, fd = 0x22}
>
>
>
> (gdb) p /x mem->regions[1]
>
>
>
> $29 = {guest_phys_addr = 0x7fde00000000, guest_user_addr = 0x7fde00000000,
> host_user_addr = 0x7fff40000000, size = 0x40000000, mmap_addr =
> 0x7fff40000000, mmap_size = 0x40000000, fd = 0x23}
>
>
>
> (gdb) p mem->nregions
>
>
>
> $30 = 2
>
>
>
> (gdb) p /x gpa
>
>
>
> $31 = 0x107fedafb4
>
>
>
> (gdb) p len
>
>
>
> $32 = (uint64_t *) 0x7ffff5e06060
>
>
>
> (gdb) p *len
>
>
>
> $33 = 110
>
>
It seems that the memory address to be translated gpa is not in the valid
physical memory address regions[0] and regions[1].
BR,
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#16784): https://lists.fd.io/g/vpp-dev/message/16784
Mute This Topic: https://lists.fd.io/mt/74982077/21656
Group Owner: [email protected]
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
