Ravi,
For (1) which works, what dpdk version are you using in the host? Are you using
the same dpdk version as VPP is using? Since you are using VPP latest, I think
it is 18.02. Type "show dpdk version" at the VPP prompt to find out for sure.
Steven
On 5/31/18, 11:44 AM, "Ravi Kerur" <rke...@gmail.com> wrote:
Hi Steven,
i have tested following scenarios and it basically is not clear why
you think DPDK is the problem? Is it possible VPP and DPDK use
different virtio versions?
Following are the scenarios I have tested
(1) testpmd/DPDK vhost-user (running on host) and testpmd/DPDK
virito-user (in a container) -- can send and receive packets
(2) VPP-native vhost-user (running on host) and testpmd/DPDK
virtio-user (in a container) -- VPP crashes and it is in VPP code
(3) VPP-DPDK vhost user (running on host) and testpmd/DPDK virtio-user
(in a container) -- VPP crashes and in DPDK
Thanks.
On Thu, May 31, 2018 at 10:12 AM, Steven Luong (sluong)
<slu...@cisco.com> wrote:
> Ravi,
>
> I've proved my point -- there is a problem in the way that you invoke
testpmd. The shared memory region that it passes to the device is not
accessible from the device. I don't know what the correct options are that you
need to use. This is really a question for dpdk.
>
> As a further exercise, you could remove VPP in the host and instead run
testpmd in device mode using "--vdev net_vhost0,iface=/var/run/vpp/sock1.sock"
option. I bet you testpmd in the host will crash in the same place. I hope you
can find out the answer from dpdk and tell us about it.
>
> Steven
>
> On 5/31/18, 9:31 AM, "vpp-dev@lists.fd.io on behalf of Ravi Kerur"
<vpp-dev@lists.fd.io on behalf of rke...@gmail.com> wrote:
>
> Hi Steven,
>
> Thank you for your help, I removed sock1.sock and sock2.sock,
> restarted vpp, atleast interfaces get created. However, when I start
> dpdk/testpmd inside the container it crashes as well. Below are some
> details. I am using vpp code from latest repo.
>
> (1) On host
> show interface
> Name Idx State Counter
> Count
> VhostEthernet2 3 down
> VhostEthernet3 4 down
> VirtualFunctionEthernet4/10/4 1 down
> VirtualFunctionEthernet4/10/6 2 down
> local0 0 down
> vpp#
> vpp# set interface state VhostEthernet2 up
> vpp# set interface state VhostEthernet3 up
> vpp#
> vpp# set interface l2 bridge VhostEthernet2 1
> vpp# set interface l2 bridge VhostEthernet3 1
> vpp#
>
> (2) Run tespmd inside the container
> docker run -it --privileged -v
> /var/run/vpp/sock1.sock:/var/run/usvhost1 -v
> /var/run/vpp/sock2.sock:/var/run/usvhost2 -v
> /dev/hugepages:/dev/hugepages dpdk-app-testpmd ./bin/testpmd -l 16-19
> -n 4 --log-level=8 -m 64 --no-pci
> --vdev=virtio_user0,path=/var/run/usvhost1,mac=54:00:00:01:01:01
> --vdev=virtio_user1,path=/var/run/usvhost2,mac=54:00:00:01:01:02 --
> -i
> EAL: Detected 28 lcore(s)
> EAL: Detected 2 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: 8192 hugepages of size 2097152 reserved, but no mounted hugetlbfs
> found for that size
> EAL: Probing VFIO support...
> EAL: VFIO support initialized
> EAL: Setting up physically contiguous memory...
> EAL: locking hot plug lock memory...
> EAL: primary init32...
> Interactive-mode selected
> Warning: NUMA should be configured manually by using
> --port-numa-config and --ring-numa-config parameters along with
> --numa.
> testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=171456,
> size=2176, socket=0
> testpmd: preferred mempool ops selected: ring_mp_mc
> testpmd: create a new mbuf pool <mbuf_pool_socket_1>: n=171456,
> size=2176, socket=1
> testpmd: preferred mempool ops selected: ring_mp_mc
> Port 0 is now not stopped
> Port 1 is now not stopped
> Please stop the ports first
> Done
> testpmd>
>
> (3) VPP crashes with the same issue but inside dpdk code
>
> (gdb) cont
> Continuing.
>
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7ffd0d08e700 (LWP 41257)]
> rte_vhost_dequeue_burst (vid=<optimized out>, queue_id=<optimized
> out>, mbuf_pool=0x7fe17fc883c0,
> pkts=pkts@entry=0x7fffb671ebc0, count=count@entry=32)
> at
/var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_vhost/virtio_net.c:1504
> 1504 free_entries = *((volatile uint16_t *)&vq->avail->idx) -
> (gdb) bt
> #0 rte_vhost_dequeue_burst (vid=<optimized out>, queue_id=<optimized
out>,
> mbuf_pool=0x7fe17fc883c0, pkts=pkts@entry=0x7fffb671ebc0,
> count=count@entry=32)
> at
/var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_vhost/virtio_net.c:1504
> #1 0x00007fffb4718e6f in eth_vhost_rx (q=0x7fe17fbbdd80,
bufs=0x7fffb671ebc0,
> nb_bufs=<optimized out>)
> at
/var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/drivers/net/vhost/rte_eth_vhost.c:410
> #2 0x00007fffb441cb7c in rte_eth_rx_burst (nb_pkts=256,
> rx_pkts=0x7fffb671ebc0, queue_id=0,
> port_id=3) at
>
/var/venom/vpp/build-root/install-vpp-native/dpdk/include/dpdk/rte_ethdev.h:3635
> #3 dpdk_device_input (queue_id=0, thread_index=<optimized out>,
> node=0x7fffb732c700,
> xd=0x7fffb7337240, dm=<optimized out>, vm=0x7fffb6703340)
> at /var/venom/vpp/build-data/../src/plugins/dpdk/device/node.c:477
> #4 dpdk_input_node_fn_avx2 (vm=<optimized out>, node=<optimized out>,
> f=<optimized out>)
> at /var/venom/vpp/build-data/../src/plugins/dpdk/device/node.c:658
> #5 0x00007ffff7954d35 in dispatch_node
> (last_time_stamp=12531752723928016, frame=0x0,
> dispatch_state=VLIB_NODE_STATE_POLLING, type=VLIB_NODE_TYPE_INPUT,
> node=0x7fffb732c700,
> vm=0x7fffb6703340) at
/var/venom/vpp/build-data/../src/vlib/main.c:988
> #6 vlib_main_or_worker_loop (is_main=0, vm=0x7fffb6703340)
> at /var/venom/vpp/build-data/../src/vlib/main.c:1507
> #7 vlib_worker_loop (vm=0x7fffb6703340) at
> /var/venom/vpp/build-data/../src/vlib/main.c:1641
> #8 0x00007ffff6ad25d8 in clib_calljmp ()
> at /var/venom/vpp/build-data/../src/vppinfra/longjmp.S:110
> #9 0x00007ffd0d08ddb0 in ?? ()
> #10 0x00007fffb4436edd in eal_thread_loop (arg=<optimized out>)
> at
/var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_eal/linuxapp/eal/eal_thread.c:153
> #11 0x0000000000000000 in ?? ()
> (gdb) frame 0
> #0 rte_vhost_dequeue_burst (vid=<optimized out>, queue_id=<optimized
out>,
> mbuf_pool=0x7fe17fc883c0, pkts=pkts@entry=0x7fffb671ebc0,
> count=count@entry=32)
> at
/var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_vhost/virtio_net.c:1504
> 1504 free_entries = *((volatile uint16_t *)&vq->avail->idx) -
> (gdb) p vq
> $1 = (struct vhost_virtqueue *) 0x7fc3ffc84b00
> (gdb) p vq->avail
> $2 = (struct vring_avail *) 0x7ffbfff98000
> (gdb) p *$2
> Cannot access memory at address 0x7ffbfff98000
> (gdb)
>
>
> Thanks.
>
> On Thu, May 31, 2018 at 12:09 AM, Steven Luong (sluong)
> <slu...@cisco.com> wrote:
> > Sorry, I was expecting to see two VhostEthernet interfaces like
this. Those VirtualFunctionEthernet are your physical interfaces.
> >
> > sh int
> > Name Idx State Counter
Count
> > VhostEthernet0 1 up
> > VhostEthernet1 2 up
> > local0 0 down
> > DBGvpp#
> >
> > You have to first manually remove /var/run/vpp/sock1.sock and
/var/run/vpp/sock2.sock before you start vpp on the host. dpdk does not like it
if they already existed. If you successfully create VhostEthernet interface,
try to send some traffic through it to see if it crashes or not.
> >
> > Steven
> >
> > On 5/30/18, 9:17 PM, "vpp-dev@lists.fd.io on behalf of Steven Luong
(sluong)" <vpp-dev@lists.fd.io on behalf of slu...@cisco.com> wrote:
> >
> > Ravi,
> >
> > I don't think you can declare (2) works fine yet. Please bring
up the dpdk vhost-user interfaces and try to send some traffic between them to
exercise the shared memory region from dpdk virtio-user which may be
"questionable".
> >
> > VirtualFunctionEthernet4/10/4 1 down
> > VirtualFunctionEthernet4/10/6 2 down
> >
> > Steven
> >
> > On 5/30/18, 4:41 PM, "Ravi Kerur" <rke...@gmail.com> wrote:
> >
> > Hi Steve,
> >
> > Thank you for your inputs, I added feature-mask to see if
it helps in
> > setting up queues correctly, it didn't so I will remove it.
I have
> > tried following combination
> >
> > (1) VPP->vhost-user (on host) and DPDK/testpmd->virtio-user
(in a
> > container) -- VPP crashes
> > (2) DPDK/testpmd->vhost-user (on host) and
DPDK/testpmd->virtio-user
> > (in a container) -- works fine
> >
> > To use DPDK vhost-user inside VPP, I defined configuration
in
> > startup.conf as mentioned by you and it looks as follows
> >
> > unix {
> > nodaemon
> > log /var/log/vpp/vpp.log
> > full-coredump
> > cli-listen /run/vpp/cli.sock
> > gid vpp
> > }
> >
> > api-segment {
> > gid vpp
> > }
> >
> > cpu {
> > main-core 1
> > corelist-workers 6-9
> > }
> >
> > dpdk {
> > dev 0000:04:10.4
> > dev 0000:04:10.6
> > uio-driver vfio-pci
> > vdev net_vhost0,iface=/var/run/vpp/sock1.sock
> > vdev net_vhost1,iface=/var/run/vpp/sock2.sock
> > huge-dir /dev/hugepages_1GB
> > socket-mem 2048,2048
> > }
> >
> > From VPP logs
> > dpdk: EAL init args: -c 3c2 -n 4 --vdev
> > net_vhost0,iface=/var/run/vpp/sock1.sock --vdev
> > net_vhost1,iface=/var/run/vpp/sock2.sock --huge-dir
/dev/hugepages_1GB
> > -w 0000:04:10.4 -w 0000:04:10.6 --master-lcore 1
--socket-mem
> > 2048,2048
> >
> > However, VPP doesn't create interface at all
> >
> > vpp# show interface
> > Name Idx State
Counter
> > Count
> > VirtualFunctionEthernet4/10/4 1 down
> > VirtualFunctionEthernet4/10/6 2 down
> > local0 0 down
> >
> > since it is a static mapping I am assuming it should be
created, correct?
> >
> > Thanks.
> >
> > On Wed, May 30, 2018 at 3:43 PM, Steven Luong (sluong)
<slu...@cisco.com> wrote:
> > > Ravi,
> > >
> > > First and foremost, get rid of the feature-mask option. I
don't know what 0x40400000 does for you. If that does not help, try testing it
with dpdk based vhost-user instead of VPP native vhost-user to make sure that
they can work well with each other first. To use dpdk vhost-user, add a vdev
command in the startup.conf for each vhost-user device that you have.
> > >
> > > dpdk { vdev net_vhost0,iface=/var/run/vpp/sock1.sock }
> > >
> > > dpdk based vhost-user interface is named VhostEthernet0,
VhostEthernet1, etc. Make sure you use the right interface name to set the
state to up.
> > >
> > > If dpdk based vhost-user does not work with testpmd
either, it looks like some problem with the way that you invoke testpmd.
> > >
> > > If dpdk based vhost-user works well with the same testpmd
device driver and not vpp native vhost-user, I can set up something similar to
yours to look into it.
> > >
> > > The device driver, testpmd, is supposed to pass the
shared memory region to VPP for TX/RX queues. It looks like VPP vhost-user
might have run into a bump there with using the shared memory (txvq->avail).
> > >
> > > Steven
> > >
> > > PS. vhost-user is not an optimum interface for
containers. You may want to look into using memif if you don't already know
about it.
> > >
> > >
> > > On 5/30/18, 2:06 PM, "Ravi Kerur" <rke...@gmail.com>
wrote:
> > >
> > > I am not sure what is wrong with the setup or a bug
in vpp, vpp
> > > crashes with vhost<-->virtio communication.
> > >
> > > (1) Vhost-interfaces are created and attached to
bridge-domain as follows
> > >
> > > create vhost socket /var/run/vpp/sock1.sock server
feature-mask 0x40400000
> > > create vhost socket /var/run/vpp/sock2.sock server
feature-mask 0x40400000
> > > set interface state VirtualEthernet0/0/0 up
> > > set interface state VirtualEthernet0/0/1 up
> > >
> > > set interface l2 bridge VirtualEthernet0/0/0 1
> > > set interface l2 bridge VirtualEthernet0/0/1 1
> > >
> > >
> > > (2) DPDK/testpmd is started in a container to talk to
vpp/vhost-user
> > > interface as follows
> > >
> > > docker run -it --privileged -v
> > > /var/run/vpp/sock1.sock:/var/run/usvhost1 -v
> > > /var/run/vpp/sock2.sock:/var/run/usvhost2 -v
> > > /dev/hugepages:/dev/hugepages dpdk-app-testpmd
./bin/testpmd -c 0x3 -n
> > > 4 --log-level=9 -m 64 --no-pci --single-file-segments
> > >
--vdev=virtio_user0,path=/var/run/usvhost1,mac=54:00:00:01:01:01
> > >
--vdev=virtio_user1,path=/var/run/usvhost2,mac=54:00:00:01:01:02 --
> > > -i
> > >
> > > (3) show vhost-user VirtualEthernet0/0/1
> > > Virtio vhost-user interfaces
> > > Global:
> > > coalesce frames 32 time 1e-3
> > > number of rx virtqueues in interrupt mode: 0
> > > Interface: VirtualEthernet0/0/1 (ifindex 4)
> > > virtio_net_hdr_sz 10
> > > features mask (0x40400000):
> > > features (0x0):
> > > protocol features (0x0)
> > >
> > > socket filename /var/run/vpp/sock2.sock type server
errno "Success"
> > >
> > > rx placement:
> > > tx placement: spin-lock
> > > thread 0 on vring 0
> > > thread 1 on vring 0
> > > thread 2 on vring 0
> > > thread 3 on vring 0
> > > thread 4 on vring 0
> > >
> > > Memory regions (total 1)
> > > region fd guest_phys_addr memory_size
userspace_addr
> > > mmap_offset mmap_addr
> > > ====== ===== ================== ==================
==================
> > > ================== ==================
> > > 0 55 0x00007ff7c0000000 0x0000000040000000
0x00007ff7c0000000
> > > 0x0000000000000000 0x00007ffbc0000000
> > >
> > > vpp# show vhost-user VirtualEthernet0/0/0
> > > Virtio vhost-user interfaces
> > > Global:
> > > coalesce frames 32 time 1e-3
> > > number of rx virtqueues in interrupt mode: 0
> > > Interface: VirtualEthernet0/0/0 (ifindex 3)
> > > virtio_net_hdr_sz 10
> > > features mask (0x40400000):
> > > features (0x0):
> > > protocol features (0x0)
> > >
> > > socket filename /var/run/vpp/sock1.sock type server
errno "Success"
> > >
> > > rx placement:
> > > tx placement: spin-lock
> > > thread 0 on vring 0
> > > thread 1 on vring 0
> > > thread 2 on vring 0
> > > thread 3 on vring 0
> > > thread 4 on vring 0
> > >
> > > Memory regions (total 1)
> > > region fd guest_phys_addr memory_size
userspace_addr
> > > mmap_offset mmap_addr
> > > ====== ===== ================== ==================
==================
> > > ================== ==================
> > > 0 51 0x00007ff7c0000000 0x0000000040000000
0x00007ff7c0000000
> > > 0x0000000000000000 0x00007ffc00000000
> > >
> > > (4) vpp stack trace
> > > Program received signal SIGSEGV, Segmentation fault.
> > > [Switching to Thread 0x7ffd0e090700 (LWP 46570)]
> > > 0x00007ffff7414642 in vhost_user_if_input
> > > (mode=VNET_HW_INTERFACE_RX_MODE_POLLING,
> > > node=0x7fffb76bab00, qid=<optimized out>,
vui=0x7fffb6739700,
> > > vum=0x7ffff78f4480 <vhost_user_main>,
vm=0x7fffb672a9c0)
> > > at
/var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1596
> > > 1596 if (PREDICT_FALSE (txvq->avail->flags &
0xFFFE))
> > > (gdb) bt
> > > #0 0x00007ffff7414642 in vhost_user_if_input
> > > (mode=VNET_HW_INTERFACE_RX_MODE_POLLING,
> > > node=0x7fffb76bab00, qid=<optimized out>,
vui=0x7fffb6739700,
> > > vum=0x7ffff78f4480 <vhost_user_main>,
vm=0x7fffb672a9c0)
> > > at
/var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1596
> > > #1 vhost_user_input (f=<optimized out>,
node=<optimized out>,
> > > vm=<optimized out>)
> > > at
/var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1947
> > > #2 vhost_user_input_avx2 (vm=<optimized out>,
node=<optimized out>,
> > > frame=<optimized out>)
> > > at
/var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1972
> > > #3 0x00007ffff7954d35 in dispatch_node
> > > (last_time_stamp=12391212490024174, frame=0x0,
> > > dispatch_state=VLIB_NODE_STATE_POLLING,
type=VLIB_NODE_TYPE_INPUT,
> > > node=0x7fffb76bab00,
> > > vm=0x7fffb672a9c0) at
/var/venom/vpp/build-data/../src/vlib/main.c:988
> > > #4 vlib_main_or_worker_loop (is_main=0,
vm=0x7fffb672a9c0)
> > > at
/var/venom/vpp/build-data/../src/vlib/main.c:1507
> > > #5 vlib_worker_loop (vm=0x7fffb672a9c0) at
> > > /var/venom/vpp/build-data/../src/vlib/main.c:1641
> > > #6 0x00007ffff6ad25d8 in clib_calljmp ()
> > > at
/var/venom/vpp/build-data/../src/vppinfra/longjmp.S:110
> > > #7 0x00007ffd0e08fdb0 in ?? ()
> > > #8 0x00007fffb4436edd in eal_thread_loop
(arg=<optimized out>)
> > > at
/var/venom/vpp/build-root/build-vpp-native/dpdk/dpdk-stable-18.02.1/lib/librte_eal/linuxapp/eal/eal_thread.c:153
> > > #9 0x0000000000000000 in ?? ()
> > > (gdb) frame 0
> > > #0 0x00007ffff7414642 in vhost_user_if_input
> > > (mode=VNET_HW_INTERFACE_RX_MODE_POLLING,
> > > node=0x7fffb76bab00, qid=<optimized out>,
vui=0x7fffb6739700,
> > > vum=0x7ffff78f4480 <vhost_user_main>,
vm=0x7fffb672a9c0)
> > > at
/var/venom/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1596
> > > 1596 if (PREDICT_FALSE (txvq->avail->flags &
0xFFFE))
> > > (gdb) p txvq
> > > $1 = (vhost_user_vring_t *) 0x7fffb6739ac0
> > > (gdb) p *txvq
> > > $2 = {cacheline0 = 0x7fffb6739ac0 "?", qsz_mask =
255, last_avail_idx
> > > = 0, last_used_idx = 0,
> > > n_since_last_int = 0, desc = 0x7ffbfff97000, avail
= 0x7ffbfff98000,
> > > used = 0x7ffbfff99000,
> > > int_deadline = 0, started = 1 '\001', enabled = 0
'\000', log_used = 0 '\000',
> > > cacheline1 = 0x7fffb6739b00 "????\n", errfd = -1,
callfd_idx = 10,
> > > kickfd_idx = 14,
> > > log_guest_addr = 0, mode = 1}
> > > (gdb) p *(txvq->avail)
> > > Cannot access memory at address 0x7ffbfff98000
> > > (gdb)
> > >
> > > On Tue, May 29, 2018 at 10:47 AM, Ravi Kerur
<rke...@gmail.com> wrote:
> > > > Steve,
> > > >
> > > > Thanks for inputs on debugs and gdb. I am using gdb
on my development
> > > > system to debug the issue. I would like to have
reliable core
> > > > generation on the system on which I don't have
access to install gdb.
> > > > I installed corekeeper and it still doesn't
generate core. I am
> > > > running vpp inside a VM (VirtualBox/vagrant), not
sure if I need to
> > > > set something inside vagrant config file.
> > > >
> > > > dpkg -l corekeeper
> > > > Desired=Unknown/Install/Remove/Purge/Hold
> > > > |
Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> > > > |/ Err?=(none)/Reinst-required (Status,Err:
uppercase=bad)
> > > > ||/ Name Version
Architecture Description
> > > >
+++-====================-===============-===============-==============================================
> > > > ii corekeeper 1.6 amd64
enable core
> > > > files and report crashes to the system
> > > >
> > > > Thanks.
> > > >
> > > > On Tue, May 29, 2018 at 9:38 AM, Steven Luong
(sluong) <slu...@cisco.com> wrote:
> > > >> Ravi,
> > > >>
> > > >> I install corekeeper and the core file is kept in
/var/crash. But why not use gdb to attach to the VPP process?
> > > >> To turn on VPP vhost-user debug, type "debug
vhost-user on" at the VPP prompt.
> > > >>
> > > >> Steven
> > > >>
> > > >> On 5/29/18, 9:10 AM, "vpp-dev@lists.fd.io on
behalf of Ravi Kerur" <vpp-dev@lists.fd.io on behalf of rke...@gmail.com> wrote:
> > > >>
> > > >> Hi Marco,
> > > >>
> > > >>
> > > >> On Tue, May 29, 2018 at 6:30 AM, Marco Varlese
<mvarl...@suse.de> wrote:
> > > >> > Ravi,
> > > >> >
> > > >> > On Sun, 2018-05-27 at 12:20 -0700, Ravi
Kerur wrote:
> > > >> >> Hello,
> > > >> >>
> > > >> >> I have a VM(16.04.4 Ubuntu x86_64) with 2
cores and 4G RAM. I have
> > > >> >> installed VPP successfully on it. Later I
have created vhost-user
> > > >> >> interfaces via
> > > >> >>
> > > >> >> create vhost socket /var/run/vpp/sock1.sock
server
> > > >> >> create vhost socket /var/run/vpp/sock2.sock
server
> > > >> >> set interface state VirtualEthernet0/0/0 up
> > > >> >> set interface state VirtualEthernet0/0/1 up
> > > >> >>
> > > >> >> set interface l2 bridge
VirtualEthernet0/0/0 1
> > > >> >> set interface l2 bridge
VirtualEthernet0/0/1 1
> > > >> >>
> > > >> >> I then run 'DPDK/testpmd' inside a
container which will use
> > > >> >> virtio-user interfaces using the following
command
> > > >> >>
> > > >> >> docker run -it --privileged -v
> > > >> >> /var/run/vpp/sock1.sock:/var/run/usvhost1 -v
> > > >> >> /var/run/vpp/sock2.sock:/var/run/usvhost2 -v
> > > >> >> /dev/hugepages:/dev/hugepages
dpdk-app-testpmd ./bin/testpmd -c 0x3 -n
> > > >> >> 4 --log-level=9 -m 64 --no-pci
--single-file-segments
> > > >> >>
--vdev=virtio_user0,path=/var/run/usvhost1,mac=54:01:00:01:01:01
> > > >> >>
--vdev=virtio_user1,path=/var/run/usvhost2,mac=54:01:00:01:01:02 --
> > > >> >> -i
> > > >> >>
> > > >> >> VPP Vnet crashes with following message
> > > >> >>
> > > >> >> May 27 11:44:00 localhost vnet[6818]:
received signal SIGSEGV, PC
> > > >> >> 0x7fcca4620187, faulting address
0x7fcb317ac000
> > > >> >>
> > > >> >> Questions:
> > > >> >> I have 'ulimit -c unlimited' and
/etc/vpp/startup.conf has
> > > >> >> unix {
> > > >> >> nodaemon
> > > >> >> log /var/log/vpp/vpp.log
> > > >> >> full-coredump
> > > >> >> cli-listen /run/vpp/cli.sock
> > > >> >> gid vpp
> > > >> >> }
> > > >> >>
> > > >> >> But I couldn't locate corefile?
> > > >> > The location of the coredump file depends on
your system configuration.
> > > >> >
> > > >> > Please, check "cat
/proc/sys/kernel/core_pattern"
> > > >> >
> > > >> > If you have systemd-coredump in the output
of the above command, then likely the
> > > >> > location of the coredump files is
"/var/lib/systemd/coredump/"
> > > >> >
> > > >> > You can also change the location of where
your system places the coredump files:
> > > >> > echo '/PATH_TO_YOU_LOCATION/core_%e.%p' |
sudo tee /proc/sys/kernel/core_pattern
> > > >> >
> > > >> > See if that helps...
> > > >> >
> > > >>
> > > >> Initially '/proc/sys/kernel/core_pattern' was
set to 'core'. I changed
> > > >> it to 'systemd-coredump'. Still no core
generated. VPP crashes
> > > >>
> > > >> May 29 08:54:34 localhost vnet[4107]: received
signal SIGSEGV, PC
> > > >> 0x7f0167751187, faulting address 0x7efff43ac000
> > > >> May 29 08:54:34 localhost systemd[1]:
vpp.service: Main process
> > > >> exited, code=killed, status=6/ABRT
> > > >> May 29 08:54:34 localhost systemd[1]:
vpp.service: Unit entered failed state.
> > > >> May 29 08:54:34 localhost systemd[1]:
vpp.service: Failed with result 'signal'.
> > > >>
> > > >>
> > > >> cat /proc/sys/kernel/core_pattern
> > > >> systemd-coredump
> > > >>
> > > >>
> > > >> ulimit -a
> > > >> core file size (blocks, -c) unlimited
> > > >> data seg size (kbytes, -d) unlimited
> > > >> scheduling priority (-e) 0
> > > >> file size (blocks, -f) unlimited
> > > >> pending signals (-i) 15657
> > > >> max locked memory (kbytes, -l) 64
> > > >> max memory size (kbytes, -m) unlimited
> > > >> open files (-n) 1024
> > > >> pipe size (512 bytes, -p) 8
> > > >> POSIX message queues (bytes, -q) 819200
> > > >> real-time priority (-r) 0
> > > >> stack size (kbytes, -s) 8192
> > > >> cpu time (seconds, -t) unlimited
> > > >> max user processes (-u) 15657
> > > >> virtual memory (kbytes, -v) unlimited
> > > >> file locks (-x) unlimited
> > > >>
> > > >> cd /var/lib/systemd/coredump/
> > > >> root@localhost:/var/lib/systemd/coredump# ls
> > > >> root@localhost:/var/lib/systemd/coredump#
> > > >>
> > > >> >>
> > > >> >> (2) How to enable debugs? I have used 'make
build' but no additional
> > > >> >> logs other than those shown below
> > > >> >>
> > > >> >>
> > > >> >> VPP logs from /var/log/syslog is shown below
> > > >> >> cat /var/log/syslog
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
vlib_plugin_early_init:361:
> > > >> >> plugin path
/usr/lib/vpp_plugins:/usr/lib64/vpp_plugins
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: abf_plugin.so (ACL based Forwarding)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: acl_plugin.so (Access Control Lists)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: avf_plugin.so (Intel Adaptive
Virtual Function (AVF) Device
> > > >> >> Plugin)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:191: Loaded
> > > >> >> plugin: cdp_plugin.so
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: dpdk_plugin.so (Data Plane
Development Kit (DPDK))
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: flowprobe_plugin.so (Flow per
Packet)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: gbp_plugin.so (Group Based Policy)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: gtpu_plugin.so (GTPv1-U)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: igmp_plugin.so (IGMP messaging)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: ila_plugin.so (Identifier-locator
addressing for IPv6)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: ioam_plugin.so (Inbound OAM)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:117: Plugin
> > > >> >> disabled (default): ixge_plugin.so
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: l2e_plugin.so (L2 Emulation)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: lacp_plugin.so (Link Aggregation
Control Protocol)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: lb_plugin.so (Load Balancer)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: memif_plugin.so (Packet Memory
Interface (experimetal))
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: nat_plugin.so (Network Address
Translation)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: pppoe_plugin.so (PPPoE)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: srv6ad_plugin.so (Dynamic SRv6
proxy)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: srv6am_plugin.so (Masquerading SRv6
proxy)
> > > >> >> May 27 11:40:28 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: srv6as_plugin.so (Static SRv6 proxy)
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: stn_plugin.so (VPP Steals the NIC
for Container integration)
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: tlsmbedtls_plugin.so (mbedtls based
TLS Engine)
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
load_one_plugin:189: Loaded
> > > >> >> plugin: tlsopenssl_plugin.so (openssl based
TLS Engine)
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
dpdk_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: dpdk_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
lb_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
flowprobe_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
stn_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
nat_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
udp_ping_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
pppoe_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
lacp_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: lb_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
acl_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
ioam_export_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
ioam_trace_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
> > > >> >> vxlan_gpe_ioam_export_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
gtpu_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
cdp_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
ioam_vxlan_gpe_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
memif_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]:
> > > >> >> load_one_vat_plugin:67: Loaded plugin:
ioam_pot_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: flowprobe_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: stn_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: nat_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: udp_ping_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: pppoe_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: lacp_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: acl_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: ioam_export_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: ioam_trace_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin:
vxlan_gpe_ioam_export_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: gtpu_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: cdp_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: ioam_vxlan_gpe_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: memif_test_plugin.so
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: load_one_vat_plugin:67:
> > > >> >> Loaded plugin: ioam_pot_test_plugin.so
> > > >> >> May 27 11:40:29 localhost vpp[6818]:
/usr/bin/vpp[6818]: dpdk: EAL
> > > >> >> init args: -c 1 -n 4 --no-pci --huge-dir
/dev/hugepages --master-lcore
> > > >> >> 0 --socket-mem 256,0
> > > >> >> May 27 11:40:29 localhost
/usr/bin/vpp[6818]: dpdk: EAL init args: -c
> > > >> >> 1 -n 4 --no-pci --huge-dir /dev/hugepages
--master-lcore 0
> > > >> >> --socket-mem 256,0
> > > >> >> May 27 11:40:29 localhost vnet[6818]:
dpdk_ipsec_process:1019: not
> > > >> >> enough DPDK crypto resources, default to
OpenSSL
> > > >> >> May 27 11:43:19 localhost vnet[6818]: show
vhost-user: unknown input `detail
> > > >> >> May 27 11:44:00 localhost vnet[6818]:
received signal SIGSEGV, PC
> > > >> >> 0x7fcca4620187, faulting address
0x7fcb317ac000
> > > >> >> May 27 11:44:00 localhost systemd[1]:
vpp.service: Main process
> > > >> >> exited, code=killed, status=6/ABRT
> > > >> >> May 27 11:44:00 localhost systemd[1]:
vpp.service: Unit entered failed state.
> > > >> >> May 27 11:44:00 localhost systemd[1]:
vpp.service: Failed with result
> > > >> >> 'signal'.
> > > >> >> May 27 11:44:00 localhost systemd[1]:
vpp.service: Service hold-off
> > > >> >> time over, scheduling restart
> > > >> >>
> > > >>
> > > >> Thanks,
> > > >> Ravi
> > > >>
> > > >> >>
> > > >> >>
> > > >> >> Thanks.
> > > >> > Cheers,
> > > >> > Marco
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> > --
> > > >> > Marco V
> > > >> >
> > > >> > SUSE LINUX GmbH | GF: Felix Imendörffer,
Jane Smithard, Graham Norton
> > > >> > HRB 21284 (AG Nürnberg) Maxfeldstr. 5,
D-90409, Nürnberg
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
> >
> >
> >
> >
> >
>
>
>
>
>
-=-=-=-=-=-=-=-=-=-=-=-
Links:
You receive all messages sent to this group.
View/Reply Online (#9483): https://lists.fd.io/g/vpp-dev/message/9483
View All Messages In Topic (15): https://lists.fd.io/g/vpp-dev/topic/20346431
Mute This Topic: https://lists.fd.io/mt/20346431/21656
New Topic: https://lists.fd.io/g/vpp-dev/post
Change Your Subscription: https://lists.fd.io/g/vpp-dev/editsub/21656
Group Home: https://lists.fd.io/g/vpp-dev
Contact Group Owner: vpp-dev+ow...@lists.fd.io
Terms of Service: https://lists.fd.io/static/tos
Unsubscribe: https://lists.fd.io/g/vpp-dev/unsub
Email sent to: arch...@mail-archive.com
-=-=-=-=-=-=-=-=-=-=-=-
