Hi Sergio/All,
Upon further investigation found that, on Guest VM if I remove ?intel_iommu=on? boot line parameters and run app/test and then crypto_qat_autotest tool, it shows all the test cases are passed. Thereafter when I run ipsec-segw application, I got the below error in dmesg on host. Is it an issue with crypto device configuration and queue pair setup? Please suggest. [root at stag48 ~(keystone_admin)]# dmesg -c [ 1264.266965] dmar: DRHD: handling fault status reg 2 [ 1264.266972] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000f000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.266978] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #5 Error Uncorrectable Error: Yes Control Store Error: No Control Store information: Address 0x2A4C - syndrome 0x0 Register ECC Error No - Parity Error: Yes Register information: address 0x2 - type Transfer [ 1264.266980] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000e000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.266986] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000e000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.266990] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000f000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.266993] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000e000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.266997] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #7 Error Uncorrectable Error: Yes Control Store Error: No Control Store information: Address 0x3FFD - syndrome 0x0 Register ECC Error No - Parity Error: Yes Register information: address 0x22 - type Transfer [ 1264.266999] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000e000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.267003] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000e000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.267007] dmar: DMAR:[DMA Read] Request device [88:04.7] fault addr ff53f000f000 DMAR:[fault reason 06] PTE Read access is not set [ 1264.267014] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #8 Error Uncorrectable Error: Yes Control Store Error: No Control Store information: Address 0x2102 - syndrome 0x0 Register ECC Error No - Parity Error: Yes Register information: address 0x2 - type Transfer [ 1264.267020] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #9 Error Uncorrectable Error: Yes Control Store Error: No Control Store information: Address 0x0 - syndrome 0x0 Register ECC Error No - Parity Error: Yes Register information: address 0x22 - type Transfer [ 1264.267026] icp_qa_al err: adf_dh895xcc_adf_isr_LogAE_Int: AE #b Error Uncorrectable Error: Yes Control Store Error: No Control Store information: Address 0x3FFF - syndrome 0x0 Register ECC Error No - Parity Error: Yes Register information: address 0x2 - type Transfer [ 1264.267032] icp_qa_al err: adf_dh895xcc_adf_isr_handleUncoInterrupt: Uncorrectable Push/Pull Misc Error memory status: No errors occurred - Transaction Id 0x0 - Error type reserved Bus Operation Type Push - Id 0x3859C020 [ 1264.267038] Reset needed for device: icp_dev1 [ 1264.267039] Auto Reset disabled. Please reset device1 [root at stag48 ~(keystone_admin)]# Regards, Chinmaya On Thu, Jun 23, 2016 at 5:18 PM, Chinmaya Dwibedy <ckdwibedy at gmail.com> wrote: > Hi All, > > On host, I installed dpdk-2.2.0 and then run 'cryptodev_qat_autotest?. I > found this test case to be failed in dequeue burst API ().Upon looking into > the below system logs (#dmesg), it seems, when this function is called, the > QAT > > hardware fails to write the crypto packet to the DMA buffer. When I passed > intel_iommu=off command in boot line, it works. I think, when we have no > IOMMU, "physical" address space is accessed directly by hardware, so code > works. > > The issue has been observed if the system runs with the kernel parameter > ?intel_iommu=on?. > > > > [root at stag48 ~]# dmesg > > [ 794.746134] dmar: DRHD: handling fault status reg 2 > > [ 794.746141] dmar: DMAR:[DMA Read] Request device [83:01.0] fault addr > 3398c000 > > DMAR:[fault reason 02] Present bit in context entry is clear > > [ 794.746158] icp_qa_al err: adf_dh895xcc_adf_isr_handleUncoInterrupt: > Uncorrectable Push/Pull Misc Error > > memory status: No errors occurred - Transaction Id 0x0 - Error type > reserved > > Bus Operation Type Push - Id 0x80000 > > [ 794.746167] Reset needed for device: icp_dev0 > > [ 794.746167] Auto Reset disabled. Please reset device0 > > [root at stag48 ~]# > > > > *Note: *The QAT VF (*83:01.0*) is bound to the DPDK UIO driver. > > > > 83:01.0 Co-processor: Intel Corporation DH895XCC Series QAT Virtual > Function > > Subsystem: Intel Corporation Device 0000 > > Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- > ParErr- Stepping- SERR- FastB2B- DisINTx- > > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > <TAbort- <MAbort- >SERR- <PERR- INTx- > > Latency: 0 > > Region 0: [virtual] Memory at c8280000 (64-bit, non-prefetchable) > [size=4K] > > Region 2: [virtual] Memory at c82a0000 (64-bit, non-prefetchable) > [size=4K] > > Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit+ > > Address: 0000000000000000 Data: 0000 > > Masking: 00000000 Pending: 00000000 > > Kernel driver in use: igb_uio > > > > > > Now testing the same on Guest and finding the same issue (i.e., failing on > dequeue burst API ()). Upon looking into error message (in dmesg) on > host/hypervisor it says i.e., [28475.040313] > > vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state. The PCI > device 88:04.6 is the QAT VF, being assigned to Guest. I am trying to figure > out what causes ?buffer not found? issue . > > Please suggest if anyone has an idea on it. > > > > Here goes the output of dmesg > > > > [27086.464014] vfio-pci 0000:88:04.6: enabling device (0000 -> 0002) > > [27086.464040] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state > > [27086.564610] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state > > [27086.574052] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state > > [27089.786623] kvm: zapping shadow pages for mmio generation wraparound > > [27106.320042] vfio-pci 0000:88:04.6: irq 136 for MSI/MSI-X > > [28474.784373] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state > > [28475.040313] vfio-pci 0000:88:04.6: buffer not found in pci_save_pcie_state > > [28493.271950] vfio-pci 0000:88:04.6: irq 136 for MSI/MSI-X > > > > > > Regards, > > Chinmaya > > > > > > > > On Wed, Jun 22, 2016 at 1:39 PM, Chinmaya Dwibedy <ckdwibedy at gmail.com> > wrote: > >> Hi Sergio, >> >> >> Thank your for clarification. I am using dpdk-2.2.0. Here goes our >> configuration on VM. Can you please have a look into the same and let me >> know if anything is wrong. Will try to run the autotest, ipsec-segw on host >> as per your suggestion. >> >> >> [root at vpn-s ~]# uname -r >> >> 3.12.9-301.fc20.x86_64 >> >> [root at vpn-s ~]# >> >> >> [root at vpn-s ~]# cat /etc/redhat-release >> >> Fedora release 20 (Heisenbug) >> >> [root at vpn-s ~]# >> >> >> >> >> [root at vpn-s ~]# cat /proc/cmdline >> >> BOOT_IMAGE=/boot/vmlinuz-3.12.9-301.fc20.x86_64 >> root=UUID=ab47cbc9-68ee-403c-96a5-184e68238e65 ro console=ttyS0,115200n8 >> iommu=pt intel_iommu=on >> >> [root at vpn-s ~]# >> >> >> [root at vpn-s QAT]# grep -i "iommu.*enabled" /var/log/messages >> >> Jun 22 07:31:17 vpn-s kernel: [ 0.000000] Intel-IOMMU: enabled >> >> >> [root at vpn-s QAT]# >> >> [root at vpn-s dpdk-2.2.0]# lspci -nn | grep 0443 >> >> 00:06.0 Co-processor [0b40]: Intel Corporation Device [8086:0443] >> >> [root at vpn-s dpdk-2.2.0]# >> >> >> Installed Intel qat driver installation on VM (downloaded from >> https://01.org/packet-processing/intel%C2%AE-quickassist-technology-drivers-and-patches) >> and followed the >> >> https://01.org/sites/default/files/page/330750-005_qat_gsg.pdf guide. >> Note, used the following installation options i.e., ./installer.sh install >> QAT1.6 guest. After this , it detects QAT VF. >> >> >> [root at vpn-s QAT]# service qat_service status >> >> There is 1 acceleration device(s) in the system: >> >> icp_dev0 - type=dh895xcc, inst_id=0, node_id=0, bdf=00:06:0, #accel=1, >> #engines=1, state=up >> >> [root at vpn-s QAT]# >> >> >> [root at vpn-s QAT]# lsmod | grep qa >> >> icp_qa_al_vf 813442 1 >> >> [root at vpn-s QAT]# >> >> >> >> The below are DPDK configuration options in >> config/defconfig_x86_64-native-linuxapp-gcc. >> >> >> include "common_linuxapp" >> >> >> >> CONFIG_RTE_MACHINE="native" >> >> CONFIG_RTE_ARCH="x86_64" >> >> CONFIG_RTE_ARCH_X86_64=y >> >> CONFIG_RTE_ARCH_64=y >> >> CONFIG_RTE_TOOLCHAIN="gcc" >> >> CONFIG_RTE_TOOLCHAIN_GCC=y >> >> CONFIG_RTE_LIBRTE_PMD_AESNI_MB=y >> >> CONFIG_RTE_LIBRTE_CRYPTODEV_DEBUG=y >> >> CONFIG_RTE_LIBRTE_VIRTIO_PMD=y >> >> CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=y >> >> CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=y >> >> CONFIG_RTE_LIBRTE_PMD_QAT=y >> >> CONFIG_RTE_QAT_PMD_MAX_NB_SESSIONS=128 >> >> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_INIT=y >> >> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_TX=y >> >> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_RX=y >> >> CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_DRIVER=y >> >> >> >> The below is my setup.sh >> >> >> export RTE_SDK=/opt/dpdk-2.2.0 >> >> export RTE_TARGET=x86_64-native-linuxapp-gcc >> >> export RTE_ARCH=x86_64 >> >> export AESNI_MULTI_BUFFER_LIB_PATH=/opt/code >> >> make config T=x86_64-native-linuxapp-gcc >> >> sed -ri 's,(PMD_PCAP=).*,\1y,' build/.config >> >> make install T=x86_64-native-linuxapp-gcc >> >> mkdir -p /mnt/huge >> >> mount -t hugetlbfs nodev /mnt/huge >> >> echo 2048 > >> /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages >> >> sudo modprobe uio_pci_generic >> >> sudo modprobe uio >> >> rmmod igb_uio >> >> insmod >> ./x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko >> >> echo -n "0000:00:06.0" > /sys/bus/pci/devices/0000\:00\:06.0/driver/unbind >> >> echo "8086 0443" > /sys/bus/pci/drivers/igb_uio/new_id >> >> lspci -vvd:443 >> >> ./tools/dpdk_nic_bind.py --bind=uio_pci_generic eth1 >> >> ./tools/dpdk_nic_bind.py --bind=uio_pci_generic 00:03.0 >> >> ./tools/dpdk_nic_bind.py -s >> >> >> Also modified the cryptodevs_init() ( examples/ipsec-secgw/ipsec-secgw.c) >> to rte_cryptodev_start(cdev_id); after rte_cryptodev_queue_pair_setup(). >> The purpose of calling this to start the transmit and the receive units >> of the QAT device. It returns success. But still getting the same issue >> i.e. rte_cryptodev_dequeue_burst () does not return crypto packet. Note >> that, when I call rte_cryptodev_queue_pair_setup () with 3rd argument as >> NULL (in which case default configuration will be used) , it crashes. Is >> it a known issue? >> >> >> >> >> >> Regards, >> >> Chinmaya >> >> On Tue, Jun 21, 2016 at 9:10 PM, Sergio Gonzalez Monroy < >> sergio.gonzalez.monroy at intel.com> wrote: >> >>> On 21/06/2016 08:26, Chinmaya Dwibedy wrote: >>> >>>> >>>> Hi Sergio, >>>> >>>> >>>> As suggested by you, run ?app/test' application and then run >>>> 'cryptodev_qat_autotest' . Found the same issue i.e., It halts in >>>> rte_cryptodev_dequeue_burst () (while loop) in process_crypto_request () >>>> function (app/test/test_cryptodev.c). >>>> >>>> Debugged the qat_crypto_pkt_rx_burst() (in >>>> drivers/crypto/qat/qat_crypto.c) and found that , the value of resp_msg is >>>> 0x7F7F7F7F (i.e., ADF_RING_EMPTY_SIG) . Thus it is not getting crypto >>>> packet (i.e., rte_mbuf) from the RX queue. But I find that, in >>>> qat_crypto_pkt_tx_burst(), the qat_alg_write_mbuf_entry() is being called. >>>> The qat_alg_write_mbuf_entry () prints qat_req using rte_hexdump () and >>>> returns zero. >>>> >>>> >>>> One question, the base_addr + tail of tx_q (in >>>> qat_crypto_pkt_tx_burst()) should be same as base_addr + head (in >>>> qat_crypto_pkt_rx_burst()). Right ? Please feel free to correct me if I am >>>> wrong. >>>> >>>> >>>> Also can you please provide some pointers for further debugging? Would >>>> it be configuration issue? Note that, I am using VF in VM. >>>> >>>> >>> In theory you should be able to run ipsec-secgw sample app with QAT VF >>> as long as the setup is right. >>> That the autotest fails would likely point to some possible issues with >>> the setup. >>> >>> Have you tried to run the autotest in the host? >>> >>> Sergio >>> >>> >>>> Thank you once again for your prompt response and great support. >>>> >>>> >>>> Regards, >>>> >>>> Chinmaya >>>> >>>> >>>> On Mon, Jun 20, 2016 at 3:04 PM, Sergio Gonzalez Monroy < >>>> sergio.gonzalez.monroy at intel.com <mailto: >>>> sergio.gonzalez.monroy at intel.com>> wrote: >>>> >>>> On 20/06/2016 10:08, Chinmaya Dwibedy wrote: >>>> >>>> >>>> Hi Sergio, >>>> >>>> >>>> Agreed. We might not dequeue the same amount of crypto ops we >>>> just previously enqueued, it's asynchronous. But in this case, >>>> I have sent just one UDP packet. So there will be one crypto >>>> ops. Right? Also I put a sleep (50) after the >>>> rte_crypto_enqueue_burst() function in ipsec_processing() >>>> (ipsec.c) , so as to allow more time ( for QAT device) for >>>> processing. Still getting the same result i.e., the >>>> rte_crypto_dequeue_burst () functionreturns zero. >>>> >>>> >>>> In case of S/W crypto device (i.e., AESNI), the VM gets >>>> inbound UDP packets on Port 1/eth1, encapsulates (after >>>> consulting its SPD) in an IPsec ESP packet and sends to its >>>> peer through Port 0/eth0 interface. >>>> >>>> >>>> Yes, the security policy, security association and Routing >>>> entries/configurations are exactly same. Please feel free to >>>> let me know if you need additional information. >>>> >>>> >>>> Could you try to run 'app/test' application then run >>>> 'cryptodev_qat_autotest' ? That is a functional test for cryptodev >>>> QAT PMD. >>>> >>>> Sergio >>>> >>>> >>>> >>> >> >
