Hi All,
I do have CentOS Linux release 7.2.1511 (Core) host system, where two Intel QAT devices are installed. I have installed dpdk-2.2.0 and QAT driver as mentioned in section 6.4. (i.e., Installation using 01.org QAT driver) of http://dpdk.readthedocs.io/en/v16.04/cryptodevs/qat.html. I tested "cryptodev_qat_autotest" under app/test. I found this test case to be failed. It appears that, it the dequeue burst API fails to retrieve any processed operations available from the queue pair (on the QAT Crypto device), from the devices processed queue. I have captured and attached the console.log here. Can anyone please suggest, is it an issue with the setup/configuration or QAT PMD code? Kindly point me to right direction whenever you find me wrong.Thank you in advance for your time and support. Here goes my configuration. [root at localhost ~]# service qat_service status There is 2 acceleration device(s) in the system: icp_dev0 - type=dh895xcc, inst_id=0, node_id=1, bdf=83:00:0, #accel=6, #engines=12, state=up icp_dev1 - type=dh895xcc, inst_id=1, node_id=1, bdf=88:00:0, #accel=6, #engines=12, state=up [root at localhost ~]# lspci -nn | grep 0435 83:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [8086:0435] 88:00.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT [8086:0435] [root at localhost ~]# [root at localhost ~]# lspci -nn|grep -e "QAT.*Virtual" 83:01.0 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT Virtual Function [8086:0443] 83:01.1 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT Virtual Function [8086:0443] 83:01.2 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT Virtual Function [8086:0443] 83:01.3 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT Virtual Function [8086:0443] 83:01.4 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT Virtual Function [8086:0443] 83:01.5 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT Virtual Function [8086:0443] ------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------------------------------- 88:04.1 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT Virtual Function [8086:0443] 88:04.2 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT Virtual Function [8086:0443] 88:04.3 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT Virtual Function [8086:0443] 88:04.4 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT Virtual Function [8086:0443] 88:04.5 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT Virtual Function [8086:0443] 88:04.6 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT Virtual Function [8086:0443] 88:04.7 Co-processor [0b40]: Intel Corporation DH895XCC Series QAT Virtual Function [8086:0443] [root at localhost ~]# lspci -nn|grep -e "QAT.*Virtual" | wc -l 64 [root at localhost ~]# Note: This host is capable of PCI passthrough and SR-IOV. [root at localhost ~]# grep -i dmar /var/log/dmesg [ 0.000000] ACPI: DMAR 000000007b69a000 00130 (v01 INTEL S2600WT 00000001 INTL 20091013) [ 0.135336] dmar: Host address width 46 [ 0.135587] dmar: DRHD base: 0x000000fbffc000 flags: 0x0 [ 0.135855] dmar: IOMMU 0: reg_base_addr fbffc000 ver 1:0 cap d2078c106f0466 ecap f020de [ 0.136309] dmar: DRHD base: 0x000000c7ffc000 flags: 0x1 [ 0.136569] dmar: IOMMU 1: reg_base_addr c7ffc000 ver 1:0 cap d2078c106f0466 ecap f020de [ 0.137016] dmar: RMRR base: 0x0000007a3db000 end: 0x0000007a3ddfff [ 0.137274] dmar: ATSR flags: 0x0 [ 0.137522] dmar: ATSR flags: 0x0 [root at localhost ~]# grep -i "iommu.*enabled" /var/log/dmesg [ 0.000000] Intel-IOMMU: enabled [root at localhost test(keystone_admin)]# cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-327.18.2.el7.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/swap crashkernel=auto nomodeset rd.lvm.lv=centos/root rhgb iommu=pt intel_iommu=on [root at localhost test(keystone_admin)]# The below output confirms that, QAT driver is correctly installed. [root at localhost ~]# cat /proc/icp_dh895xcc_dev0/version +--------------------------------------------------+ | Hardware and Software versions for device 0 | +--------------------------------------------------+ Hardware Version: A0 SKU2 Firmware Version: 2.6.0 MMP Version: 1.0.0 Driver Version: 2.6.0 Lowest Compatible Driver: 2.3 QuickAssist API CY Version: 1.9 QuickAssist API DC Version: 1.6 +--------------------------------------------------+ [root at localhost ~]# cat /proc/icp_dh895xcc_dev1/version +--------------------------------------------------+ | Hardware and Software versions for device 1 | +--------------------------------------------------+ Hardware Version: A0 SKU2 Firmware Version: 2.6.0 MMP Version: 1.0.0 Driver Version: 2.6.0 Lowest Compatible Driver: 2.3 QuickAssist API CY Version: 1.9 QuickAssist API DC Version: 1.6 +--------------------------------------------------+ [root at localhost ~]# The # lspci -d:443 confirms the bdf of the 32 VF devices are available per QAT device. [root at localhost ~]# lspci -d:443 83:01.0 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function 83:01.1 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function 83:01.2 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function 83:01.3 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function ------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------- 88:04.3 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function 88:04.4 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function 88:04.5 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function 88:04.6 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function 88:04.7 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function [root at localhost ~]# *DPDK configuration options in config/defconfig_x86_64-native-linuxapp-gcc* CONFIG_RTE_LIBRTE_PMD_AESNI_MB=n CONFIG_RTE_LIBRTE_CRYPTODEV_DEBUG=y CONFIG_RTE_LIBRTE_VIRTIO_PMD=n CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_INIT=n CONFIG_RTE_LIBRTE_VIRTIO_DEBUG_DRIVER=n CONFIG_RTE_LIBRTE_PMD_QAT=y CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_INIT=y CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_TX=y CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_RX=y CONFIG_RTE_LIBRTE_PMD_QAT_DEBUG_DRIVER=y CONFIG_RTE_MAX_MEMSEG=4096 [root at localhost dpdk-2.2.0]# Here goes my setup.sh make config T=x86_64-native-linuxapp-gcc sed -ri 's,(PMD_PCAP=).*,\1y,' build/.config make install T=x86_64-native-linuxapp-gcc mkdir -p /mnt/huge mount -t hugetlbfs nodev /mnt/huge echo 8192 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages echo 8192 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages sudo modprobe uio_pci_generic sudo modprobe uio rmmod igb_uio insmod ./x86_64-native-linuxapp-gcc/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.ko echo -n "0000:83:01.0" > /sys/bus/pci/devices/0000\:83\:01.0/driver/unbind echo "8086 0443" > /sys/bus/pci/drivers/igb_uio/new_id The below OP confirms that, the QAT VF is bound to the DPDK UIO driver. 83:01.0 Co-processor: Intel Corporation DH895XCC Series QAT Virtual Function Subsystem: Intel Corporation Device 0000 Control: I/O- Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0 Region 0: [virtual] Memory at c8280000 (64-bit, non-prefetchable) [size=4K] Region 2: [virtual] Memory at c82a0000 (64-bit, non-prefetchable) [size=4K] Capabilities: [90] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Kernel driver in use: igb_uio Regards, Chinmaya
