Hi Christoph,
We are facing an issue with masked MSIX vectors received while trying to get
pci vectors when BLK/SCSI-MQ is enabled when number of CPUs are lesser than the
available MSIX vectors. For our ISP25xx chipset, hardware supports 32 MSIX
vectors with MQ enabled. We originally found this issue on system using RH8.0
kernel which is at 4.19 version. The system that failed has 12 CPUs and maximum
MSIX vectors requested were 32.
We observed with new pci_alloc_irq_affinity() callback driver is returning 32
vectors when system has only 12 CPUs. As far as we understand, this call should
have returned maximum 14 MSIX vectors (12 for CPU affinity + 2 reserved in
.pre_vectors of irq_affinity structure). Also, we see that vectors returned
include masked ones. Since driver received 32 vectors, We create 30 qpairs (2
less for reserved). In this scenario, we observed that on some qpairs, driver
is not able to process interrupt because CPUs are masked at the PCI layer.
Looking at the code, we noticed that ‘pre/post’ vectors sets in struct
irq_affinity don’t appear to help here.
From below call we should get only online_cpus() + reserved number of vectors
back while requesting number of vectors, instead we get back numbers that
driver requested.
int pci_alloc_irq_vectors_affinity(struct pci_dev *dev, unsigned int min_vecs,
unsigned int max_vecs, unsigned int flags,
const struct irq_affinity *affd)
{
if (flags & PCI_IRQ_MSIX) {
vecs = __pci_enable_msix_range(dev, NULL, min_vecs, max_vecs,
affd);
if (vecs > 0)
return vecs;
}
}
static int __pci_enable_msix_range(struct pci_dev *dev,
struct msix_entry *entries, int minvec,
int maxvec, const struct irq_affinity *affd)
{
for (;;) {
if (affd) {
nvec = irq_calc_affinity_vectors(minvec, nvec, affd);
if (nvec < minvec)
return -ENOSPC;
}
}
Which in-turn calls irq_calc_affinity_vectors(), Which should return min of
num_online_cpus() + resv
/**
* irq_calc_affinity_vectors - Calculate the optimal number of vectors
* @minvec: The minimum number of vectors available
* @maxvec: The maximum number of vectors available
* @affd: Description of the affinity requirements
*/
int irq_calc_affinity_vectors(int minvec, int maxvec, const struct irq_affinity
*affd)
{
int resv = affd->pre_vectors + affd->post_vectors;
int vecs = maxvec - resv;
int ret;
if (resv > minvec)
return 0;
get_online_cpus();
ret = min_t(int, cpumask_weight(cpu_possible_mask), vecs) + resv;
put_online_cpus();
return ret;
}
We do see the same using 4.20.0-rc6 kernel. See below table, we experimented by
forcing maxcpu parameter to expose lower number of CPUs than vectors requested.
Upstream - 4.20-rc6
MaxCPU= Cores Result
MQ Enabled ISP25xx Unset 48 Pass
MQ Enabled ISP25xx 2 24 Failed
MQ Enabled ISP25xx 4 30 Failed
MQ Enabled ISP27xx Unset 48 Pass
MQ Enabled ISP27xx 2 24 Failed
MQ Enabled ISP27xx 4 30 Failed
Note that RH8.0 kernel which has the code from 4.19 kernel behaves the same
way. We have not be able to do extensive testing with SLES.
We want to make sure we are reading this code right and our understanding is
right. If not, please advise the right expectations and what changes are needed
to address this.
In case our understanding is right, whether we have any known issue in this
area in 4.19 kernel which got addressed in 4.20-rc6 kernel. If yes, can you
please point us to the commit message. If not, what additional data is needed
to debug this further. We have captured PCIe trace and ruled out any issues at
hardware/firmware level and we also see that the MSIX vector associate with the
queue pair where we are not getting interrupts is masked.
We want to understand how to calculate IRQ vectors that driver can request in
such scenario.
Thanks,
Himanshu
