Hi Durrant,
On x86 platform, you also need to change another constraints
"apic_msix_max" in uts/i86pc/io/pcplusmp/apic_introp.c, this variable
will take effect in apic_alloc_msix_vectors() when you allocate
interrupt handlers. This limitation is x86 platform specific.
Paul Durrant <> wrote:
> Wesley Shao wrote:
>>
>> Just to understand your device's capability and requirement. On x86
>> platforms, if you actually get 64 vectors, they will be on several
>> different interrupt priorities, will that be a problem for you?
>>
>> Some interrupt priority has the maximum of 16 vectors, some has only
>> 8, for example.
>>
>
> The device is a NIC and so is getting, as expected, priority 6. I
> enabled some API and implementation debugging in the kernel and I got
> the following log:
>
> Feb 22 13:31:12 loansun genunix: [ID 210173 kern.notice]
> ddi_intr_alloc: name sfxge dip 0xffffff050d02adb8 type 4 inum 0 count
> 10 behavior 0
> Feb 22 13:31:12 loansun pcplusmp: [ID 593478 kern.notice]
> apic_intr_ops: dip: ffffff050d02adb8 hdlp: ffffff001f768268 intr_op: b
> Feb 22 13:31:12 loansun genunix: [ID 580217 kern.notice]
> ddi_intr_alloc: Requested MSI/Xs 16 Max MSI/Xs limit 8
>
> ^^^^ I set ddi_msix_alloc_limit = 8, and this seems to have taken
> effect
>
> Feb 22 13:31:12 loansun genunix: [ID 915629 kern.notice]
> i_ddi_intr_devi_init: dip ffffff050d02adb8
> Feb 22 13:31:12 loansun pcplusmp: [ID 593478 kern.notice]
> apic_intr_ops: dip: ffffff050d02adb8 hdlp: ffffff001f768238 intr_op: b
> Feb 22 13:31:12 loansun pcplusmp: [ID 593478 kern.notice]
> apic_intr_ops: dip: ffffff050d02adb8 hdlp: ffffff001f768528 intr_op: 0
> Feb 22 13:31:12 loansun pcplusmp: [ID 514352 kern.notice]
> apic_navail_vector: dip: ffffff050d02adb8, pri: 6
> Feb 22 13:31:12 loansun pcplusmp: [ID 803547 kern.info] pcplusmp:
> pci1924,710 (sfxge) instance 0 vector 0x30 ioapic 0xff intin 0xff is
> bound to cpu 7
> Feb 22 13:31:12 loansun pcplusmp: [ID 803547 kern.info] pcplusmp:
> pci1924,710 (sfxge) instance 0 vector 0x32 ioapic 0xff intin 0xff is
> bound to cpu 8
> Feb 22 13:31:12 loansun genunix: [ID 765116 kern.notice]
> ddi_intr_alloc: getting capability
> Feb 22 13:31:12 loansun pcplusmp: [ID 593478 kern.notice]
> apic_intr_ops: dip: ffffff050d02adb8 hdlp: ffffff001f768528 intr_op: 7
> Feb 22 13:31:12 loansun genunix: [ID 428426 kern.notice]
> ddi_intr_alloc: hdlp = 0xffffff052eddd5e8
> Feb 22 13:31:12 loansun genunix: [ID 428426 kern.notice]
> ddi_intr_alloc: hdlp = 0xffffff0516695770
> Feb 22 13:31:12 loansun genunix: [ID 321273 kern.notice]
> ddi_intr_get_pri: hdlp = ffffff052eddd5e8
> Feb 22 13:31:12 loansun genunix: [ID 200900 kern.notice]
> ddi_intr_add_handler: hdlp = 0xffffff052eddd5e8
> Feb 22 13:31:12 loansun genunix: [ID 321273 kern.notice]
> ddi_intr_get_pri: hdlp = ffffff0516695770
> Feb 22 13:31:12 loansun genunix: [ID 200900 kern.notice]
> ddi_intr_add_handler: hdlp = 0xffffff0516695770
> Feb 22 13:31:12 loansun genunix: [ID 453560 kern.notice]
> ddi_intr_get_cap: hdlp = ffffff052eddd5e8
> Feb 22 13:31:12 loansun genunix: [ID 667760 kern.notice]
> ddi_intr_enable: hdlp = ffffff052eddd5e8
> Feb 22 13:31:12 loansun pcplusmp: [ID 593478 kern.notice]
> apic_intr_ops: dip: ffffff050d02adb8 hdlp: ffffff052eddd5e8 intr_op: 3
> Feb 22 13:31:12 loansun pcplusmp: [ID 670866 kern.notice]
> apic_introp_xlate: dip=0xffffff050d02adb8 name=pci1924,710 type=4
> irqno=0x0 Feb 22 13:31:12 loansun pcplusmp: [ID 923188 kern.notice]
> apic_find_irq: dip=0xffffff050d02adb8 vec=0x0 ipl=0x6 type=0x4
> Feb 22 13:31:12 loansun pcplusmp: [ID 589289 kern.notice] apic_addspl:
> dip=0xffffff050d02adb8 type=-5 irqno=0x30 vector=0x60
> Feb 22 13:31:12 loansun pcplusmp: [ID 835447 kern.notice]
> apic_pci_msi_enable_vector: dip=0xffffff050d02adb8
> Feb 22 13:31:12 loansun driver = sfxge, inum=0x0 vector=0x60
> apicid=0x9
> Feb 22 13:31:12 loansun pcplusmp: [ID 323193 kern.notice]
> apic_pci_msi_enable_vector: addr=0xfee09000 data=0x60
> Feb 22 13:31:12 loansun genunix: [ID 667760 kern.notice]
> ddi_intr_enable: hdlp = ffffff0516695770
> Feb 22 13:31:12 loansun pcplusmp: [ID 593478 kern.notice]
> apic_intr_ops: dip: ffffff050d02adb8 hdlp: ffffff0516695770 intr_op: 3
> Feb 22 13:31:12 loansun pcplusmp: [ID 670866 kern.notice]
> apic_introp_xlate: dip=0xffffff050d02adb8 name=pci1924,710 type=4
> irqno=0x1 Feb 22 13:31:12 loansun pcplusmp: [ID 923188 kern.notice]
> apic_find_irq: dip=0xffffff050d02adb8 vec=0x1 ipl=0x6 type=0x4
> Feb 22 13:31:12 loansun pcplusmp: [ID 589289 kern.notice] apic_addspl:
> dip=0xffffff050d02adb8 type=-5 irqno=0x32 vector=0x62
> Feb 22 13:31:12 loansun pcplusmp: [ID 835447 kern.notice]
> apic_pci_msi_enable_vector: dip=0xffffff050d02adb8
> Feb 22 13:31:12 loansun driver = sfxge, inum=0x1 vector=0x62
> apicid=0xa
> Feb 22 13:31:12 loansun pcplusmp: [ID 323193 kern.notice]
> apic_pci_msi_enable_vector: addr=0xfee0a000 data=0x62
> Feb 22 13:31:19 loansun pcplusmp: [ID 593478 kern.notice]
> apic_intr_ops: dip: 0 hdlp: ffffff00202bc9a8 intr_op: d
> #
>
> I'm still only getting two interrupts. I traced the issue to
> apic_navail_vector() which seems to limit me to two because it insists
> on finding consecutive vectors. Why is this? Surely the whole point of
> MSI-X vs. MSI was that the vectors do not have to be consecutive?
>
> Paul
_______________________________________________
driver-discuss mailing list
driver-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/driver-discuss
