On 2009/09/15 20:53, Andrew Gallatin wrote:
xiuyan.w...@sun.com wrote:
/etc/system. But on x86, the total number of interrupt vectors for
interrupt priority 5 is 32 (5 is the default interrupt priority for
network devices), so you can not allocate more interrupts than
32 even with tuning.
With IRM, the above limitation is removed and the allocation algorithm
is more flexible. But for x86, the driver can only allocate more
interrupts for a pci function when the fix of the below CR is available:
http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6669984
So, can network devices use more than 32 interrupts when using IRM,
or are we still limited to a total of 32 interrupts for all network
devices, even when using IRM? Why does Solaris have this limit, and
not Linux or FreeBSD?
I've not studied the design details of IRM. For sparc, each PCI-Express
port of root complex supports 256 MSI/X vectors and the total MSI/X
vectors scale with the number of root PCIe port in system. With IRM, the
DDI framework will try to allocate as more vectors as requested provided
that the driver agrees to reclaim some vectors when the system is short
of vectors.
For APIC-based x86 system, APIC prioritizes the interrupt according to
the vector number and there are 16 vectors for each interrupt level. But
Solaris manages this in a different way. It determines the default
priority of I/O interrupts according to the device class. And it does
not allocate the available vectors evenly among the different interrupt
levels. In doing so, more vectors are available to frequently used
device class. The below table is from
http://cvs.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/uts/i86pc/io/pcplusmp/apic.c
which gives the vector assignment among different interrupt levels:
156 * This array is used to initialize apic_ipls[] (in apic_init()).
157 *
158 * IPL Vector range. as passed to intr_enter
159 * 0 none.
160 * 1,2,3 0x20-0x2f 0x0-0xf
161 * 4 0x30-0x3f 0x10-0x1f
162 * 5 0x40-0x5f 0x20-0x3f
163 * 6 0x60-0x7f 0x40-0x5f
164 * 7,8,9 0x80-0x8f 0x60-0x6f
165 * 10 0x90-0x9f 0x70-0x7f
166 * 11 0xa0-0xaf 0x80-0x8f
167 * ... ...
168 * 15 0xe0-0xef 0xc0-0xcf
169 * 15 0xf0-0xff 0xd0-0xdf
With the fix of 6669984, the available of vectors will scale with the #
of LAPIC. From the current webrev of 6669984, the interrupt allocation
should still base on priority. I'm not sure about the exactly
limitation, but undoubtedly more vectors will be available.
Thanks,
Lucy
In my opinion, it's quite challenging for a IRM-capable driver to handle
the situation of interrupt reclaim when the system is short of interrupt
resources.
Indeed. Allocating interrupts is generally one of the
first things a driver does. At least in my driver, I base the number
of rings off of how many interrupts I am able to use. Asking a
driver to reconfigure like this is like changing a tire on a moving
automobile.
Drew
_______________________________________________
driver-discuss mailing list
driver-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/driver-discuss
