Hi all,
I'm trying to add a NIC to an x86 system, but apparently I'm missing a
step - probably to do with the memory range list of the membus-iobus
bridge - can anybody straighten me out?
* So far, I've made a copy of configs/examples/fs.py, and added to it a
copy of the routines makeX86System() and makeLinuxX86System() from
configs/common/FSConfig.py so I can customise makeX86System().
(In addition to adding a NIC, I've also fiddled with the default kernel
images and disc configuration, including deleting the swap image and
adding a disc for /home with the COW layer disabled, even if, for now,
/home is mounted read-only.)
After the disc-related code, I've added:
self.eth0 = IGbE_igb( pci_bus=0, pci_dev=2, pci_func=0 )
self.eth0.pio = self.iobus.master
self.eth0.config = self.iobus.master
self.eth0.dma = self.iobus.slave
which is all that the other architectures appear to be doing.
* Additionally, I've compiled a custom 2.6.28.4 kernel (based on the
configs available from m5sim.org) that includes the igb and e1000e
drivers, and built a custom ubuntu OS image using debootstrap.
* What's going wrong:
As shown in kernel boot log 1 below, the address range allocated to the
NIC's BAR0 (x10000000) is for some reason inaccessible, and also the
interrupt configuration isn't happy.
The machine boots to a shell okay. The device appears in lspci, and the
relevant line of /proc/iomem is:
10000000-1001ffff : 0000:00:02.0
there is no entry in /proc/interrupts, but that's probably not
surprising given the foregoing.
So, the kernel is at least aware that there is a PCI device there.
During boot, as shown in the kernel boot log, attempts to read that BAR
go pear-shaped. After the machine has booted, an attempt to read that
address manually produces the following message in the gem5 trace log:
1449801470671000: system.membus: Unable to find destination for addr
0x10000000, will use default port
warning: caught [10000000:10000003] BadAddressError
the second line being a message I hacked into src/cpu/simple/atomic.cc
in lieu of the assert which otherwise causes such an access to kill the
simulation.
At this point, my best guess is that the reason is that the assignment
of x86_sys.bridge.ranges in FSConfig.connectX86ClassicSystem doesn't
appear to allow for 32-bit BARs, and particularly not the physical
address 0x10000000. As shown in the trace log below, the iobus knows
about that range, but grep says membus does not.
Short of manually adding that address range into bridge.ranges, what is
the right way to map IO space into the mem space? Is there some way of
forwarding the address range change through the bridge? Obviously, a
device with a 32-bit BAR can't take a 64-bit address, so I can't map it
into the IO space beginning at 800000000000000.
How best to resolve this?
By the way, is the kernel panic (boot log 2 below) important? I've seen
others mention it, but no resolution of it.
* I'm confused about how to connect the interrupt line appropriately.
There seem to be two mechanisms for connecting interrupt lines:
* X86IntLine: eg as used in SouthBridge to attach the legacy devices to
the IOAPIC. I'm unclear what this struct does precisely, or why the
legacy devices can't be directly attached to the IOAPIC.
As far as I can tell, the list of struct X86IntLine instances,
system.pc.south_bridge.int_lines, is somehow used to make the
appropriate connections during elaboration, but I don't understand how
this mechanism works (or even where to start).
* X86IntelMPIOIntAssignment: creates entries in the intel MP table so
the kernel can discover what interrupt sources there are and where
they're attached though, again, I can't see where the connections
between the bus/device and the IOAPIC are actually made.
Using the IDE controller as a reference, I see it instantiated in
SouthBridge.py, and there is nothing about the IDE controller in Pc.py
(where SouthBridge is instantiated). The IDE controller is not
mentioned explicitly in makeX86system() either, but there is an
instantiation of X86IntelMPIOIntAssignment called pci_dev4_inta there,
which happens to be the device ID of the IDE controller.
This X86IntelMPIOIntAssignment declaration really confuses me because
the interrupt number in this declaration (destination = IOAPIC pin 16)
doesn't appear to match the parameters reported by the kernel in
/proc/interrupts or as set in SouthBridge.py, both of which say that the
primary IDE controller is on int 14.
Similarly, source_bus_id = 1 (earlier associated with the PCI bus — so
far so good), but I don't see how this bus ID is associated with the
iobus instance (other than in the MP table entry, but I don't see how
that's connected with system.iobus either), nor do I see where PCI bus 0
device ID 4 is associated in the declaration, other than by
source_bus_irq, but if that, then what's the << 2 for? (can't be
allowance for function number, that has 3 bits.)
I haven't long been using gem5 and, as I'm sure it was with the rest of
you, it's a pretty daunting amount of code to absorb all at once,
especially given the last time I did anything with even a moderately
complex C++ code base was something like 15 years ago in my
undergraduate degree.
I've spent most of the intervening time messing with FPGAs and PCI
endpoints. My interests with gem5 are to model interconnect performance
and, if I can produce code of adequate quality, I hope to be able to
contribute those improved models back to the gem5 community.
I'm grateful for any direct assistance you can provide, even
(especially) general advice wrt how to go about adapting the bus and
bridge models for these ends.
=== kernel boot log 1 ===
Intel(R) Gigabit Ethernet Network Driver - version 1.2.45-k2
Copyright (c) 2008 Intel Corporation.
igb 0000:00:02.0: enabling device (0000 -> 0002)
igb 0000:00:02.0: can't find IRQ for PCI INT A; probably buggy MP table
ioremap: invalid physical address 10000000
------------[ cut here ]------------
WARNING: at arch/x86/mm/ioremap.c:212 __ioremap_caller+0x33a/0x380()
Modules linked in:
Pid: 1, comm: swapper Tainted: G W 2.6.28.4 #2
Call Trace:
[<ffffffff8022ee3a>] warn_on_slowpath+0x5a/0x90
[<ffffffff80228bb5>] place_entity.clone.77+0xb5/0xd0
[<ffffffff805eac93>] igb_probe+0x251/0xb4c
[<ffffffff805fb113>] printk+0x40/0x45
[<ffffffff8022198a>] __ioremap_caller+0x33a/0x380
[<ffffffff8055ca60>] ether_setup+0x0/0x80
[<ffffffff8054b6de>] alloc_netdev_mq+0x11e/0x160
[<ffffffff805eac93>] igb_probe+0x251/0xb4c
[<ffffffff802ced3d>] sysfs_addrm_finish+0x1d/0x220
[<ffffffff802cef9d>] sysfs_find_dirent+0x2d/0x40
[<ffffffff802cf175>] __sysfs_add_one+0x75/0xb0
[<ffffffff80391898>] pci_device_probe+0x78/0xa0
[<ffffffff803c7322>] driver_sysfs_add+0x62/0xa0
[<ffffffff803c7549>] driver_probe_device+0x99/0x1b0
[<ffffffff803c7703>] __driver_attach+0xa3/0xb0
[<ffffffff803c7660>] __driver_attach+0x0/0xb0
[<ffffffff803c66c6>] bus_for_each_dev+0x56/0x80
[<ffffffff803c6f0a>] bus_add_driver+0xaa/0x250
[<ffffffff808222d2>] igb_init_module+0x0/0x58
[<ffffffff803c78d1>] driver_register+0x61/0x130
[<ffffffff808222d2>] igb_init_module+0x0/0x58
[<ffffffff80391b25>] __pci_register_driver+0x55/0x90
[<ffffffff8020917a>] do_one_initcall+0x4a/0x1c0
[<ffffffff8037d038>] ida_get_new_above+0x158/0x1c0
[<ffffffff802c7cc2>] proc_register+0x42/0x1c0
[<ffffffff802c82a3>] create_proc_entry+0x53/0xa0
[<ffffffff80256d49>] register_irq_proc+0x89/0xa0
[<ffffffff802c0000>] load_elf_binary+0x16d0/0x1950
[<ffffffff80806b21>] kernel_init+0x70/0xbf
[<ffffffff8020bf79>] child_rip+0xa/0x11
[<ffffffff80806ab1>] kernel_init+0x0/0xbf
[<ffffffff8020bf6f>] child_rip+0x0/0x11
---[ end trace 4eaa2a86a8e2da22 ]---
igb: probe of 0000:00:02.0 failed with error -5
=== kernel boot log 2 ===
WARNING: at arch/x86/kernel/cpu/mtrr/main.c:1604
mtrr_trim_uncached_memory+0x2f9/0x2fb()
WARNING: strange, CPU MTRRs all blank?
Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.28.4 #2
Call Trace:
[<ffffffff8022eda8>] warn_slowpath+0xb8/0xf0
[<ffffffff8021d464>] early_serial_write+0x24/0x50
[<ffffffff8021d464>] early_serial_write+0x24/0x50
[<ffffffff8022f0d9>] __call_console_drivers+0x49/0x60
[<ffffffff80218cf0>] post_set+0x20/0x40
[<ffffffff8080fe63>] mtrr_trim_uncached_memory+0x2f9/0x2fb
[<ffffffff805fb113>] printk+0x40/0x45
[<ffffffff8080c8e1>] setup_arch+0x37a/0x5bb
[<ffffffff80806837>] start_kernel+0x5e/0x2d8
[<ffffffff8080632f>] x86_64_start_kernel+0x0/0xe4
---[ end trace 4eaa2a86a8e2da22 ]---
=== excerpts from gem5 trace output ===
command line: build/X86/gem5.opt
--debug-flags=PCIDEV,PciConfigAll,Bridge,BaseBus,BusAddrRanges,PCEvent,PacketQueue
-d rt/fs-djm configs/fs-djm.py
0: system.iobus: Received range change from slave port system.eth0-pciconf
0: system.iobus: Adding range [0xc000000000001000 : 0xc0000000000010ff]
for id 19
0: system.iobus: Received range change from slave port system.eth0.pio
0: system.iobus: Got address ranges from all slaves
0: system.iobus: Aggregating bus ranges
0: system.iobus: -- Adding default [0xc000000000000000 :
0xc000000000ffffff]
0: system.iobus: -- Adding range [0 : 0x7ffffff]
0: system.iobus: -- Adding range [0xfec00000 : 0xfec00003]
0: system.iobus: -- Adding range [0xfec00010 : 0xfec00013]
12690582500: system.eth0: readConfig: dev 0x2 func 0 reg 0xa 2 bytes: data =
0x200
12690609000: system.eth0: readConfig: dev 0x2 func 0 reg 0 2 bytes: data =
0x8086
12690633000: system.eth0: readConfig: dev 0x2 func 0 reg 0x2 2 bytes: data =
0x10c9
(snip more PCI config cycles)
301374837000: system.iobus: Received range change from slave port
system.eth0.pio
301374837000: system.iobus: Adding range [0x10000000 : 0x1001ffff] for id 18
301374837000: system.iobus: Aggregating bus ranges
301374837000: system.iobus: -- Adding default [0xc000000000000000 :
0xc000000000ffffff]
301374837000: system.iobus: -- Adding range [0 : 0x7ffffff]
301374837000: system.iobus: -- Adding range [0x10000000 : 0x1001ffff]
301374837000: system.iobus: -- Adding range [0xfec00000 : 0xfec00003]
301374837000: system.iobus: -- Adding range [0xfec00010 : 0xfec00013]
301374837000: system.iobus: -- Adding range [0x8000000000000000 :
0x800000000000000f]
301374837000: system.iobus: -- Adding range [0x8000000000000020 :
0x8000000000000021]
301374837000: system.eth0: writeConfig: dev 0x2 func 0 reg 0x10 4 bytes: data =
0x10000000
301374928500: system.eth0: readConfig: dev 0x2 func 0 reg 0x10 4 bytes: data =
0x10000000
1449801470671000: system.membus: Unable to find destination for addr
0x10000000, will use default port
warning: caught [10000000:10000003] BadAddressError
_______________________________________________
gem5-users mailing list
gem5-users@gem5.org
http://m5sim.org/cgi-bin/mailman/listinfo/gem5-users
