Hi, I have a use case where there are two PCIe devices connected to two CPU sockets. Each of the device is exposing 1GB device memory over their one of the BARs(BAR5/6). Since other two BARs(BAR 1/2/3/4) also expose some amount of memory, the total memory requirement on the upstream PCI-bridge goes upto 1.5GB(some round-offs happen at the bridge) per socket. With this the linux kernel fails to allocate region in the physical address space. I see the following errors in the kernel bootup:
[ 6.589964] pci 0000:84:06.0: BAR 14: can't assign mem (size 0x60000000) [ 6.611992] pci 0000:84:00.0: BAR 14: can't assign mem (size 0x100000) [ 6.633446] pci 0000:84:01.0: BAR 14: can't assign mem (size 0x100000) [ 6.654898] pci 0000:84:02.0: BAR 14: can't assign mem (size 0x100000) [ 6.676351] pci 0000:84:03.0: BAR 14: can't assign mem (size 0x100000) [ 6.697806] pci 0000:84:04.0: BAR 14: can't assign mem (size 0x100000) [ 6.719261] pci 0000:84:05.0: BAR 14: can't assign mem (size 0x100000) [ 6.740716] pci 0000:85:00.0: BAR 0: can't assign mem (size 0x10000) ... [ 7.087688] pci 0000:8b:00.0: BAR 4: can't assign mem (size 0x40000000) [ 7.109427] pci 0000:8b:00.0: BAR 0: can't assign mem (size 0x100000) [ 7.130599] pci 0000:8b:00.0: BAR 2: can't assign mem (size 0x2000) I looked at the kernel code specially the files drivers/pci/setup-res.c and pci_bus_alloc_resource() functions, further down to e820.c. There I found this macro #define MAX_GAP_END 0x100000000ull Does this mean that PCI devices can be allocated addresses in 32-bit space only even though the kernel is 64 bit and the device supports 64-bit registers. I can see that one of the reason could be to be backward compatible with 32-bit devices. Is there a way we can overcome this if at all, it is a limitation. kernel version: 3.10.62 Kernel boot logs attached Thanks in Advance for any responses. -Anil
kernel-pci-boot.rtf
Description: RTF file
