Re: [PATCH 1/2] MIPS: SiByte: Set 32-bit bus mask for BCM1250 PCI
On Wed, 7 Nov 2018, Christoph Hellwig wrote: > > +static int sb1250_bus_dma_mask(struct pci_dev *dev, void *data) > > +{ > > + struct sb1250_bus_dma_mask_exclude *exclude = data; > > + > > + if (!exclude->set && (dev->vendor == PCI_VENDOR_ID_SIBYTE && > > + dev->device == PCI_DEVICE_ID_BCM1250_HT)) { > > + exclude->start = dev->subordinate->number; > > + exclude->end = pci_bus_max_busnr(dev->subordinate); > > + exclude->set = true; > > + dev_dbg(&dev->dev, "not disabling DAC for [bus %02x-%02x]", > > + exclude->start, exclude->end); > > + } else if (!exclude->set || > > + (exclude->set && (dev->bus->number < exclude->start || > > +dev->bus->number > exclude->end))) { > > + dev_dbg(&dev->dev, "disabling DAC for device"); > > + dev->dev.bus_dma_mask = DMA_BIT_MASK(32); > > + } else { > > + dev_dbg(&dev->dev, "not disabling DAC for device"); > > + } > > + return 0; > > Hmm, these conditions look very hard to read to me. Wouldn't this > have the same effect? > > if (exclude->set) > return; Nope, `exclude->set' only means we already know what range to exclude (and that gets set mid-way through scanning as the HT bridge is encountered). Then if it's unset, we know we are (still) outside that range. Maybe I can flatten the conditions at the small cost of executing some code unnecessarily. But that won't be a big deal as this stuff is only executed once at boot and isn't performance critical. It'll have to wait until next week though as I'll be travelling throughout the rest of this and won't be able to test anything. Maciej
Re: [PATCH 1/2] MIPS: SiByte: Set 32-bit bus mask for BCM1250 PCI
On Wed, Nov 07, 2018 at 12:08:23AM +, Maciej W. Rozycki wrote: > The Broadcom SiByte BCM1250, BCM1125H and BCM1125 SOCs have an onchip > 32-bit PCI host bridge, and the two former SOCs also have an onchip HT > host bridge. The HT host bridge, where present, appears in the PCI > configuration space as if it was a device on the 32-bit PCI bus behind > the PCI host bridge, however at the hardware level its signals are > routed separately, so these two devices are actually peer host bridges. > > As documented[1] and observed in reality the 32-bit PCI host bridge does > not support 64-bit addressing as it does not support the Dual Address > Cycle (DAC) PCI command, and naturally, being 32-bit only, it has no > means to carry the high 32 address bits otherwise. However the DRAM > controller also included in the SOC supports memory amounts of up to > 16GiB, and due to how the address decoder has been wired in the SOC any > memory beyond 1GiB is actually mapped starting from 4GiB physical up, > that is beyond the 32-bit addressable limit. Consequently if the > maximum amount of memory has been installed, then it will span up to > 19GiB. > > Contrariwise, the HT host bridge does support full 40-bit addressing > defined by the HyperTransport (formerly LDT) specification the bridge > adheres to, depending on the peripherals revision of the SOC[2] either > revision 0.17[3] or revision 1.03[4]. This allows addressing any and > all memory installed, and well beyond. > > Set the bus mask then to limit DMA addressing to 32 bits for all the > devices down the 32-bit PCI host bridge, excluding however any devices > that are down the HT host bridge. > > References: > > [1] "BCM1250/BCM1125/BCM1125H User Manual", Revision 1250_1125-UM100-R, > Broadcom Corporation, 21 Oct 2002, Section 8: "PCI Bus and > HyperTransport Fabric", "Introduction", p. 190 > > [2] same, Table 140: "HyperTransport Configuration Header (Type 1)", p. > 245 > > [3] "Lightning Data Transport IO Specification", Revision 0.17, Advanced > Micro Devices, 21 Jan 2000, Section 3.2.1.2 "Command Packet", p. 8 > > [4] "HyperTransport I/O Link Specification", Revision 1.03, > HyperTransport Technology Consortium, 10 Oct 2001, Section 3.2.1.2 > "Request Packet", pp. 27-28 > > Signed-off-by: Maciej W. Rozycki > --- > Hi, > > This has been verified with a Broadcom SWARM board and an XHCI USB 32-bit > PCI option board plugged to one of the mainboard's 32-bit slots wired to > the PCI host bridge, and then a flash storage device plugged to adapter's > USB socket. > > With 2/2 applied first so that the bus mask is respected and a diagnostic > patch for `dma_direct_alloc' made to debug an earlier issue also applied > the system shows these messages upon boot: > > xhci_hcd :03:00.0: assign IRQ: got 56 > xhci_hcd :03:00.0: enabling bus mastering > xhci_hcd :03:00.0: xHCI Host Controller > xhci_hcd :03:00.0: new USB bus registered, assigned bus number 1 > xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 > xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b48000 > xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 > xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b4c000 > xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 > xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b54000 > xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 > xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b58000 > xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 > xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b5c000 > xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 > xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b6 > xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 > xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b64000 > xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 > xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b68000 > xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 > xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b6c000 > xhci_hcd :03:00.0: hcc params 0x014051c7 hci version 0x100 quirks > 0x00010090 > xhci_hcd :03:00.0: enabling Mem-Wr-Inval > usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.19 > usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 > usb usb1: Product: xHCI Host Controller > usb usb1: Manufacturer: Linux 4.19.0 xhci-hcd > usb usb1: SerialNumber: :03:00.0 > hub 1-0:1.0: USB hub found > hub 1-0:1.0: 2 ports detected > xhci_hcd :03:00.0: xHCI Host Controller > xhci_hcd :03:00.0: new USB bus registered, assigned bus number 2 > xhci_hcd :03:00.0: Host supports USB 3.0 SuperSpeed > xhci_hcd :03:00.0: Host took too long to start, waited 16000 microseconds. > xhci_hcd :03:00.0: startup error -19 > xhci_hcd :03:00.0: USB bus 2 deregistered > xhci_hcd
[PATCH 1/2] MIPS: SiByte: Set 32-bit bus mask for BCM1250 PCI
The Broadcom SiByte BCM1250, BCM1125H and BCM1125 SOCs have an onchip 32-bit PCI host bridge, and the two former SOCs also have an onchip HT host bridge. The HT host bridge, where present, appears in the PCI configuration space as if it was a device on the 32-bit PCI bus behind the PCI host bridge, however at the hardware level its signals are routed separately, so these two devices are actually peer host bridges. As documented[1] and observed in reality the 32-bit PCI host bridge does not support 64-bit addressing as it does not support the Dual Address Cycle (DAC) PCI command, and naturally, being 32-bit only, it has no means to carry the high 32 address bits otherwise. However the DRAM controller also included in the SOC supports memory amounts of up to 16GiB, and due to how the address decoder has been wired in the SOC any memory beyond 1GiB is actually mapped starting from 4GiB physical up, that is beyond the 32-bit addressable limit. Consequently if the maximum amount of memory has been installed, then it will span up to 19GiB. Contrariwise, the HT host bridge does support full 40-bit addressing defined by the HyperTransport (formerly LDT) specification the bridge adheres to, depending on the peripherals revision of the SOC[2] either revision 0.17[3] or revision 1.03[4]. This allows addressing any and all memory installed, and well beyond. Set the bus mask then to limit DMA addressing to 32 bits for all the devices down the 32-bit PCI host bridge, excluding however any devices that are down the HT host bridge. References: [1] "BCM1250/BCM1125/BCM1125H User Manual", Revision 1250_1125-UM100-R, Broadcom Corporation, 21 Oct 2002, Section 8: "PCI Bus and HyperTransport Fabric", "Introduction", p. 190 [2] same, Table 140: "HyperTransport Configuration Header (Type 1)", p. 245 [3] "Lightning Data Transport IO Specification", Revision 0.17, Advanced Micro Devices, 21 Jan 2000, Section 3.2.1.2 "Command Packet", p. 8 [4] "HyperTransport I/O Link Specification", Revision 1.03, HyperTransport Technology Consortium, 10 Oct 2001, Section 3.2.1.2 "Request Packet", pp. 27-28 Signed-off-by: Maciej W. Rozycki --- Hi, This has been verified with a Broadcom SWARM board and an XHCI USB 32-bit PCI option board plugged to one of the mainboard's 32-bit slots wired to the PCI host bridge, and then a flash storage device plugged to adapter's USB socket. With 2/2 applied first so that the bus mask is respected and a diagnostic patch for `dma_direct_alloc' made to debug an earlier issue also applied the system shows these messages upon boot: xhci_hcd :03:00.0: assign IRQ: got 56 xhci_hcd :03:00.0: enabling bus mastering xhci_hcd :03:00.0: xHCI Host Controller xhci_hcd :03:00.0: new USB bus registered, assigned bus number 1 xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b48000 xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b4c000 xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b54000 xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b58000 xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b5c000 xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b6 xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b64000 xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b68000 xhci_hcd :03:00.0: dma_direct_alloc: coherent: 1 xhci_hcd :03:00.0: dma_direct_alloc: returned: a80185b6c000 xhci_hcd :03:00.0: hcc params 0x014051c7 hci version 0x100 quirks 0x00010090 xhci_hcd :03:00.0: enabling Mem-Wr-Inval usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 4.19 usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 usb usb1: Product: xHCI Host Controller usb usb1: Manufacturer: Linux 4.19.0 xhci-hcd usb usb1: SerialNumber: :03:00.0 hub 1-0:1.0: USB hub found hub 1-0:1.0: 2 ports detected xhci_hcd :03:00.0: xHCI Host Controller xhci_hcd :03:00.0: new USB bus registered, assigned bus number 2 xhci_hcd :03:00.0: Host supports USB 3.0 SuperSpeed xhci_hcd :03:00.0: Host took too long to start, waited 16000 microseconds. xhci_hcd :03:00.0: startup error -19 xhci_hcd :03:00.0: USB bus 2 deregistered xhci_hcd :03:00.0: remove, state 1 usb usb1: USB disconnect, device number 1 xhci_hcd :03:00.0: USB bus 1 deregistered usbcore: registered new interface driver usb-storage As you can see `dma_direct_alloc' hands out addresses outside the 32-bit physical range an