Re: [PATCH v8 21/45] powerpc/powernv: Create PEs at PCI hot plugging time
On Wed, Apr 20, 2016 at 01:00:38PM +1000, Alexey Kardashevskiy wrote: >On 04/20/2016 11:12 AM, Gavin Shan wrote: >>On Tue, Apr 19, 2016 at 02:16:42PM +1000, Alexey Kardashevskiy wrote: >>>On 02/17/2016 02:44 PM, Gavin Shan wrote: Currently, the PEs and their associated resources are assigned in ppc_md.pcibios_fixup() except those used by SRIOV VFs. >>> >>>But this new code does not affect IOV and VF's PEs will still be created >>>somewhere else rather than pnv_pci_setup_bridge()? >>> >> >>Correct. VF PEs cannot be created in pnv_pci_setup_bridge() as the PF's >>IOV capability isn't enabled at that point. >> >>> The function is called for once after PCI probing and resources assignment is completed. So it isn't hotplug friendly. This creates PEs dynamically by ppc_md.pcibios_setup_bridge(), which is called on the event during system bootup and PCI hotplug: updating PCI bridge's windows after resource assignment/reassignment are done. For partial hotplug case, where not all PCI devices belonging to the PE are unplugged and plugged again, we just need unbinding/binding the affected PCI devices with the corresponding PE without creating new one. As there is no upstream bridge for root bus that needs to be covered by PE, we have to create PE for root bus in ppc_md.pcibios_setup_bridge() before any other PEs can be created, as PE for root bus is the ancestor to anyone else. >>> >>>We did not need a root bus PE before? What is the other PE reserved for? >>>Comments only say "reserved"... >>> >> >>No, A PE for root bus is needed before. > >Ok. We needed a PE for the root bus and we need it now. What changed? Why do >you reserve another PE? > Originally, all PEs (include the one for root bus) were created at PHB fixup time in pnv_pci_ioda_fixup(). With this patch, all PEs are created in pnv_pci_setup_bridge(). pnv_pci_setup_bridge() is called for every PCI buses other than root bus. It means pnv_pci_setup_bridge() isn't called for root bus. So we have to create PE for root bus before the left PEs are created there. The PE# for root bus is reserved in advance and used in pnv_pci_setup_bridge() at that point. > >> >other PEs can be for the PCI bus >>originated from root port and the subordinate domains. >> Also, the windows of root port or the upstream port of PCIe switch behind root port are extended to be PHB's apertures to accommodate the additional resources needed by newly plugged devices based on the fact: hotpluggable slot is behind root port or downstream port of the PCIe switch behind root port. The extension for those PCI brdiges' windows is done in ppc_md.pcibios_setup_bridge() as well. >>> >>> >>>This patch seems to be doing way too many things, hard to follow. >>> >>>Could you please split the patch into smaller chunks? For example (you can do >>>it totally different): >>>- move pnv_pci_ioda_setup_opal_tce_kill() >>>- move PE creation from pnv_pci_ioda_fixup() to pnv_pci_setup_bridge(); >>>- add pnv_pci_fixup_bridge_resources() >>>- add an extra reserved PE for the root bus (and all this magic with >>>root_pe_idx/root_pe_populated) >>>- ... >>> >> >>I'll evaluate it later. It's always nice to have small patches. Thanks >>for the comments. >> >>> >>> >>> >>>-- >>>Alexey >>> >> >>-- >>To unsubscribe from this list: send the line "unsubscribe linux-pci" in >>the body of a message to majord...@vger.kernel.org >>More majordomo info at http://vger.kernel.org/majordomo-info.html >> > > >-- >Alexey > ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH v8 21/45] powerpc/powernv: Create PEs at PCI hot plugging time
On 04/20/2016 11:12 AM, Gavin Shan wrote: On Tue, Apr 19, 2016 at 02:16:42PM +1000, Alexey Kardashevskiy wrote: On 02/17/2016 02:44 PM, Gavin Shan wrote: Currently, the PEs and their associated resources are assigned in ppc_md.pcibios_fixup() except those used by SRIOV VFs. But this new code does not affect IOV and VF's PEs will still be created somewhere else rather than pnv_pci_setup_bridge()? Correct. VF PEs cannot be created in pnv_pci_setup_bridge() as the PF's IOV capability isn't enabled at that point. The function is called for once after PCI probing and resources assignment is completed. So it isn't hotplug friendly. This creates PEs dynamically by ppc_md.pcibios_setup_bridge(), which is called on the event during system bootup and PCI hotplug: updating PCI bridge's windows after resource assignment/reassignment are done. For partial hotplug case, where not all PCI devices belonging to the PE are unplugged and plugged again, we just need unbinding/binding the affected PCI devices with the corresponding PE without creating new one. As there is no upstream bridge for root bus that needs to be covered by PE, we have to create PE for root bus in ppc_md.pcibios_setup_bridge() before any other PEs can be created, as PE for root bus is the ancestor to anyone else. We did not need a root bus PE before? What is the other PE reserved for? Comments only say "reserved"... No, A PE for root bus is needed before. Ok. We needed a PE for the root bus and we need it now. What changed? Why do you reserve another PE? other PEs can be for the PCI bus originated from root port and the subordinate domains. Also, the windows of root port or the upstream port of PCIe switch behind root port are extended to be PHB's apertures to accommodate the additional resources needed by newly plugged devices based on the fact: hotpluggable slot is behind root port or downstream port of the PCIe switch behind root port. The extension for those PCI brdiges' windows is done in ppc_md.pcibios_setup_bridge() as well. This patch seems to be doing way too many things, hard to follow. Could you please split the patch into smaller chunks? For example (you can do it totally different): - move pnv_pci_ioda_setup_opal_tce_kill() - move PE creation from pnv_pci_ioda_fixup() to pnv_pci_setup_bridge(); - add pnv_pci_fixup_bridge_resources() - add an extra reserved PE for the root bus (and all this magic with root_pe_idx/root_pe_populated) - ... I'll evaluate it later. It's always nice to have small patches. Thanks for the comments. -- Alexey -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- Alexey ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH v8 21/45] powerpc/powernv: Create PEs at PCI hot plugging time
On Tue, Apr 19, 2016 at 02:16:42PM +1000, Alexey Kardashevskiy wrote: >On 02/17/2016 02:44 PM, Gavin Shan wrote: >>Currently, the PEs and their associated resources are assigned >>in ppc_md.pcibios_fixup() except those used by SRIOV VFs. > >But this new code does not affect IOV and VF's PEs will still be created >somewhere else rather than pnv_pci_setup_bridge()? > Correct. VF PEs cannot be created in pnv_pci_setup_bridge() as the PF's IOV capability isn't enabled at that point. > >>The >>function is called for once after PCI probing and resources >>assignment is completed. So it isn't hotplug friendly. >> >>This creates PEs dynamically by ppc_md.pcibios_setup_bridge(), which >>is called on the event during system bootup and PCI hotplug: updating >>PCI bridge's windows after resource assignment/reassignment are done. >>For partial hotplug case, where not all PCI devices belonging to the >>PE are unplugged and plugged again, we just need unbinding/binding >>the affected PCI devices with the corresponding PE without creating >>new one. >> >>As there is no upstream bridge for root bus that needs to be covered >>by PE, we have to create PE for root bus in ppc_md.pcibios_setup_bridge() >>before any other PEs can be created, as PE for root bus is the ancestor >>to anyone else. > >We did not need a root bus PE before? What is the other PE reserved for? >Comments only say "reserved"... > No, A PE for root bus is needed before. other PEs can be for the PCI bus originated from root port and the subordinate domains. >> >>Also, the windows of root port or the upstream port of PCIe switch behind >>root port are extended to be PHB's apertures to accommodate the additional >>resources needed by newly plugged devices based on the fact: hotpluggable >>slot is behind root port or downstream port of the PCIe switch behind >>root port. The extension for those PCI brdiges' windows is done in >>ppc_md.pcibios_setup_bridge() as well. > > >This patch seems to be doing way too many things, hard to follow. > >Could you please split the patch into smaller chunks? For example (you can do >it totally different): >- move pnv_pci_ioda_setup_opal_tce_kill() >- move PE creation from pnv_pci_ioda_fixup() to pnv_pci_setup_bridge(); >- add pnv_pci_fixup_bridge_resources() >- add an extra reserved PE for the root bus (and all this magic with >root_pe_idx/root_pe_populated) >- ... > I'll evaluate it later. It's always nice to have small patches. Thanks for the comments. > > > >-- >Alexey > ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH v8 21/45] powerpc/powernv: Create PEs at PCI hot plugging time
On 02/17/2016 02:44 PM, Gavin Shan wrote: Currently, the PEs and their associated resources are assigned in ppc_md.pcibios_fixup() except those used by SRIOV VFs. But this new code does not affect IOV and VF's PEs will still be created somewhere else rather than pnv_pci_setup_bridge()? The function is called for once after PCI probing and resources assignment is completed. So it isn't hotplug friendly. This creates PEs dynamically by ppc_md.pcibios_setup_bridge(), which is called on the event during system bootup and PCI hotplug: updating PCI bridge's windows after resource assignment/reassignment are done. For partial hotplug case, where not all PCI devices belonging to the PE are unplugged and plugged again, we just need unbinding/binding the affected PCI devices with the corresponding PE without creating new one. As there is no upstream bridge for root bus that needs to be covered by PE, we have to create PE for root bus in ppc_md.pcibios_setup_bridge() before any other PEs can be created, as PE for root bus is the ancestor to anyone else. We did not need a root bus PE before? What is the other PE reserved for? Comments only say "reserved"... Also, the windows of root port or the upstream port of PCIe switch behind root port are extended to be PHB's apertures to accommodate the additional resources needed by newly plugged devices based on the fact: hotpluggable slot is behind root port or downstream port of the PCIe switch behind root port. The extension for those PCI brdiges' windows is done in ppc_md.pcibios_setup_bridge() as well. This patch seems to be doing way too many things, hard to follow. Could you please split the patch into smaller chunks? For example (you can do it totally different): - move pnv_pci_ioda_setup_opal_tce_kill() - move PE creation from pnv_pci_ioda_fixup() to pnv_pci_setup_bridge(); - add pnv_pci_fixup_bridge_resources() - add an extra reserved PE for the root bus (and all this magic with root_pe_idx/root_pe_populated) - ... -- Alexey ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
[PATCH v8 21/45] powerpc/powernv: Create PEs at PCI hot plugging time
Currently, the PEs and their associated resources are assigned in ppc_md.pcibios_fixup() except those used by SRIOV VFs. The function is called for once after PCI probing and resources assignment is completed. So it isn't hotplug friendly. This creates PEs dynamically by ppc_md.pcibios_setup_bridge(), which is called on the event during system bootup and PCI hotplug: updating PCI bridge's windows after resource assignment/reassignment are done. For partial hotplug case, where not all PCI devices belonging to the PE are unplugged and plugged again, we just need unbinding/binding the affected PCI devices with the corresponding PE without creating new one. As there is no upstream bridge for root bus that needs to be covered by PE, we have to create PE for root bus in ppc_md.pcibios_setup_bridge() before any other PEs can be created, as PE for root bus is the ancestor to anyone else. Also, the windows of root port or the upstream port of PCIe switch behind root port are extended to be PHB's apertures to accommodate the additional resources needed by newly plugged devices based on the fact: hotpluggable slot is behind root port or downstream port of the PCIe switch behind root port. The extension for those PCI brdiges' windows is done in ppc_md.pcibios_setup_bridge() as well. Signed-off-by: Gavin Shan --- arch/powerpc/platforms/powernv/pci-ioda.c | 294 +- arch/powerpc/platforms/powernv/pci.h | 2 + 2 files changed, 168 insertions(+), 128 deletions(-) diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c index 565725b..d360607 100644 --- a/arch/powerpc/platforms/powernv/pci-ioda.c +++ b/arch/powerpc/platforms/powernv/pci-ioda.c @@ -197,14 +197,14 @@ static int pnv_ioda2_init_m64(struct pnv_phb *phb) set_bit(phb->ioda.m64_bar_idx, &phb->ioda.m64_bar_alloc); /* -* Strip off the segment used by the reserved PE, which is -* expected to be 0 or last one of PE capabicity. +* Exclude the segments for reserved and root bus PE, which +* are first or last two PEs. */ r = &phb->hose->mem_resources[1]; if (phb->ioda.reserved_pe_idx == 0) - r->start += phb->ioda.m64_segsize; + r->start += (2 * phb->ioda.m64_segsize); else if (phb->ioda.reserved_pe_idx == (phb->ioda.total_pe_num - 1)) - r->end -= phb->ioda.m64_segsize; + r->end -= (2 * phb->ioda.m64_segsize); else pr_warn(" Cannot strip M64 segment for reserved PE#%d\n", phb->ioda.reserved_pe_idx); @@ -284,14 +284,14 @@ static int pnv_ioda1_init_m64(struct pnv_phb *phb) } /* -* Exclude the segment used by the reserved PE, which -* is expected to be 0 or last supported PE#. +* Exclude the segments for reserved and root bus PE, which +* are first or last two PEs. */ r = &phb->hose->mem_resources[1]; if (phb->ioda.reserved_pe_idx == 0) - r->start += phb->ioda.m64_segsize; + r->start += (2 * phb->ioda.m64_segsize); else if (phb->ioda.reserved_pe_idx == (phb->ioda.total_pe_num - 1)) - r->end -= phb->ioda.m64_segsize; + r->end -= (2 * phb->ioda.m64_segsize); else pr_warn(" Cannot cut M64 segment for reserved PE#%d\n", phb->ioda.reserved_pe_idx); @@ -1022,6 +1022,15 @@ static void pnv_ioda_setup_same_PE(struct pci_bus *bus, struct pnv_ioda_pe *pe) pci_name(dev)); continue; } + + /* +* In partial hotplug case, the PCI device might be still +* associated with the PE and needn't be attached to the +* PE again. +*/ + if (pdn->pe_number != IODA_INVALID_PE) + continue; + pdn->pcidev = dev; pdn->pe_number = pe->pe_number; if ((pe->flags & PNV_IODA_PE_BUS_ALL) && dev->subordinate) @@ -1040,9 +1049,26 @@ static struct pnv_ioda_pe *pnv_ioda_setup_bus_PE(struct pci_bus *bus, bool all) struct pci_controller *hose = pci_bus_to_host(bus); struct pnv_phb *phb = hose->private_data; struct pnv_ioda_pe *pe = NULL; + int pe_num; + + /* +* In partial hotplug case, the PE instance might be still alive. +* We should reuse it instead of allocating a new one. +*/ + pe_num = phb->ioda.pe_rmap[bus->number << 8]; + if (pe_num != IODA_INVALID_PE) { + pe = &phb->ioda.pe_array[pe_num]; + pnv_ioda_setup_same_PE(bus, pe); + return NULL; + } + + /* PE number for root bus should have been reserved */ + if (pci_is_root_bus(bus) && + phb->ioda.root_pe_idx != IODA_INVALID_PE) +