Re: [PATCH RESEND v2 7/7] PCI/hotplug: PowerPC PowerNV PCI hotplug driver
On Wed, 2015-02-18 at 08:30 -0600, Bjorn Helgaas wrote: So the hypervisor call that removes the device from the partition will fail if there are any translations that reference the memory of the device. Let me go through this in excruciating detail to see if I understand what's going on: - PCI core enumerates device D1 - PCI core sets device D1 BAR 0 = 0x1000 - driver claims D1 - driver ioremaps 0x1000 at virtual address V - translation V - 0x1000 is in TLB - driver iounmaps V (but V - 0x1000 translation may remain in TLB) - driver releases D1 - hot-remove D1 (without vm_unmap_aliases(), hypervisor would fail this) - it would be a bug to reference V here, but if we did, the virt-to-phys translation would succeed and we'd have a Master Abort or Unsupported Request on PCI/PCIe - hot-add D2 - PCI core enumerates device D2 - PCI core sets device D2 BAR 0 = 0x1000 - it would be a bug to reference V here (before ioremapping), but if we did, the reference would reach D2 I don't see anything hypervisor-specific here except for the fact that the hypervisor checks for existing translations and most other platforms don't. But it seems like the unexpected PCI aborts could happen on any platform. Well, only if we incorrectly dereferenced an ioremap'ed address for which the driver who owns it is long gone so fairly unlikely. I'm not saying you shouldn't put the vm_unmap_aliases() in the generic unplug code, I wouldn't mind that, but I don't think we have a nasty bug to squash here :) Are we saying that those PCI aborts are OK, since it's a bug to make those references in the first place? Or would we rather take a TLB miss fault instead so the references never make it to PCI? I think a miss fault which is basically a page fault - oops is preferable for debugging (after all that MMIO might hvae been reassigned to another device, so that abort might actually instead turn into writing to the wrong device... bad). However I also think the scenario is very unlikely. I would think there would be similar issues when unmapping and re-mapping plain old physical memory. But I don't see vm_unmap_aliases() calls there, so those issues must be handled differently. Should we handle this PCI hotplug issue the same way we handle RAM? If we don't have a vm_unmap_aliases() in the memory unplug path we probably have a bug on those HVs too :-) Cheers, Ben. ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH RESEND v2 7/7] PCI/hotplug: PowerPC PowerNV PCI hotplug driver
[+cc linux-mm, linux-kernel] For context, the start of this discussion was here: http://lkml.kernel.org/r/1424157203-691-8-git-send-email-gws...@linux.vnet.ibm.com where Gavin is adding a new PCI hotplug driver for PowerNV. That new driver calls vm_unmap_aliases() the same way we do in the existing RPA hotplug driver here: https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/pci/hotplug/rpaphp_core.c#n432 I'm trying to figure out whether it's correct to use vm_unmap_aliases() here, but I'm not an mm person so all I have is my gut feeling that something doesn't smell right. On Tue, Feb 17, 2015 at 6:30 PM, Benjamin Herrenschmidt b...@kernel.crashing.org wrote: On Wed, 2015-02-18 at 11:16 +1100, Gavin Shan wrote: What is vm_unmap_aliases() for? I see this is probably copied from rpaphp_core.c, where it was added by b4a26be9f6f8 (powerpc/pseries: Flush lazy kernel mappings after unplug operations). But I don't know whether: - this is something specific to powerpc, - the lack of vm_unmap_aliases() in other hotplug paths is a bug, - the fact that we only do this on powerpc is covering up a powerpc bug somewhere Yes, I copied this piece of code from rpaphp_core.c. I think Ben might help to answer the questions as he added the patch. I had very quick check on mm/vmalloc.c and it's reasonable to have vm_unmap_aliases() here to flush TLB entries for ioremap() regions, which were unmapped previously. if I'm correct. I don't think it's powerpc specific. It's specific to running under the PowerVM hypervisor, and thus doesn't affect PowerNV, just don't copy it over. It comes from the fact that the generic ioremap code nowadays delays TLB flushing on unmap. The TLB flushing code is what, on powerpc, ensures that we remove the translations from the MMU hash table (the hash table is essentially treated as an extended in-memory TLB), which on pseries turns into hypervisor calls. When running under that hypervisor, the HV ensures that no translation still exists in the hash before allowing a device to be removed from a partition. If translations still exist, the removal fails. So we need to force the generic ioremap code to perform all the TLB flushes for iounmap'ed regions before we complete the unplug operation from a kernel perspective so that the device can be re-assigned to another partition. This is thus useless on platforms like powernv which do not run under such a hypervisor. So the hypervisor call that removes the device from the partition will fail if there are any translations that reference the memory of the device. Let me go through this in excruciating detail to see if I understand what's going on: - PCI core enumerates device D1 - PCI core sets device D1 BAR 0 = 0x1000 - driver claims D1 - driver ioremaps 0x1000 at virtual address V - translation V - 0x1000 is in TLB - driver iounmaps V (but V - 0x1000 translation may remain in TLB) - driver releases D1 - hot-remove D1 (without vm_unmap_aliases(), hypervisor would fail this) - it would be a bug to reference V here, but if we did, the virt-to-phys translation would succeed and we'd have a Master Abort or Unsupported Request on PCI/PCIe - hot-add D2 - PCI core enumerates device D2 - PCI core sets device D2 BAR 0 = 0x1000 - it would be a bug to reference V here (before ioremapping), but if we did, the reference would reach D2 I don't see anything hypervisor-specific here except for the fact that the hypervisor checks for existing translations and most other platforms don't. But it seems like the unexpected PCI aborts could happen on any platform. Are we saying that those PCI aborts are OK, since it's a bug to make those references in the first place? Or would we rather take a TLB miss fault instead so the references never make it to PCI? I would think there would be similar issues when unmapping and re-mapping plain old physical memory. But I don't see vm_unmap_aliases() calls there, so those issues must be handled differently. Should we handle this PCI hotplug issue the same way we handle RAM? Bjorn ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH RESEND v2 7/7] PCI/hotplug: PowerPC PowerNV PCI hotplug driver
On Wed, 2015-02-18 at 11:16 +1100, Gavin Shan wrote: What is vm_unmap_aliases() for? I see this is probably copied from rpaphp_core.c, where it was added by b4a26be9f6f8 (powerpc/pseries: Flush lazy kernel mappings after unplug operations). But I don't know whether: - this is something specific to powerpc, - the lack of vm_unmap_aliases() in other hotplug paths is a bug, - the fact that we only do this on powerpc is covering up a powerpc bug somewhere Yes, I copied this piece of code from rpaphp_core.c. I think Ben might help to answer the questions as he added the patch. I had very quick check on mm/vmalloc.c and it's reasonable to have vm_unmap_aliases() here to flush TLB entries for ioremap() regions, which were unmapped previously. if I'm correct. I don't think it's powerpc specific. It's specific to running under the PowerVM hypervisor, and thus doesn't affect PowerNV, just don't copy it over. It comes from the fact that the generic ioremap code nowadays delays TLB flushing on unmap. The TLB flushing code is what, on powerpc, ensures that we remove the translations from the MMU hash table (the hash table is essentially treated as an extended in-memory TLB), which on pseries turns into hypervisor calls. When running under that hypervisor, the HV ensures that no translation still exists in the hash before allowing a device to be removed from a partition. If translations still exist, the removal fails. So we need to force the generic ioremap code to perform all the TLB flushes for iounmap'ed regions before we complete the unplug operation from a kernel perspective so that the device can be re-assigned to another partition. This is thus useless on platforms like powernv which do not run under such a hypervisor. Cheers, Ben. ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH RESEND v2 7/7] PCI/hotplug: PowerPC PowerNV PCI hotplug driver
On Tue, Feb 17, 2015 at 04:09:16PM -0600, Bjorn Helgaas wrote:
On Tue, Feb 17, 2015 at 06:13:23PM +1100, Gavin Shan wrote:
The patch intends to add standalone driver to support PCI hotplug
for PowerPC PowerNV platform, which runs on top of skiboot firmware.
The firmware identified hotpluggable slots and marked their device
tree node with proper ibm,slot-pluggable and ibm,reset-by-firmware.
The driver simply scans device-tree to create/register PCI hotplug slot
accordingly.
If the skiboot firmware doesn't support slot status retrieval, the PCI
slot device node shouldn't have property ibm,reset-by-firmware. In
that case, none of valid PCI slots will be detected from device tree.
The skiboot firmware doesn't export the capability to access attention
LEDs yet and it's something for TBD.
Signed-off-by: Gavin Shan gws...@linux.vnet.ibm.com
...
+static int disable_slot(struct hotplug_slot *php_slot)
+{
+struct powernv_php_slot *slot = php_slot-private;
+
+if (slot-state != POWERNV_PHP_SLOT_STATE_POPULATED)
+return 0;
+
+pci_lock_rescan_remove();
+pcibios_remove_pci_devices(slot-bus);
+pci_unlock_rescan_remove();
+vm_unmap_aliases();
What is vm_unmap_aliases() for? I see this is probably copied from
rpaphp_core.c, where it was added by b4a26be9f6f8 (powerpc/pseries: Flush
lazy kernel mappings after unplug operations).
But I don't know whether:
- this is something specific to powerpc,
- the lack of vm_unmap_aliases() in other hotplug paths is a bug,
- the fact that we only do this on powerpc is covering up a
powerpc bug somewhere
Yes, I copied this piece of code from rpaphp_core.c. I think Ben might
help to answer the questions as he added the patch. I had very quick
check on mm/vmalloc.c and it's reasonable to have vm_unmap_aliases()
here to flush TLB entries for ioremap() regions, which were unmapped
previously. if I'm correct. I don't think it's powerpc specific.
Thanks,
Gavin
+
+/* Detach the child hotpluggable slots */
+powernv_php_unregister(slot-dn);
+
+/* Update slot state */
+slot-state = POWERNV_PHP_SLOT_STATE_REGISTER;
+return 0;
+}
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: [PATCH RESEND v2 7/7] PCI/hotplug: PowerPC PowerNV PCI hotplug driver
On Tue, Feb 17, 2015 at 06:13:23PM +1100, Gavin Shan wrote:
The patch intends to add standalone driver to support PCI hotplug
for PowerPC PowerNV platform, which runs on top of skiboot firmware.
The firmware identified hotpluggable slots and marked their device
tree node with proper ibm,slot-pluggable and ibm,reset-by-firmware.
The driver simply scans device-tree to create/register PCI hotplug slot
accordingly.
If the skiboot firmware doesn't support slot status retrieval, the PCI
slot device node shouldn't have property ibm,reset-by-firmware. In
that case, none of valid PCI slots will be detected from device tree.
The skiboot firmware doesn't export the capability to access attention
LEDs yet and it's something for TBD.
Signed-off-by: Gavin Shan gws...@linux.vnet.ibm.com
...
+static int disable_slot(struct hotplug_slot *php_slot)
+{
+ struct powernv_php_slot *slot = php_slot-private;
+
+ if (slot-state != POWERNV_PHP_SLOT_STATE_POPULATED)
+ return 0;
+
+ pci_lock_rescan_remove();
+ pcibios_remove_pci_devices(slot-bus);
+ pci_unlock_rescan_remove();
+ vm_unmap_aliases();
What is vm_unmap_aliases() for? I see this is probably copied from
rpaphp_core.c, where it was added by b4a26be9f6f8 (powerpc/pseries: Flush
lazy kernel mappings after unplug operations).
But I don't know whether:
- this is something specific to powerpc,
- the lack of vm_unmap_aliases() in other hotplug paths is a bug,
- the fact that we only do this on powerpc is covering up a
powerpc bug somewhere
+
+ /* Detach the child hotpluggable slots */
+ powernv_php_unregister(slot-dn);
+
+ /* Update slot state */
+ slot-state = POWERNV_PHP_SLOT_STATE_REGISTER;
+ return 0;
+}
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
[PATCH RESEND v2 7/7] PCI/hotplug: PowerPC PowerNV PCI hotplug driver
The patch intends to add standalone driver to support PCI hotplug
for PowerPC PowerNV platform, which runs on top of skiboot firmware.
The firmware identified hotpluggable slots and marked their device
tree node with proper ibm,slot-pluggable and ibm,reset-by-firmware.
The driver simply scans device-tree to create/register PCI hotplug slot
accordingly.
If the skiboot firmware doesn't support slot status retrieval, the PCI
slot device node shouldn't have property ibm,reset-by-firmware. In
that case, none of valid PCI slots will be detected from device tree.
The skiboot firmware doesn't export the capability to access attention
LEDs yet and it's something for TBD.
Signed-off-by: Gavin Shan gws...@linux.vnet.ibm.com
---
drivers/pci/hotplug/Kconfig| 12 ++
drivers/pci/hotplug/Makefile | 4 +
drivers/pci/hotplug/powernv_php.c | 126 +++
drivers/pci/hotplug/powernv_php.h | 70 ++
drivers/pci/hotplug/powernv_php_slot.c | 382 +
5 files changed, 594 insertions(+)
create mode 100644 drivers/pci/hotplug/powernv_php.c
create mode 100644 drivers/pci/hotplug/powernv_php.h
create mode 100644 drivers/pci/hotplug/powernv_php_slot.c
diff --git a/drivers/pci/hotplug/Kconfig b/drivers/pci/hotplug/Kconfig
index df8caec..ef55dae 100644
--- a/drivers/pci/hotplug/Kconfig
+++ b/drivers/pci/hotplug/Kconfig
@@ -113,6 +113,18 @@ config HOTPLUG_PCI_SHPC
When in doubt, say N.
+config HOTPLUG_PCI_POWERNV
+ tristate PowerPC PowerNV PCI Hotplug driver
+ depends on PPC_POWERNV EEH
+ help
+ Say Y here if you run PowerPC PowerNV platform that supports
+ PCI Hotplug
+
+ To compile this driver as a module, choose M here: the
+ module will be called powernv-php.
+
+ When in doubt, say N.
+
config HOTPLUG_PCI_RPA
tristate RPA PCI Hotplug driver
depends on PPC_PSERIES EEH
diff --git a/drivers/pci/hotplug/Makefile b/drivers/pci/hotplug/Makefile
index 4a9aa08..a69665e 100644
--- a/drivers/pci/hotplug/Makefile
+++ b/drivers/pci/hotplug/Makefile
@@ -14,6 +14,7 @@ obj-$(CONFIG_HOTPLUG_PCI_PCIE)+= pciehp.o
obj-$(CONFIG_HOTPLUG_PCI_CPCI_ZT5550) += cpcihp_zt5550.o
obj-$(CONFIG_HOTPLUG_PCI_CPCI_GENERIC) += cpcihp_generic.o
obj-$(CONFIG_HOTPLUG_PCI_SHPC) += shpchp.o
+obj-$(CONFIG_HOTPLUG_PCI_POWERNV) += powernv-php.o
obj-$(CONFIG_HOTPLUG_PCI_RPA) += rpaphp.o
obj-$(CONFIG_HOTPLUG_PCI_RPA_DLPAR)+= rpadlpar_io.o
obj-$(CONFIG_HOTPLUG_PCI_SGI) += sgi_hotplug.o
@@ -50,6 +51,9 @@ ibmphp-objs := ibmphp_core.o \
acpiphp-objs := acpiphp_core.o \
acpiphp_glue.o
+powernv-php-objs := powernv_php.o \
+ powernv_php_slot.o
+
rpaphp-objs:= rpaphp_core.o \
rpaphp_pci.o\
rpaphp_slot.o
diff --git a/drivers/pci/hotplug/powernv_php.c
b/drivers/pci/hotplug/powernv_php.c
new file mode 100644
index 000..e36eaf1
--- /dev/null
+++ b/drivers/pci/hotplug/powernv_php.c
@@ -0,0 +1,126 @@
+/*
+ * PCI Hotplug Driver for PowerPC PowerNV platform.
+ *
+ * Copyright Gavin Shan, IBM Corporation 2015.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include linux/kernel.h
+#include linux/module.h
+#include linux/sysfs.h
+#include linux/pci.h
+#include linux/pci_hotplug.h
+#include linux/string.h
+#include linux/slab.h
+#include asm/opal.h
+#include asm/pnv-pci.h
+
+#include powernv_php.h
+
+#define DRIVER_VERSION 0.1
+#define DRIVER_AUTHOR Gavin Shan, IBM Corporation
+#define DRIVER_DESCPowerPC PowerNV PCI Hotplug Driver
+
+static int powernv_php_register_one(struct device_node *dn)
+{
+ struct powernv_php_slot *slot;
+ const __be32 *prop32;
+ int ret;
+
+ /* Check if it's hotpluggable slot */
+ prop32 = of_get_property(dn, ibm,slot-pluggable, NULL);
+ if (!prop32 || !of_read_number(prop32, 1))
+ return 0;
+
+ prop32 = of_get_property(dn, ibm,reset-by-firmware, NULL);
+ if (!prop32 || !of_read_number(prop32, 1))
+ return 0;
+
+ /* Allocate slot */
+ slot = powernv_php_slot_alloc(dn);
+ if (!slot)
+ return -ENODEV;
+
+ /* Register it */
+ ret = powernv_php_slot_register(slot);
+ if (ret) {
+ powernv_php_slot_put(slot);
+ return ret;
+ }
+
+ return powernv_php_slot_enable(slot-php_slot, false);
+}
+
+int powernv_php_register(struct device_node *dn)
+{
+ struct device_node *child;
+ int ret = 0;
+
+ for_each_child_of_node(dn, child) {
+ ret =
