Re: Status of PCIe Hotplug?

[Eric van Gyzen]

On 09/27/2016 08:57, Borja Marcos wrote:
> 
>> On 27 Sep 2016, at 15:48, Jan Henrik Sylvester  wrote:
>> 
>> On 09/27/2016 12:16, Borja Marcos wrote:
>>> I have noticed that the GENERIC kernel in 11-STABLE includes the
>>> PCI_HP option, and the hotplug bits seem to be present in the
>>> kernel, but I don’t see any userland support for it.
>>> 
>>> Is it somewhat complete and in that case am I missing something?
>> 
>> I do not know kind of userland support you mean. I just tried:
>> 
>> Plugging in my USB 3.0 ExpressCard while 11.0 is running, the
>> controller was detected and I was able to use USB devices with it.
>> Great.
> 
> Thanks :)
> 
> I was hoping (and I assume it’s the ultimate goal of the project) to
> be able to hot plug PCIe devices such as NVMe drives.
> 
> On Solaris you can replace them provided you power them off
> previously (there’s a command for that, “hotplug”).
> 
> On FreeBSD I’ve tried using devctl but powering off, disabling a
> device and enabling it again has led to a panic.
> 
> Interestingly, I disabled nvme0 using devctl and "nvmecontrol
> devlist" didn’t find any nvme controllers despite having 10
> controllers and 10 drives. However, the ZFS pool of 10 NVMe drives
> was working happily. Degraded of course, with one NVMe missing.

To my knowledge, all the necessary PCIe-layer code is present.  However,
that's just one layer:  Many drivers will likely need changes in order
to cope with surprise removal of their devices.

For that reason, HotPlug needs a lot of testing on a variety of
platforms.  The FreeBSD developer base is much smaller than its user
base, of course, so the variety of our testing is rather limited.  You
can help immensely by giving us detailed bug reports, either on a
mailing list or in Bugzilla.  For a panic, the panic messages and stack
trace of the current thread will be very helpful.  Complete crashinfo(8)
output would be great.

The most relevant userland tool is devctl, followed closely by devinfo
and pciconf.

In the case of Jan's USB 3.0 ExpressCard, it's possible that one or all
of the USB controller drivers (xhci, ehci, uhci) didn't cope with the
surprise removal of the controller.  Before removing the card, try
detaching the driver(s) with "devctl detach xhciN".  There might be more
than one device.  Use "pciconf -lc" to find the HotPlug-capable pcib
devices (bridges).  Use devinfo to find which one is your ExpressCard
slot and find all the devices attached to it.  Then use devctl to detach
the devices.  There could be a tree of devices; in that case, you can
usually start at the level immediately under pcibN; you don't need to
detach every device from the bottom up.  Once all the devices are
detached, you should be able to remove the card safely.

Eric
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Status of PCIe Hotplug?

[Borja Marcos]

> On 27 Sep 2016, at 15:48, Jan Henrik Sylvester  wrote:
> 
> On 09/27/2016 12:16, Borja Marcos wrote:
>> I have noticed that the GENERIC kernel in 11-STABLE includes the PCI_HP 
>> option, and the
>> hotplug bits seem to be present in the kernel, but I don’t see any userland 
>> support for it.
>> 
>> Is it somewhat complete and in that case am I missing something?
> 
> I do not know kind of userland support you mean. I just tried:
> 
> Plugging in my USB 3.0 ExpressCard while 11.0 is running, the controller
> was detected and I was able to use USB devices with it. Great.

Thanks :)

I was hoping (and I assume it’s the ultimate goal of the project) to be able to 
hot plug
PCIe devices such as NVMe drives.

On Solaris you can replace them provided you power them off previously (there’s 
a command
for that, “hotplug”).

On FreeBSD I’ve tried using devctl but powering off, disabling a device and 
enabling it
again has led to a panic.

Interestingly, I disabled nvme0 using devctl and "nvmecontrol devlist" didn’t 
find any
nvme controllers despite having 10 controllers and 10 drives. However, the ZFS 
pool of 10 
NVMe drives was working happily. Degraded of course, with one NVMe missing.




Borja.

___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Status of PCIe Hotplug?

[Jan Henrik Sylvester]

On 09/27/2016 12:16, Borja Marcos wrote:
> I have noticed that the GENERIC kernel in 11-STABLE includes the PCI_HP 
> option, and the
> hotplug bits seem to be present in the kernel, but I don’t see any userland 
> support for it.
> 
> Is it somewhat complete and in that case am I missing something?

I do not know kind of userland support you mean. I just tried:

Plugging in my USB 3.0 ExpressCard while 11.0 is running, the controller
was detected and I was able to use USB devices with it. Great.

Unplugging the ExpressCard led to a panic two out of two trials, no
matter if I had used it for USB devices before or not.

So we have hotplug, but no hotunplug...

With 10.3, it was the other way around. Of course, there was no hotplug,
but when I accidentally pulled the ExpressCard, the USB controller would
just disappear and never reappear without a panic.

I do not know what I like better, though.

Cheers,
Jan Henrik
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Status of PCIe Hotplug?

[Borja Marcos]

Hi,

I have noticed that the GENERIC kernel in 11-STABLE includes the PCI_HP option, 
and the
hotplug bits seem to be present in the kernel, but I don’t see any userland 
support for it.

Is it somewhat complete and in that case am I missing something?

Thanks!





Borja.


___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"