Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
Hi Baolu, On Friday, October 11, 2019 8:54:09 AM CEST Lu Baolu wrote: > Hi Janusz, > > On 9/3/19 3:41 PM, Janusz Krzysztofik wrote: > > Hi Baolu, > > > > On Tuesday, September 3, 2019 3:29:40 AM CEST Lu Baolu wrote: > >> Hi Janusz, > >> > >> On 9/2/19 4:37 PM, Janusz Krzysztofik wrote: > I am not saying that keeping data is not acceptable. I just want to > check whether there are any other solutions. > >>> Then reverting 458b7c8e0dde and applying this patch still resolves the > > issue > >>> for me. No errors appear when mappings are unmapped on device close after > > the > >>> device has been removed, and domain info preserved on device removal is > >>> successfully reused on device re-plug. > >> This patch doesn't look good to me although I agree that keeping data is > >> acceptable. It updates dev->archdata.iommu, but leaves the hardware > >> context/pasid table unchanged. This might cause problems somewhere. > >> > >>> Is there anything else I can do to help? > >> Can you please tell me how to reproduce the problem? > > The most simple way to reproduce the issue, assuming there are no non-Intel > > graphics adapters installed, is to run the following shell commands: > > > > #!/bin/sh > > # load i915 module > > modprobe i915 > > # open an i915 device and keep it open in background > > cat /dev/dri/card0 >/dev/null & > > sleep 2 > > # simulate device unplug > > echo 1 >/sys/class/drm/card0/device/remove > > # make the background process close the device on exit > > kill $! > > > > I tried to reproduce the issue with above instructions. I got below > kernel messages after above operation. Is it the same as what you've > seen? I can't find anything explicitly related to iommu except an IOMMU > fault generated after device got removed and the DMA translation > structures got torn down. Can you please help me to understand how IOMMU > driver triggers the issue? > > > [ 182.217672] [ cut here ] > [ 182.217679] WARNING: CPU: 1 PID: 1285 at > drivers/gpu/drm/drm_mode_config.c:495 drm_mode_config_cleanup+0x2cb/0x2e0 > [ 182.217680] Modules linked in: nls_iso8859_1 snd_soc_skl > snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_ext_core > snd_soc_acpi_intel_match snd_soc_acpi snd_soc_core intel_rapl_msr > snd_hda_codec_hdmi snd_hda_codec_realtek snd_compress > snd_hda_codec_generic ledtrig_audio ac97_bus snd_pcm_dmaengine > snd_hda_intel snd_intel_nhlt snd_hda_codec snd_hda_core snd_hwdep > snd_pcm intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp > kvm_intel kvm snd_seq_midi irqbypass snd_seq_midi_event snd_rawmidi > crct10dif_pclmul crc32_pclmul ghash_clmulni_intel iwlmvm snd_seq > mac80211 libarc4 aesni_intel snd_seq_device snd_timer crypto_simd > ipu3_cio2 cryptd glue_helper iwlwifi v4l2_fwnode intel_cstate > videobuf2_dma_sg videobuf2_memops intel_rapl_perf asix videobuf2_v4l2 > videobuf2_common usbnet mii cfg80211 input_leds serio_raw > intel_wmi_thunderbolt snd mei_me videodev soundcore > intel_xhci_usb_role_switch mei 8250_dw mc roles intel_pch_thermal > hid_sensor_magn_3d hid_sensor_accel_3d hid_sensor_press > [ 182.217709] hid_sensor_incl_3d hid_sensor_als hid_sensor_rotation > hid_sensor_gyro_3d hid_sensor_trigger industrialio_triggered_buffer > kfifo_buf hid_sensor_iio_common industrialio intel_vbtn mac_hid > intel_hid acpi_pad sparse_keymap sch_fq_codel parport_pc ppdev lp > parport ip_tables x_tables hid_sensor_custom hid_sensor_hub > intel_ishtp_hid hid_generic dwc3 ulpi udc_core i2c_designware_platform > i2c_designware_core e1000e psmouse i2c_i801 tg3 usbhid dwc3_pci hid > intel_ish_ipc intel_lpss_pci intel_ishtp intel_lpss wmi > pinctrl_sunrisepoint pinctrl_intel > [ 182.217727] CPU: 1 PID: 1285 Comm: bash Not tainted 5.4.0-rc2+ #2704 > [ 182.217728] Hardware name: Intel Corporation Skylake Client > platform/Skylake Y LPDDR3 RVP3, BIOS SKLSE2R1.R00.B105.B00.1512051901 > 12/05/2015 > [ 182.217731] RIP: 0010:drm_mode_config_cleanup+0x2cb/0x2e0 > [ 182.217732] Code: eb 0c 48 8b 70 48 4c 89 e7 e8 31 f4 ff ff 48 8d 7d > a0 e8 d8 95 ff ff 48 85 c0 75 e6 48 8d 7d a0 e8 1a 87 ff ff e9 ef fd ff > ff <0f> 0b e9 ed fe ff ff 0f 0b eb 98 e8 05 ec 98 ff 0f 1f 44 00 00 0f > [ 182.217733] RSP: 0018:a1c481d37c78 EFLAGS: 00010216 > [ 182.217735] RAX: 95c7da326e08 RBX: 95c7dcf6 RCX: > 801e > [ 182.217735] RDX: 95c7dcf603b8 RSI: 801e RDI: > 95c7dcf60390 > [ 182.217736] RBP: a1c481d37cd8 R08: R09: > 9e035e00 > [ 182.217737] R10: a1c481d37be0 R11: 0001 R12: > 95c7dcf60250 > [ 182.217737] R13: 95c7dcf603b8 R14: a1c481d37ee8 R15: > fff2 > [ 182.217739] FS: 7f804af7f740() GS:95c7e7a8() > knlGS: > [ 182.217739] CS: 0010 DS: ES: CR0: 80050033 > [ 182.217740] CR2: 7ffebc530cec CR3: 000137e0e003 CR4: > 003606e0 > [ 182.217741] DR0:
Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
Hi Janusz, On 9/3/19 3:41 PM, Janusz Krzysztofik wrote: Hi Baolu, On Tuesday, September 3, 2019 3:29:40 AM CEST Lu Baolu wrote: Hi Janusz, On 9/2/19 4:37 PM, Janusz Krzysztofik wrote: I am not saying that keeping data is not acceptable. I just want to check whether there are any other solutions. Then reverting 458b7c8e0dde and applying this patch still resolves the issue for me. No errors appear when mappings are unmapped on device close after the device has been removed, and domain info preserved on device removal is successfully reused on device re-plug. This patch doesn't look good to me although I agree that keeping data is acceptable. It updates dev->archdata.iommu, but leaves the hardware context/pasid table unchanged. This might cause problems somewhere. Is there anything else I can do to help? Can you please tell me how to reproduce the problem? The most simple way to reproduce the issue, assuming there are no non-Intel graphics adapters installed, is to run the following shell commands: #!/bin/sh # load i915 module modprobe i915 # open an i915 device and keep it open in background cat /dev/dri/card0 >/dev/null & sleep 2 # simulate device unplug echo 1 >/sys/class/drm/card0/device/remove # make the background process close the device on exit kill $! I tried to reproduce the issue with above instructions. I got below kernel messages after above operation. Is it the same as what you've seen? I can't find anything explicitly related to iommu except an IOMMU fault generated after device got removed and the DMA translation structures got torn down. Can you please help me to understand how IOMMU driver triggers the issue? [ 182.217672] [ cut here ] [ 182.217679] WARNING: CPU: 1 PID: 1285 at drivers/gpu/drm/drm_mode_config.c:495 drm_mode_config_cleanup+0x2cb/0x2e0 [ 182.217680] Modules linked in: nls_iso8859_1 snd_soc_skl snd_soc_sst_ipc snd_soc_sst_dsp snd_hda_ext_core snd_soc_acpi_intel_match snd_soc_acpi snd_soc_core intel_rapl_msr snd_hda_codec_hdmi snd_hda_codec_realtek snd_compress snd_hda_codec_generic ledtrig_audio ac97_bus snd_pcm_dmaengine snd_hda_intel snd_intel_nhlt snd_hda_codec snd_hda_core snd_hwdep snd_pcm intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm snd_seq_midi irqbypass snd_seq_midi_event snd_rawmidi crct10dif_pclmul crc32_pclmul ghash_clmulni_intel iwlmvm snd_seq mac80211 libarc4 aesni_intel snd_seq_device snd_timer crypto_simd ipu3_cio2 cryptd glue_helper iwlwifi v4l2_fwnode intel_cstate videobuf2_dma_sg videobuf2_memops intel_rapl_perf asix videobuf2_v4l2 videobuf2_common usbnet mii cfg80211 input_leds serio_raw intel_wmi_thunderbolt snd mei_me videodev soundcore intel_xhci_usb_role_switch mei 8250_dw mc roles intel_pch_thermal hid_sensor_magn_3d hid_sensor_accel_3d hid_sensor_press [ 182.217709] hid_sensor_incl_3d hid_sensor_als hid_sensor_rotation hid_sensor_gyro_3d hid_sensor_trigger industrialio_triggered_buffer kfifo_buf hid_sensor_iio_common industrialio intel_vbtn mac_hid intel_hid acpi_pad sparse_keymap sch_fq_codel parport_pc ppdev lp parport ip_tables x_tables hid_sensor_custom hid_sensor_hub intel_ishtp_hid hid_generic dwc3 ulpi udc_core i2c_designware_platform i2c_designware_core e1000e psmouse i2c_i801 tg3 usbhid dwc3_pci hid intel_ish_ipc intel_lpss_pci intel_ishtp intel_lpss wmi pinctrl_sunrisepoint pinctrl_intel [ 182.217727] CPU: 1 PID: 1285 Comm: bash Not tainted 5.4.0-rc2+ #2704 [ 182.217728] Hardware name: Intel Corporation Skylake Client platform/Skylake Y LPDDR3 RVP3, BIOS SKLSE2R1.R00.B105.B00.1512051901 12/05/2015 [ 182.217731] RIP: 0010:drm_mode_config_cleanup+0x2cb/0x2e0 [ 182.217732] Code: eb 0c 48 8b 70 48 4c 89 e7 e8 31 f4 ff ff 48 8d 7d a0 e8 d8 95 ff ff 48 85 c0 75 e6 48 8d 7d a0 e8 1a 87 ff ff e9 ef fd ff ff <0f> 0b e9 ed fe ff ff 0f 0b eb 98 e8 05 ec 98 ff 0f 1f 44 00 00 0f [ 182.217733] RSP: 0018:a1c481d37c78 EFLAGS: 00010216 [ 182.217735] RAX: 95c7da326e08 RBX: 95c7dcf6 RCX: 801e [ 182.217735] RDX: 95c7dcf603b8 RSI: 801e RDI: 95c7dcf60390 [ 182.217736] RBP: a1c481d37cd8 R08: R09: 9e035e00 [ 182.217737] R10: a1c481d37be0 R11: 0001 R12: 95c7dcf60250 [ 182.217737] R13: 95c7dcf603b8 R14: a1c481d37ee8 R15: fff2 [ 182.217739] FS: 7f804af7f740() GS:95c7e7a8() knlGS: [ 182.217739] CS: 0010 DS: ES: CR0: 80050033 [ 182.217740] CR2: 7ffebc530cec CR3: 000137e0e003 CR4: 003606e0 [ 182.217741] DR0: DR1: DR2: [ 182.217742] DR3: DR6: fffe0ff0 DR7: 0400 [ 182.217742] Call Trace: [ 182.217748] ? _cond_resched+0x19/0x40 [ 182.217752] intel_modeset_driver_remove+0xd1/0x150 [ 182.217754] i915_driver_remove+0xb8/0x120 [ 182.217756]
Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
Hi, On 10/1/19 11:01 PM, Janusz Krzysztofik wrote: Hi Baolu, On Tuesday, September 3, 2019 9:41:23 AM CEST Janusz Krzysztofik wrote: Hi Baolu, On Tuesday, September 3, 2019 3:29:40 AM CEST Lu Baolu wrote: Hi Janusz, On 9/2/19 4:37 PM, Janusz Krzysztofik wrote: I am not saying that keeping data is not acceptable. I just want to check whether there are any other solutions. Then reverting 458b7c8e0dde and applying this patch still resolves the issue for me. No errors appear when mappings are unmapped on device close after the device has been removed, and domain info preserved on device removal is successfully reused on device re-plug. This patch doesn't look good to me although I agree that keeping data is acceptable. Any progress with that? Which mailing list should I watch for updates?\ We had a holiday last week. I will go ahead with reproducing it locally. Feel free to let me know if you have any new proposal. Best regards, Baolu Thanks, Janusz It updates dev->archdata.iommu, but leaves the hardware context/pasid table unchanged. This might cause problems somewhere. Is there anything else I can do to help? Can you please tell me how to reproduce the problem? The most simple way to reproduce the issue, assuming there are no non-Intel graphics adapters installed, is to run the following shell commands: #!/bin/sh # load i915 module modprobe i915 # open an i915 device and keep it open in background cat /dev/dri/card0 >/dev/null & sleep 2 # simulate device unplug echo 1 >/sys/class/drm/card0/device/remove # make the background process close the device on exit kill $! Thanks, Janusz Keeping the per device domain info while device is unplugged is a bit dangerous because info->dev might be a wild pointer. We need to work out a clean fix. Thanks, Janusz Best regards, Baolu ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
Hi Baolu, On Tuesday, September 3, 2019 9:41:23 AM CEST Janusz Krzysztofik wrote: > Hi Baolu, > > On Tuesday, September 3, 2019 3:29:40 AM CEST Lu Baolu wrote: > > Hi Janusz, > > > > On 9/2/19 4:37 PM, Janusz Krzysztofik wrote: > > >> I am not saying that keeping data is not acceptable. I just want to > > >> check whether there are any other solutions. > > > Then reverting 458b7c8e0dde and applying this patch still resolves the > issue > > > for me. No errors appear when mappings are unmapped on device close > > > after > the > > > device has been removed, and domain info preserved on device removal is > > > successfully reused on device re-plug. > > > > This patch doesn't look good to me although I agree that keeping data is > > acceptable. Any progress with that? Which mailing list should I watch for updates? Thanks, Janusz > > It updates dev->archdata.iommu, but leaves the hardware > > context/pasid table unchanged. This might cause problems somewhere. > > > > > > > > Is there anything else I can do to help? > > > > Can you please tell me how to reproduce the problem? > > The most simple way to reproduce the issue, assuming there are no non-Intel > graphics adapters installed, is to run the following shell commands: > > #!/bin/sh > # load i915 module > modprobe i915 > # open an i915 device and keep it open in background > cat /dev/dri/card0 >/dev/null & > sleep 2 > # simulate device unplug > echo 1 >/sys/class/drm/card0/device/remove > # make the background process close the device on exit > kill $! > > Thanks, > Janusz > > > > Keeping the per > > device domain info while device is unplugged is a bit dangerous because > > info->dev might be a wild pointer. We need to work out a clean fix. > > > > > > > > Thanks, > > > Janusz > > > > > > > Best regards, > > Baolu > > > > > > > ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
Hi Baolu, On Tuesday, September 3, 2019 3:29:40 AM CEST Lu Baolu wrote: > Hi Janusz, > > On 9/2/19 4:37 PM, Janusz Krzysztofik wrote: > >> I am not saying that keeping data is not acceptable. I just want to > >> check whether there are any other solutions. > > Then reverting 458b7c8e0dde and applying this patch still resolves the issue > > for me. No errors appear when mappings are unmapped on device close after the > > device has been removed, and domain info preserved on device removal is > > successfully reused on device re-plug. > > This patch doesn't look good to me although I agree that keeping data is > acceptable. It updates dev->archdata.iommu, but leaves the hardware > context/pasid table unchanged. This might cause problems somewhere. > > > > > Is there anything else I can do to help? > > Can you please tell me how to reproduce the problem? The most simple way to reproduce the issue, assuming there are no non-Intel graphics adapters installed, is to run the following shell commands: #!/bin/sh # load i915 module modprobe i915 # open an i915 device and keep it open in background cat /dev/dri/card0 >/dev/null & sleep 2 # simulate device unplug echo 1 >/sys/class/drm/card0/device/remove # make the background process close the device on exit kill $! Thanks, Janusz > Keeping the per > device domain info while device is unplugged is a bit dangerous because > info->dev might be a wild pointer. We need to work out a clean fix. > > > > > Thanks, > > Janusz > > > > Best regards, > Baolu > ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
Hi Janusz, On 9/2/19 4:37 PM, Janusz Krzysztofik wrote: I am not saying that keeping data is not acceptable. I just want to check whether there are any other solutions. Then reverting 458b7c8e0dde and applying this patch still resolves the issue for me. No errors appear when mappings are unmapped on device close after the device has been removed, and domain info preserved on device removal is successfully reused on device re-plug. This patch doesn't look good to me although I agree that keeping data is acceptable. It updates dev->archdata.iommu, but leaves the hardware context/pasid table unchanged. This might cause problems somewhere. Is there anything else I can do to help? Can you please tell me how to reproduce the problem? Keeping the per device domain info while device is unplugged is a bit dangerous because info->dev might be a wild pointer. We need to work out a clean fix. Thanks, Janusz Best regards, Baolu ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
Hi Baolu, On Thursday, August 29, 2019 11:08:18 AM CEST Lu Baolu wrote: > Hi, > > On 8/29/19 3:58 PM, Janusz Krzysztofik wrote: > > Hi Baolu, > > > > On Thursday, August 29, 2019 3:43:31 AM CEST Lu Baolu wrote: > >> Hi Janusz, > >> > >> On 8/28/19 10:17 PM, Janusz Krzysztofik wrote: > We should avoid kernel panic when a intel_unmap() is called against > a non-existent domain. > >>> Does that mean you suggest to replace > >>> BUG_ON(!domain); > >>> with something like > >>> if (WARN_ON(!domain)) > >>> return; > >>> and to not care of orphaned mappings left allocated? Is there a way to > > inform > >>> users that their active DMA mappings are no longer valid and they > > shouldn't > >>> call dma_unmap_*()? > >>> > But we shouldn't expect the IOMMU driver not > cleaning up the domain info when a device remove notification comes and > wait until all file descriptors being closed, right? > >>> Shouldn't then the IOMMU driver take care of cleaning up resources still > >>> allocated on device remove before it invalidates and forgets their > > pointers? > >>> > >> > >> You are right. We need to wait until all allocated resources (iova and > >> mappings) to be released. > >> > >> How about registering a callback for BUS_NOTIFY_UNBOUND_DRIVER, and > >> removing the domain info when the driver detachment completes? > > > > Device core calls BUS_NOTIFY_UNBOUND_DRIVER on each driver unbind, regardless > > of a device being removed or not. As long as the device is not unplugged and > > the BUS_NOTIFY_REMOVED_DEVICE notification not generated, an unbound driver is > > not a problem here. > > Morever, BUS_NOTIFY_UNBOUND_DRIVER is called even before > > BUS_NOTIFY_REMOVED_DEVICE so that wouldn't help anyway. > > Last but not least, bus events are independent of the IOMMU driver use via > > DMA-API it exposes. > > Fair enough. > > > > > If keeping data for unplugged devices and reusing it on device re-plug is not > > acceptable then maybe the IOMMU driver should perform reference counting of > > its internal resources occupied by DMA-API users and perform cleanups on last > > release? > > I am not saying that keeping data is not acceptable. I just want to > check whether there are any other solutions. Then reverting 458b7c8e0dde and applying this patch still resolves the issue for me. No errors appear when mappings are unmapped on device close after the device has been removed, and domain info preserved on device removal is successfully reused on device re-plug. Is there anything else I can do to help? Thanks, Janusz > > Best regards, > Baolu > ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
Hi, On 8/29/19 3:58 PM, Janusz Krzysztofik wrote: Hi Baolu, On Thursday, August 29, 2019 3:43:31 AM CEST Lu Baolu wrote: Hi Janusz, On 8/28/19 10:17 PM, Janusz Krzysztofik wrote: We should avoid kernel panic when a intel_unmap() is called against a non-existent domain. Does that mean you suggest to replace BUG_ON(!domain); with something like if (WARN_ON(!domain)) return; and to not care of orphaned mappings left allocated? Is there a way to inform users that their active DMA mappings are no longer valid and they shouldn't call dma_unmap_*()? But we shouldn't expect the IOMMU driver not cleaning up the domain info when a device remove notification comes and wait until all file descriptors being closed, right? Shouldn't then the IOMMU driver take care of cleaning up resources still allocated on device remove before it invalidates and forgets their pointers? You are right. We need to wait until all allocated resources (iova and mappings) to be released. How about registering a callback for BUS_NOTIFY_UNBOUND_DRIVER, and removing the domain info when the driver detachment completes? Device core calls BUS_NOTIFY_UNBOUND_DRIVER on each driver unbind, regardless of a device being removed or not. As long as the device is not unplugged and the BUS_NOTIFY_REMOVED_DEVICE notification not generated, an unbound driver is not a problem here. Morever, BUS_NOTIFY_UNBOUND_DRIVER is called even before BUS_NOTIFY_REMOVED_DEVICE so that wouldn't help anyway. Last but not least, bus events are independent of the IOMMU driver use via DMA-API it exposes. Fair enough. If keeping data for unplugged devices and reusing it on device re-plug is not acceptable then maybe the IOMMU driver should perform reference counting of its internal resources occupied by DMA-API users and perform cleanups on last release? I am not saying that keeping data is not acceptable. I just want to check whether there are any other solutions. Best regards, Baolu ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
Hi Baolu, On Thursday, August 29, 2019 3:43:31 AM CEST Lu Baolu wrote: > Hi Janusz, > > On 8/28/19 10:17 PM, Janusz Krzysztofik wrote: > >> We should avoid kernel panic when a intel_unmap() is called against > >> a non-existent domain. > > Does that mean you suggest to replace > > BUG_ON(!domain); > > with something like > > if (WARN_ON(!domain)) > > return; > > and to not care of orphaned mappings left allocated? Is there a way to inform > > users that their active DMA mappings are no longer valid and they shouldn't > > call dma_unmap_*()? > > > >> But we shouldn't expect the IOMMU driver not > >> cleaning up the domain info when a device remove notification comes and > >> wait until all file descriptors being closed, right? > > Shouldn't then the IOMMU driver take care of cleaning up resources still > > allocated on device remove before it invalidates and forgets their pointers? > > > > You are right. We need to wait until all allocated resources (iova and > mappings) to be released. > > How about registering a callback for BUS_NOTIFY_UNBOUND_DRIVER, and > removing the domain info when the driver detachment completes? Device core calls BUS_NOTIFY_UNBOUND_DRIVER on each driver unbind, regardless of a device being removed or not. As long as the device is not unplugged and the BUS_NOTIFY_REMOVED_DEVICE notification not generated, an unbound driver is not a problem here. Morever, BUS_NOTIFY_UNBOUND_DRIVER is called even before BUS_NOTIFY_REMOVED_DEVICE so that wouldn't help anyway. Last but not least, bus events are independent of the IOMMU driver use via DMA-API it exposes. If keeping data for unplugged devices and reusing it on device re-plug is not acceptable then maybe the IOMMU driver should perform reference counting of its internal resources occupied by DMA-API users and perform cleanups on last release? Thanks, Janusz > > Thanks, > > Janusz > > Best regards, > Baolu > ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
Hi Janusz, On 8/28/19 10:17 PM, Janusz Krzysztofik wrote: We should avoid kernel panic when a intel_unmap() is called against a non-existent domain. Does that mean you suggest to replace BUG_ON(!domain); with something like if (WARN_ON(!domain)) return; and to not care of orphaned mappings left allocated? Is there a way to inform users that their active DMA mappings are no longer valid and they shouldn't call dma_unmap_*()? But we shouldn't expect the IOMMU driver not cleaning up the domain info when a device remove notification comes and wait until all file descriptors being closed, right? Shouldn't then the IOMMU driver take care of cleaning up resources still allocated on device remove before it invalidates and forgets their pointers? You are right. We need to wait until all allocated resources (iova and mappings) to be released. How about registering a callback for BUS_NOTIFY_UNBOUND_DRIVER, and removing the domain info when the driver detachment completes? Thanks, Janusz Best regards, Baolu ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
On Wednesday, August 28, 2019 2:56:18 AM CEST Lu Baolu wrote: > Hi Janusz, > > On 8/27/19 5:35 PM, Janusz Krzysztofik wrote: > > Hi Lu, > > > > On Monday, August 26, 2019 10:29:12 AM CEST Lu Baolu wrote: > >> Hi Janusz, > >> > >> On 8/26/19 4:15 PM, Janusz Krzysztofik wrote: > >>> Hi Lu, > >>> > >>> On Friday, August 23, 2019 3:51:11 AM CEST Lu Baolu wrote: > Hi, > > On 8/22/19 10:29 PM, Janusz Krzysztofik wrote: > > When a perfectly working i915 device is hot unplugged (via sysfs) and > > hot re-plugged again, its dev->archdata.iommu field is not populated > > again with an IOMMU pointer. As a result, the device probe fails on > > DMA mapping error during scratch page setup. > > > > It looks like that happens because devices are not detached from their > > MMUIO bus before they are removed on device unplug. Then, when an > > already registered device/IOMMU association is identified by the > > reinstantiated device's bus and function IDs on IOMMU bus re-attach > > attempt, the device's archdata is not populated with IOMMU information > > and the bad happens. > > > > I'm not sure if this is a proper fix but it works for me so at least it > > confirms correctness of my analysis results, I believe. So far I > > haven't been able to identify a good place where the possibly missing > > IOMMU bus detach on device unplug operation could be added. > > Which kernel version are you testing with? Does it contain below commit? > > commit 458b7c8e0dde12d140e3472b80919cbb9ae793f4 > Author: Lu Baolu > Date: Thu Aug 1 11:14:58 2019 +0800 > >>> > >>> I was using an internal branch based on drm-tip which didn't contain this > >>> commit yet. Fortunately it has been already merged into drm-tip over last > >>> weekend and has effectively fixed the issue. > >> > >> Thanks for testing this. > > > > My testing appeared not sufficiently exhaustive. The fix indeed resolved my > > initially discovered issue of not being able to rebind the i915 driver to a > > re-plugged device, however it brought another, probably more serious problem > > to light. > > > > When an open i915 device is hot unplugged, IOMMU bus notifier now cleans up > > IOMMU info for the device on PCI device remove while the i915 driver is still > > not released, kept by open file descriptors. Then, on last device close, > > cleanup attempts lead to kernel panic raised from intel_unmap() on unresolved > > IOMMU domain. > > We should avoid kernel panic when a intel_unmap() is called against > a non-existent domain. Does that mean you suggest to replace BUG_ON(!domain); with something like if (WARN_ON(!domain)) return; and to not care of orphaned mappings left allocated? Is there a way to inform users that their active DMA mappings are no longer valid and they shouldn't call dma_unmap_*()? > But we shouldn't expect the IOMMU driver not > cleaning up the domain info when a device remove notification comes and > wait until all file descriptors being closed, right? Shouldn't then the IOMMU driver take care of cleaning up resources still allocated on device remove before it invalidates and forgets their pointers? Thanks, Janusz > Best regards, > Baolu > > > > > With commit 458b7c8e0dde reverted and my fix applied, both late device close > > and device re-plug work for me. However, I can realize that's probably still > > not a complete solution, possibly missing some protection against reuse of a > > removed device other than for cleanup. If you think that's the right way to > > go, I can work more on that. > > > > I've had a look at other drivers and found AMD is using somehow similar > > approach. On the other hand, looking at the IOMMU common code I couldn't > > identify any arrangement that would support deferred device cleanup. > > > > If that approach is not acceptable for Intel IOMMU, please suggest a way you'd > > like to have it resolved and I can try to implement it. > > > > Thanks, > > Janusz > > > >> Best regards, > >> Lu Baolu > >> > ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
Hi Janusz, On 8/27/19 5:35 PM, Janusz Krzysztofik wrote: Hi Lu, On Monday, August 26, 2019 10:29:12 AM CEST Lu Baolu wrote: Hi Janusz, On 8/26/19 4:15 PM, Janusz Krzysztofik wrote: Hi Lu, On Friday, August 23, 2019 3:51:11 AM CEST Lu Baolu wrote: Hi, On 8/22/19 10:29 PM, Janusz Krzysztofik wrote: When a perfectly working i915 device is hot unplugged (via sysfs) and hot re-plugged again, its dev->archdata.iommu field is not populated again with an IOMMU pointer. As a result, the device probe fails on DMA mapping error during scratch page setup. It looks like that happens because devices are not detached from their MMUIO bus before they are removed on device unplug. Then, when an already registered device/IOMMU association is identified by the reinstantiated device's bus and function IDs on IOMMU bus re-attach attempt, the device's archdata is not populated with IOMMU information and the bad happens. I'm not sure if this is a proper fix but it works for me so at least it confirms correctness of my analysis results, I believe. So far I haven't been able to identify a good place where the possibly missing IOMMU bus detach on device unplug operation could be added. Which kernel version are you testing with? Does it contain below commit? commit 458b7c8e0dde12d140e3472b80919cbb9ae793f4 Author: Lu Baolu Date: Thu Aug 1 11:14:58 2019 +0800 I was using an internal branch based on drm-tip which didn't contain this commit yet. Fortunately it has been already merged into drm-tip over last weekend and has effectively fixed the issue. Thanks for testing this. My testing appeared not sufficiently exhaustive. The fix indeed resolved my initially discovered issue of not being able to rebind the i915 driver to a re-plugged device, however it brought another, probably more serious problem to light. When an open i915 device is hot unplugged, IOMMU bus notifier now cleans up IOMMU info for the device on PCI device remove while the i915 driver is still not released, kept by open file descriptors. Then, on last device close, cleanup attempts lead to kernel panic raised from intel_unmap() on unresolved IOMMU domain. We should avoid kernel panic when a intel_unmap() is called against a non-existent domain. But we shouldn't expect the IOMMU driver not cleaning up the domain info when a device remove notification comes and wait until all file descriptors being closed, right? Best regards, Baolu With commit 458b7c8e0dde reverted and my fix applied, both late device close and device re-plug work for me. However, I can realize that's probably still not a complete solution, possibly missing some protection against reuse of a removed device other than for cleanup. If you think that's the right way to go, I can work more on that. I've had a look at other drivers and found AMD is using somehow similar approach. On the other hand, looking at the IOMMU common code I couldn't identify any arrangement that would support deferred device cleanup. If that approach is not acceptable for Intel IOMMU, please suggest a way you'd like to have it resolved and I can try to implement it. Thanks, Janusz Best regards, Lu Baolu ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
Hi Lu, On Monday, August 26, 2019 10:29:12 AM CEST Lu Baolu wrote: > Hi Janusz, > > On 8/26/19 4:15 PM, Janusz Krzysztofik wrote: > > Hi Lu, > > > > On Friday, August 23, 2019 3:51:11 AM CEST Lu Baolu wrote: > >> Hi, > >> > >> On 8/22/19 10:29 PM, Janusz Krzysztofik wrote: > >>> When a perfectly working i915 device is hot unplugged (via sysfs) and > >>> hot re-plugged again, its dev->archdata.iommu field is not populated > >>> again with an IOMMU pointer. As a result, the device probe fails on > >>> DMA mapping error during scratch page setup. > >>> > >>> It looks like that happens because devices are not detached from their > >>> MMUIO bus before they are removed on device unplug. Then, when an > >>> already registered device/IOMMU association is identified by the > >>> reinstantiated device's bus and function IDs on IOMMU bus re-attach > >>> attempt, the device's archdata is not populated with IOMMU information > >>> and the bad happens. > >>> > >>> I'm not sure if this is a proper fix but it works for me so at least it > >>> confirms correctness of my analysis results, I believe. So far I > >>> haven't been able to identify a good place where the possibly missing > >>> IOMMU bus detach on device unplug operation could be added. > >> > >> Which kernel version are you testing with? Does it contain below commit? > >> > >> commit 458b7c8e0dde12d140e3472b80919cbb9ae793f4 > >> Author: Lu Baolu > >> Date: Thu Aug 1 11:14:58 2019 +0800 > > > > I was using an internal branch based on drm-tip which didn't contain this > > commit yet. Fortunately it has been already merged into drm-tip over last > > weekend and has effectively fixed the issue. > > Thanks for testing this. My testing appeared not sufficiently exhaustive. The fix indeed resolved my initially discovered issue of not being able to rebind the i915 driver to a re-plugged device, however it brought another, probably more serious problem to light. When an open i915 device is hot unplugged, IOMMU bus notifier now cleans up IOMMU info for the device on PCI device remove while the i915 driver is still not released, kept by open file descriptors. Then, on last device close, cleanup attempts lead to kernel panic raised from intel_unmap() on unresolved IOMMU domain. With commit 458b7c8e0dde reverted and my fix applied, both late device close and device re-plug work for me. However, I can realize that's probably still not a complete solution, possibly missing some protection against reuse of a removed device other than for cleanup. If you think that's the right way to go, I can work more on that. I've had a look at other drivers and found AMD is using somehow similar approach. On the other hand, looking at the IOMMU common code I couldn't identify any arrangement that would support deferred device cleanup. If that approach is not acceptable for Intel IOMMU, please suggest a way you'd like to have it resolved and I can try to implement it. Thanks, Janusz > Best regards, > Lu Baolu > ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
Hi Janusz, On 8/26/19 4:15 PM, Janusz Krzysztofik wrote: Hi Lu, On Friday, August 23, 2019 3:51:11 AM CEST Lu Baolu wrote: Hi, On 8/22/19 10:29 PM, Janusz Krzysztofik wrote: When a perfectly working i915 device is hot unplugged (via sysfs) and hot re-plugged again, its dev->archdata.iommu field is not populated again with an IOMMU pointer. As a result, the device probe fails on DMA mapping error during scratch page setup. It looks like that happens because devices are not detached from their MMUIO bus before they are removed on device unplug. Then, when an already registered device/IOMMU association is identified by the reinstantiated device's bus and function IDs on IOMMU bus re-attach attempt, the device's archdata is not populated with IOMMU information and the bad happens. I'm not sure if this is a proper fix but it works for me so at least it confirms correctness of my analysis results, I believe. So far I haven't been able to identify a good place where the possibly missing IOMMU bus detach on device unplug operation could be added. Which kernel version are you testing with? Does it contain below commit? commit 458b7c8e0dde12d140e3472b80919cbb9ae793f4 Author: Lu Baolu Date: Thu Aug 1 11:14:58 2019 +0800 I was using an internal branch based on drm-tip which didn't contain this commit yet. Fortunately it has been already merged into drm-tip over last weekend and has effectively fixed the issue. Thanks for testing this. Best regards, Lu Baolu ___ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
Hi Lu,
On Friday, August 23, 2019 3:51:11 AM CEST Lu Baolu wrote:
> Hi,
>
> On 8/22/19 10:29 PM, Janusz Krzysztofik wrote:
> > When a perfectly working i915 device is hot unplugged (via sysfs) and
> > hot re-plugged again, its dev->archdata.iommu field is not populated
> > again with an IOMMU pointer. As a result, the device probe fails on
> > DMA mapping error during scratch page setup.
> >
> > It looks like that happens because devices are not detached from their
> > MMUIO bus before they are removed on device unplug. Then, when an
> > already registered device/IOMMU association is identified by the
> > reinstantiated device's bus and function IDs on IOMMU bus re-attach
> > attempt, the device's archdata is not populated with IOMMU information
> > and the bad happens.
> >
> > I'm not sure if this is a proper fix but it works for me so at least it
> > confirms correctness of my analysis results, I believe. So far I
> > haven't been able to identify a good place where the possibly missing
> > IOMMU bus detach on device unplug operation could be added.
>
> Which kernel version are you testing with? Does it contain below commit?
>
> commit 458b7c8e0dde12d140e3472b80919cbb9ae793f4
> Author: Lu Baolu
> Date: Thu Aug 1 11:14:58 2019 +0800
I was using an internal branch based on drm-tip which didn't contain this
commit yet. Fortunately it has been already merged into drm-tip over last
weekend and has effectively fixed the issue.
Thanks,
Janusz
> iommu/vt-d: Detach domain when move device out of group
>
> When removing a device from an iommu group, the domain should
> be detached from the device. Otherwise, the stale domain info
> will still be cached by the driver and the driver will refuse
> to attach any domain to the device again.
>
> Cc: Ashok Raj
> Cc: Jacob Pan
> Cc: Kevin Tian
> Fixes: b7297783c2bb6 ("iommu/vt-d: Remove duplicated code for
> device hotplug")
> Reported-and-tested-by: Vlad Buslov
> Suggested-by: Robin Murphy
> Link: https://lkml.org/lkml/2019/7/26/1133
> Signed-off-by: Lu Baolu
> Signed-off-by: Joerg Roedel
>
> Best regards,
> Lu Baolu
>
> >
> > Signed-off-by: Janusz Krzysztofik
> > ---
> > drivers/iommu/intel-iommu.c | 3 +++
> > 1 file changed, 3 insertions(+)
> >
> > diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
> > index 12d094d08c0a..7cdcd0595408 100644
> > --- a/drivers/iommu/intel-iommu.c
> > +++ b/drivers/iommu/intel-iommu.c
> > @@ -2477,6 +2477,9 @@ static struct dmar_domain
*dmar_insert_one_dev_info(struct intel_iommu *iommu,
> > if (info2) {
> > found = info2->domain;
> > info2->dev = dev;
> > +
> > + if (dev && !dev->archdata.iommu)
> > + dev->archdata.iommu = info2;
> > }
> > }
> >
> >
>
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
Re: [RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
Hi,
On 8/22/19 10:29 PM, Janusz Krzysztofik wrote:
When a perfectly working i915 device is hot unplugged (via sysfs) and
hot re-plugged again, its dev->archdata.iommu field is not populated
again with an IOMMU pointer. As a result, the device probe fails on
DMA mapping error during scratch page setup.
It looks like that happens because devices are not detached from their
MMUIO bus before they are removed on device unplug. Then, when an
already registered device/IOMMU association is identified by the
reinstantiated device's bus and function IDs on IOMMU bus re-attach
attempt, the device's archdata is not populated with IOMMU information
and the bad happens.
I'm not sure if this is a proper fix but it works for me so at least it
confirms correctness of my analysis results, I believe. So far I
haven't been able to identify a good place where the possibly missing
IOMMU bus detach on device unplug operation could be added.
Which kernel version are you testing with? Does it contain below commit?
commit 458b7c8e0dde12d140e3472b80919cbb9ae793f4
Author: Lu Baolu
Date: Thu Aug 1 11:14:58 2019 +0800
iommu/vt-d: Detach domain when move device out of group
When removing a device from an iommu group, the domain should
be detached from the device. Otherwise, the stale domain info
will still be cached by the driver and the driver will refuse
to attach any domain to the device again.
Cc: Ashok Raj
Cc: Jacob Pan
Cc: Kevin Tian
Fixes: b7297783c2bb6 ("iommu/vt-d: Remove duplicated code for
device hotplug")
Reported-and-tested-by: Vlad Buslov
Suggested-by: Robin Murphy
Link: https://lkml.org/lkml/2019/7/26/1133
Signed-off-by: Lu Baolu
Signed-off-by: Joerg Roedel
Best regards,
Lu Baolu
Signed-off-by: Janusz Krzysztofik
---
drivers/iommu/intel-iommu.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 12d094d08c0a..7cdcd0595408 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -2477,6 +2477,9 @@ static struct dmar_domain
*dmar_insert_one_dev_info(struct intel_iommu *iommu,
if (info2) {
found = info2->domain;
info2->dev = dev;
+
+ if (dev && !dev->archdata.iommu)
+ dev->archdata.iommu = info2;
}
}
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
[RFC PATCH] iommu/vt-d: Fix IOMMU field not populated on device hot re-plug
When a perfectly working i915 device is hot unplugged (via sysfs) and
hot re-plugged again, its dev->archdata.iommu field is not populated
again with an IOMMU pointer. As a result, the device probe fails on
DMA mapping error during scratch page setup.
It looks like that happens because devices are not detached from their
MMUIO bus before they are removed on device unplug. Then, when an
already registered device/IOMMU association is identified by the
reinstantiated device's bus and function IDs on IOMMU bus re-attach
attempt, the device's archdata is not populated with IOMMU information
and the bad happens.
I'm not sure if this is a proper fix but it works for me so at least it
confirms correctness of my analysis results, I believe. So far I
haven't been able to identify a good place where the possibly missing
IOMMU bus detach on device unplug operation could be added.
Signed-off-by: Janusz Krzysztofik
---
drivers/iommu/intel-iommu.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 12d094d08c0a..7cdcd0595408 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -2477,6 +2477,9 @@ static struct dmar_domain
*dmar_insert_one_dev_info(struct intel_iommu *iommu,
if (info2) {
found = info2->domain;
info2->dev = dev;
+
+ if (dev && !dev->archdata.iommu)
+ dev->archdata.iommu = info2;
}
}
--
2.21.0
___
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu
