On Thu, Feb 12, 2026 at 12:57:52AM +0900, Koichiro Den wrote: > On Tue, Feb 10, 2026 at 05:30:10PM +0100, Niklas Cassel wrote: > > On Tue, Feb 10, 2026 at 11:07:16PM +0900, Koichiro Den wrote: > > > On Tue, Feb 10, 2026 at 01:24:29PM +0100, Niklas Cassel wrote: > > > > On Mon, Feb 09, 2026 at 09:53:08PM +0900, Koichiro Den wrote: > > > > > Tested on > > > > > ========= > > > > > > > > > > I tested the embedded (DMA) doorbell fallback path (via pci-epf-test) > > > > > on > > > > > R-Car Spider boards: > > > > > > > > > > $ ./pci_endpoint_test -t DOORBELL_TEST > > > > > TAP version 13 > > > > > 1..1 > > > > > # Starting 1 tests from 1 test cases. > > > > > # RUN pcie_ep_doorbell.DOORBELL_TEST ... > > > > > # OK pcie_ep_doorbell.DOORBELL_TEST > > > > > ok 1 pcie_ep_doorbell.DOORBELL_TEST > > > > > # PASSED: 1 / 1 tests passed. > > > > > # Totals: pass:1 fail:0 xfail:0 xpass:0 skip:0 error:0 > > > > > > > > > > with the following message observed on the EP side: > > > > > > > > > > [ 80.464653] pci_epf_test pci_epf_test.0: Using embedded (DMA) > > > > > doorbell fallback > > > > > > > > > > (Note: for the test to pass on R-Car Spider, one of the following was > > > > > required: > > > > > - echo 1048576 > > > > > > functions/pci_epf_test/func1/pci_epf_test.0/bar2_size > > > > > - apply > > > > > https://lore.kernel.org/all/[email protected]/) > > > > > > > > I applied this series on top of branch pci/controller/dwc > > > > on Rock 5B (pcie-dw-rockchip.c). > > > > > > > > On EP side: > > > > [ 39.218533] pci_epf_test pci_epf_test.0: Can't find MSI domain for > > > > EPC > > > > [ 39.219125] pci_epf_test pci_epf_test.0: Using embedded (DMA) > > > > doorbell fallback > > > > > > > > On RC side: > > > > # RUN pcie_ep_doorbell.DOORBELL_TEST ... > > > > [ 40.297892] pci-endpoint-test 0000:01:00.0: Failed to trigger > > > > doorbell in endpoint > > > > # pci_endpoint_test.c:279:DOORBELL_TEST:Expected 0 (0) == ret (-22) > > > > # pci_endpoint_test.c:279:DOORBELL_TEST:Test failed for Doorbell > > > > > > > > # DOORBELL_TEST: Test failed > > > > # FAIL pcie_ep_doorbell.DOORBELL_TEST > > > > not ok 23 pcie_ep_doorbell.DOORBELL_TEST > > > > > > > > Any suggestions? > > > > > > > > (All BARs in pcie-dw-rockchip.c is marked as BAR_RESIZABLE.) > > > > > > Thank you for testing. > > > > > > If the failure was observed in a scenario other than a plain > > > `./pci_endpoint_test -t DOORBELL_TEST`, could you please try again with > > > [1] > > > applied as well? > > > > > > [1] > > > https://lore.kernel.org/linux-pci/[email protected]/ > > > > I applied that series, but I got the same problem. > > > > I added debug, and the EP side does use the correct address for the eDMA: > > [ 26.279457] msg_addr: 0xa403800a0 > > [ 26.279898] phys_addr: 0xa40300000 offset: 0x800a0 > > > > > > If I write to the msg_addr directly on the EP using devmem, I do see the > > print > > that I added in the IRQ handler: > > # devmem 0xa403800a0 32 0 > > [ 155.861989] dw_edma_interrupt_emulated:696 > > # devmem 0xa403800a0 32 0 > > [ 158.809160] dw_edma_interrupt_emulated:696 > > [ 158.809543] pci_epf_test_doorbell_primary:729 > > # [ 158.809986] pci_epf_test_doorbell_handler:703 > > # devmem 0xa403800a0 32 0 > > [ 161.241326] dw_edma_interrupt_emulated:696 > > # devmem 0xa403800a0 32 0 > > [ 163.466054] dw_edma_interrupt_emulated:696 > > # devmem 0xa403800a0 32 0 > > [ 167.378662] dw_edma_interrupt_emulated:696 > > [ 167.379045] pci_epf_test_doorbell_primary:729 > > # [ 167.379512] pci_epf_test_doorbell_handler:703 > > # devmem 0xa403800a0 32 0 > > [ 168.880179] dw_edma_interrupt_emulated:696 > > # devmem 0xa403800a0 32 0 > > [ 170.492176] dw_edma_interrupt_emulated:696 > > # devmem 0xa403800a0 32 0 > > [ 171.729154] dw_edma_interrupt_emulated:696 > > # devmem 0xa403800a0 32 0 > > [ 173.481271] dw_edma_interrupt_emulated:696 > > # devmem 0xa403800a0 32 0 > > [ 174.985787] dw_edma_interrupt_emulated:696 > > # devmem 0xa403800a0 32 0 > > [ 176.517131] dw_edma_interrupt_emulated:696 > > [ 176.517511] pci_epf_test_doorbell_primary:729 > > # [ 176.517963] pci_epf_test_doorbell_handler:703 > > > > But not on every write.... > > > > I'm not sure, but could this perhaps be because we are missing this patch: > > https://lore.kernel.org/dmaengine/[email protected]/ > > Thank you for the detailed debugging. > > I don't have a Rock 5B to reproduce locally, but your log indicates that > the emulated interrupt is not always delivered on the same eDMA IRQ line. > On RK3588 (rk3588-extra.dtsi) there are multiple eDMA IRQs (dma0-4), while > pci-epf-test requests only epf->db_msg[0].virq (IRQ 53 in your > /proc/interrupts). For R-Car S4 Spider, chip->nr_irqs == 1 and I wasn't > able to verify whether my earlier concern here: > https://lore.kernel.org/linux-pci/p4ommmpcjegvb4lafzecf67tzmdodtuqexeoifcn5eh7xqyp2y@ss76d3ubbsw7/ > > > The proposed dmaengine_{register,unregister}_selfirq() APIs are > > device-wide (i.e. not per channel), so I'm not sure which "channel" you > > refer to here. Also, when chip->nr_irqs > 1 on EP, dw-edma distributes > > channels across multiple IRQ vectors, and it's unclear (at least to me) > > which IRQ vector the emulated interrupt ("fake irq") is expected to be > > delivered on. > > actually holds true in practice. Your report makes it clear that the > emulated interrupt can indeed be delivered on different IRQ vectors. > > One hypothesis is that: we currently program msg.data = 0 for the > "embedded" doorbell, and we write to DMA_READ_INT_STATUS_OFF. The register > field (RD_DONE_INT_STATUS) is defined per-channel, and the register > supports interrupt emulation by writes, so it might be that writing BIT(n) > selects the channel/irq line, while writing 0 does not consistently map to > a specific one. > > Could you try a quick experiment on the Rock 5B EP side? > > devmem 0xa403800a0 32 1 > devmem 0xa403800a0 32 2 > devmem 0xa403800a0 32 4 > devmem 0xa403800a0 32 8 > > and see if the interrupt consistently lands on a specific one of IRQ53-56 > for each value? > > If that is the case, we can make msg.data non-zero value instead of always 0. > > If that is not the case, [...]
Thanks to the experiment by Niklas, it turns out that this is not the case. > [...] then we may need to consider two less ideal > options: > - switch back to the ~v3 approach, where we run the registered hook > exactly at the time when the emulated interrupt is deasserted. (ref. > > https://lore.kernel.org/linux-pci/[email protected]/) > - or, require users to request_irq() for all irq vectors associated with > all channels. However, this would not be very attractive from a design > perspective. So now revisiting this, in hindsight, this part (the two options) was premature. I'm considering taking a different approach, like: - alloc a single virtual IRQ dedicated to the emulated interrupt, associate it with dummy_irq_chip + handle_simple_irq, and have the parent handlers clear the emulated interrupt (i.e. write 0 to INT_CLEAR) before invoking generic_handle_irq() from parent handlers unconditionally (*). Or, create a dedicated irq_chip and use handle_level_irq(), where the emulated interrupt is cleared in the irq_ack path. The virtual IRQ is taught via pci_epc_get_aux_resources(). (*): Because the Done/Abort bits are not set by the write into the INT_STATUS, and because a DMA transfer completion may set those bits between the write and the IRQ handling, as I understand it, we still need to call generic_handle_irq() unconditionally. I'm planning to rework the relevant code for v7. Kind regards, Koichiro > > > > > # dmesg | grep eDMA > > [ 1.243339] rockchip-dw-pcie a40000000.pcie-ep: eDMA: unroll T, 2 wr, 2 > > rd > > > > # cat /proc/interrupts | grep edma > > 53: 8 0 0 0 0 0 > > 0 0 GICv3 303 Level dw-edma-core:a40000000.pcie-ep, > > pci-ep-test-doorbell > > 54: 7 0 0 0 0 0 > > 0 0 GICv3 304 Level dw-edma-core:a40000000.pcie-ep > > 55: 15 0 0 0 0 0 > > 0 0 GICv3 301 Level dw-edma-core:a40000000.pcie-ep > > 56: 7 0 0 0 0 0 > > 0 0 GICv3 302 Level dw-edma-core:a40000000.pcie-ep > > > > > > > > Anyway, I was still curious why this did never worked when writing from the > > host side, even when running the test case many, many times. > > AFAICT, the inbound translation looks correct. > > > > RK3588 (pcie-dw-rockchip.c) exposes the DMA registers in BAR4 by default. > > If I hack pci-epf-test on top of your patch to unconditionally return BAR4 > > with > > offset 0xa0, it works. So my best guess is that the fixed inbound > > translation > > in BAR4 (to the eDMA registers) somehow messes with the inbound translation > > if > > another BAR tries to use an inbound translation to the eDMA registers as > > well. > > Thanks a lot for letting me know that. I see two possible ways forward: > > (a) extend PCI_EPC_AUX_DMA_CTRL_MMIO to optionally describe that the DMA > MMIO window is already mapped to a fixed BAR and should be reused, so > EPFs avoid creating a second mapping to the same target. I guess it > could be treated as a quirk for "rockchip,rk3588-pcie-ep". > > (b) alternatively, clear the default BAR4 mapping on RK3588 at least > temporarily when using the pci-epf-msi doorbell fallback. > > Koichiro > > > > > > > Kind regards, > > Niklas > > >
