Re: [PATCH RESEND] PCI: dwc: Fix MSI not work after resume
Hi,
On 23/03/21 8:42 pm, Bjorn Helgaas wrote:
> [-cc Dilip (mail to him bounced)]
>
> On Tue, Mar 23, 2021 at 11:01:15AM +0800, Jisheng Zhang wrote:
>> On Mon, 22 Mar 2021 20:24:41 -0500 Bjorn Helgaas wrote:
>>>
>>> [+cc Kishon, Richard, Lucas, Dilip]
>>>
>>> On Mon, Mar 01, 2021 at 11:10:31AM +0800, Jisheng Zhang wrote:
After we move dw_pcie_msi_init() into core -- dw_pcie_host_init(), the
MSI stops working after resume. Because dw_pcie_host_init() is only
called once during probe. To fix this issue, we move dw_pcie_msi_init()
to dw_pcie_setup_rc().
>>>
>>> This patch looks fine, but I don't think the commit log tells the
>>> whole story.
>>>
>>> Prior to 59fbab1ae40e, it looks like the only dwc-based drivers with
>>> resume functions were dra7xx, imx6, intel-gw, and tegra [1].
>>>
>>> Only tegra called dw_pcie_msi_init() in the resume path, and I do
>>> think 59fbab1ae40e broke MSI after resume because it removed the
>>> dw_pcie_msi_init() call from tegra_pcie_enable_msi_interrupts().
>>>
>>> I'm not convinced this patch fixes it reliably, though. The call
>>> chain looks like this:
>>>
>>> tegra_pcie_dw_resume_noirq
>>> tegra_pcie_dw_start_link
>>> if (dw_pcie_wait_for_link(pci))
>>> dw_pcie_setup_rc
>>>
>>> dw_pcie_wait_for_link() returns 0 if the link is up, so we only call
>>> dw_pcie_setup_rc() in the case where the link *didn't* come up. If
>>> the link comes up nicely without retry, we won't call
>>> dw_pcie_setup_rc() and hence won't call dw_pcie_msi_init().
>>
>> The v1 version patch was sent before commit 275e88b06a (PCI: tegra: Fix host
>> link initialization"). At that time, the resume path looks like this:
>>
>> tegra_pcie_dw_resume_noirq
>> tegra_pcie_dw_host_init
>> tegra_pcie_prepare_host
>> dw_pcie_setup_rc
>>
>> so after patch, dw_pcie_msi_init() will be called. But now it seems that
>> the tegra version needs one more fix for the resume.
>>
>> So could I sent a new patch to update the commit-msg a bit?
>
> This patch only touches the dwc core, and the commit log says
> generically that it fixes MSI after resume, so one could assume that
> it applies to all dwc-based drivers. But I don't think it's that
> simple, so I'd like to know *which* drivers are fixed and which
> commits are related. I don't see how 59fbab1ae40e breaks anything
> except tegra.
>
>>> Since then, exynos added a resume function. My guess is MSI never
>>> worked after resume for dra7xx, exynos, imx6, and intel-gw because
>>> they don't call dw_pcie_msi_init() in their resume functions.
>>>
>>> This patch looks like it should fix MSI after resume for exynos, imx6,
>>> and intel-gw because they *do* call dw_pcie_setup_rc() from their
>>> resume functions [2], and after this patch, dw_pcie_msi_init() will be
>>> called from there.
>>>
>>> I suspect MSI after resume still doesn't work on dra7xx.
>>
>> I checked the dra7xx history, I'm afraid that the resume never works
>> from the beginning if the host lost power during suspend, I guess the
>> platform never power off the host but only the phy?
Suspend on dra7xx disabled clocks and powered off phy (at-least while
suspend/resume hooks were merged) and resume enabled clocks and phy.
However suspend/resume is broken in dra7xx system and is not validated.
I'll send a patch to remove the suspend/resume hooks in dra7xx.
Thanks
Kishon
>
> Sounds like that would make sense.
>
>>> [1] git grep -A20 -e "static.*resume_noirq"
>>> 59fbab1ae40e^:drivers/pci/controller/dwc
>>> [2] git grep -A20 -e "static.*resume_noirq" drivers/pci/controller/dwc
>>>
Fixes: 59fbab1ae40e ("PCI: dwc: Move dw_pcie_msi_init() into core")
Reviewed-by: Rob Herring
Signed-off-by: Jisheng Zhang
---
Since v1:
- collect Reviewed-by tag
drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c
b/drivers/pci/controller/dwc/pcie-designware-host.c
index 7e55b2b66182..e6c274f4485c 100644
--- a/drivers/pci/controller/dwc/pcie-designware-host.c
+++ b/drivers/pci/controller/dwc/pcie-designware-host.c
@@ -400,7 +400,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
}
dw_pcie_setup_rc(pp);
- dw_pcie_msi_init(pp);
if (!dw_pcie_link_up(pci) && pci->ops && pci->ops->start_link) {
ret = pci->ops->start_link(pci);
@@ -551,6 +550,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
}
}
+ dw_pcie_msi_init(pp);
+
/* Setup RC BARs */
dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 0x0004);
dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_1, 0x);
--
2.30.1
>>
Re: [PATCH RESEND] PCI: dwc: Fix MSI not work after resume
[-cc Dilip (mail to him bounced)]
On Tue, Mar 23, 2021 at 11:01:15AM +0800, Jisheng Zhang wrote:
> On Mon, 22 Mar 2021 20:24:41 -0500 Bjorn Helgaas wrote:
> >
> > [+cc Kishon, Richard, Lucas, Dilip]
> >
> > On Mon, Mar 01, 2021 at 11:10:31AM +0800, Jisheng Zhang wrote:
> > > After we move dw_pcie_msi_init() into core -- dw_pcie_host_init(), the
> > > MSI stops working after resume. Because dw_pcie_host_init() is only
> > > called once during probe. To fix this issue, we move dw_pcie_msi_init()
> > > to dw_pcie_setup_rc().
> >
> > This patch looks fine, but I don't think the commit log tells the
> > whole story.
> >
> > Prior to 59fbab1ae40e, it looks like the only dwc-based drivers with
> > resume functions were dra7xx, imx6, intel-gw, and tegra [1].
> >
> > Only tegra called dw_pcie_msi_init() in the resume path, and I do
> > think 59fbab1ae40e broke MSI after resume because it removed the
> > dw_pcie_msi_init() call from tegra_pcie_enable_msi_interrupts().
> >
> > I'm not convinced this patch fixes it reliably, though. The call
> > chain looks like this:
> >
> > tegra_pcie_dw_resume_noirq
> > tegra_pcie_dw_start_link
> > if (dw_pcie_wait_for_link(pci))
> > dw_pcie_setup_rc
> >
> > dw_pcie_wait_for_link() returns 0 if the link is up, so we only call
> > dw_pcie_setup_rc() in the case where the link *didn't* come up. If
> > the link comes up nicely without retry, we won't call
> > dw_pcie_setup_rc() and hence won't call dw_pcie_msi_init().
>
> The v1 version patch was sent before commit 275e88b06a (PCI: tegra: Fix host
> link initialization"). At that time, the resume path looks like this:
>
> tegra_pcie_dw_resume_noirq
> tegra_pcie_dw_host_init
> tegra_pcie_prepare_host
> dw_pcie_setup_rc
>
> so after patch, dw_pcie_msi_init() will be called. But now it seems that
> the tegra version needs one more fix for the resume.
>
> So could I sent a new patch to update the commit-msg a bit?
This patch only touches the dwc core, and the commit log says
generically that it fixes MSI after resume, so one could assume that
it applies to all dwc-based drivers. But I don't think it's that
simple, so I'd like to know *which* drivers are fixed and which
commits are related. I don't see how 59fbab1ae40e breaks anything
except tegra.
> > Since then, exynos added a resume function. My guess is MSI never
> > worked after resume for dra7xx, exynos, imx6, and intel-gw because
> > they don't call dw_pcie_msi_init() in their resume functions.
> >
> > This patch looks like it should fix MSI after resume for exynos, imx6,
> > and intel-gw because they *do* call dw_pcie_setup_rc() from their
> > resume functions [2], and after this patch, dw_pcie_msi_init() will be
> > called from there.
> >
> > I suspect MSI after resume still doesn't work on dra7xx.
>
> I checked the dra7xx history, I'm afraid that the resume never works
> from the beginning if the host lost power during suspend, I guess the
> platform never power off the host but only the phy?
Sounds like that would make sense.
> > [1] git grep -A20 -e "static.*resume_noirq"
> > 59fbab1ae40e^:drivers/pci/controller/dwc
> > [2] git grep -A20 -e "static.*resume_noirq" drivers/pci/controller/dwc
> >
> > > Fixes: 59fbab1ae40e ("PCI: dwc: Move dw_pcie_msi_init() into core")
> > > Reviewed-by: Rob Herring
> > > Signed-off-by: Jisheng Zhang
> > > ---
> > > Since v1:
> > > - collect Reviewed-by tag
> > >
> > > drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++-
> > > 1 file changed, 2 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c
> > > b/drivers/pci/controller/dwc/pcie-designware-host.c
> > > index 7e55b2b66182..e6c274f4485c 100644
> > > --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> > > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> > > @@ -400,7 +400,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> > > }
> > >
> > > dw_pcie_setup_rc(pp);
> > > - dw_pcie_msi_init(pp);
> > >
> > > if (!dw_pcie_link_up(pci) && pci->ops && pci->ops->start_link) {
> > > ret = pci->ops->start_link(pci);
> > > @@ -551,6 +550,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
> > > }
> > > }
> > >
> > > + dw_pcie_msi_init(pp);
> > > +
> > > /* Setup RC BARs */
> > > dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 0x0004);
> > > dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_1, 0x);
> > > --
> > > 2.30.1
> > >
>
Re: [PATCH RESEND] PCI: dwc: Fix MSI not work after resume
Hi
On Mon, 22 Mar 2021 20:24:41 -0500 Bjorn Helgaas wrote:
>
> [+cc Kishon, Richard, Lucas, Dilip]
>
> On Mon, Mar 01, 2021 at 11:10:31AM +0800, Jisheng Zhang wrote:
> > After we move dw_pcie_msi_init() into core -- dw_pcie_host_init(), the
> > MSI stops working after resume. Because dw_pcie_host_init() is only
> > called once during probe. To fix this issue, we move dw_pcie_msi_init()
> > to dw_pcie_setup_rc().
>
> This patch looks fine, but I don't think the commit log tells the
> whole story.
>
> Prior to 59fbab1ae40e, it looks like the only dwc-based drivers with
> resume functions were dra7xx, imx6, intel-gw, and tegra [1].
>
> Only tegra called dw_pcie_msi_init() in the resume path, and I do
> think 59fbab1ae40e broke MSI after resume because it removed the
> dw_pcie_msi_init() call from tegra_pcie_enable_msi_interrupts().
>
> I'm not convinced this patch fixes it reliably, though. The call
> chain looks like this:
>
> tegra_pcie_dw_resume_noirq
> tegra_pcie_dw_start_link
> if (dw_pcie_wait_for_link(pci))
> dw_pcie_setup_rc
>
> dw_pcie_wait_for_link() returns 0 if the link is up, so we only call
> dw_pcie_setup_rc() in the case where the link *didn't* come up. If
> the link comes up nicely without retry, we won't call
> dw_pcie_setup_rc() and hence won't call dw_pcie_msi_init().
The v1 version patch was sent before commit 275e88b06a (PCI: tegra: Fix host
link initialization"). At that time, the resume path looks like this:
tegra_pcie_dw_resume_noirq
tegra_pcie_dw_host_init
tegra_pcie_prepare_host
dw_pcie_setup_rc
so after patch, dw_pcie_msi_init() will be called. But now it seems that
the tegra version needs one more fix for the resume.
So could I sent a new patch to update the commit-msg a bit?
>
> Since then, exynos added a resume function. My guess is MSI never
> worked after resume for dra7xx, exynos, imx6, and intel-gw because
> they don't call dw_pcie_msi_init() in their resume functions.
>
> This patch looks like it should fix MSI after resume for exynos, imx6,
> and intel-gw because they *do* call dw_pcie_setup_rc() from their
> resume functions [2], and after this patch, dw_pcie_msi_init() will be
> called from there.
>
> I suspect MSI after resume still doesn't work on dra7xx.
I checked the dra7xx history, I'm afraid that the resume never works
from the beginning if the host lost power during suspend, I guess the
platform never power off the host but only the phy?
>
> [1] git grep -A20 -e "static.*resume_noirq"
> 59fbab1ae40e^:drivers/pci/controller/dwc
> [2] git grep -A20 -e "static.*resume_noirq" drivers/pci/controller/dwc
>
> > Fixes: 59fbab1ae40e ("PCI: dwc: Move dw_pcie_msi_init() into core")
> > Reviewed-by: Rob Herring
> > Signed-off-by: Jisheng Zhang
> > ---
> > Since v1:
> > - collect Reviewed-by tag
> >
> > drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++-
> > 1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c
> > b/drivers/pci/controller/dwc/pcie-designware-host.c
> > index 7e55b2b66182..e6c274f4485c 100644
> > --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> > +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> > @@ -400,7 +400,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> > }
> >
> > dw_pcie_setup_rc(pp);
> > - dw_pcie_msi_init(pp);
> >
> > if (!dw_pcie_link_up(pci) && pci->ops && pci->ops->start_link) {
> > ret = pci->ops->start_link(pci);
> > @@ -551,6 +550,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
> > }
> > }
> >
> > + dw_pcie_msi_init(pp);
> > +
> > /* Setup RC BARs */
> > dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 0x0004);
> > dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_1, 0x);
> > --
> > 2.30.1
> >
Re: [PATCH RESEND] PCI: dwc: Fix MSI not work after resume
[+cc Kishon, Richard, Lucas, Dilip]
On Mon, Mar 01, 2021 at 11:10:31AM +0800, Jisheng Zhang wrote:
> After we move dw_pcie_msi_init() into core -- dw_pcie_host_init(), the
> MSI stops working after resume. Because dw_pcie_host_init() is only
> called once during probe. To fix this issue, we move dw_pcie_msi_init()
> to dw_pcie_setup_rc().
This patch looks fine, but I don't think the commit log tells the
whole story.
Prior to 59fbab1ae40e, it looks like the only dwc-based drivers with
resume functions were dra7xx, imx6, intel-gw, and tegra [1].
Only tegra called dw_pcie_msi_init() in the resume path, and I do
think 59fbab1ae40e broke MSI after resume because it removed the
dw_pcie_msi_init() call from tegra_pcie_enable_msi_interrupts().
I'm not convinced this patch fixes it reliably, though. The call
chain looks like this:
tegra_pcie_dw_resume_noirq
tegra_pcie_dw_start_link
if (dw_pcie_wait_for_link(pci))
dw_pcie_setup_rc
dw_pcie_wait_for_link() returns 0 if the link is up, so we only call
dw_pcie_setup_rc() in the case where the link *didn't* come up. If
the link comes up nicely without retry, we won't call
dw_pcie_setup_rc() and hence won't call dw_pcie_msi_init().
Since then, exynos added a resume function. My guess is MSI never
worked after resume for dra7xx, exynos, imx6, and intel-gw because
they don't call dw_pcie_msi_init() in their resume functions.
This patch looks like it should fix MSI after resume for exynos, imx6,
and intel-gw because they *do* call dw_pcie_setup_rc() from their
resume functions [2], and after this patch, dw_pcie_msi_init() will be
called from there.
I suspect MSI after resume still doesn't work on dra7xx.
[1] git grep -A20 -e "static.*resume_noirq"
59fbab1ae40e^:drivers/pci/controller/dwc
[2] git grep -A20 -e "static.*resume_noirq" drivers/pci/controller/dwc
> Fixes: 59fbab1ae40e ("PCI: dwc: Move dw_pcie_msi_init() into core")
> Reviewed-by: Rob Herring
> Signed-off-by: Jisheng Zhang
> ---
> Since v1:
> - collect Reviewed-by tag
>
> drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c
> b/drivers/pci/controller/dwc/pcie-designware-host.c
> index 7e55b2b66182..e6c274f4485c 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> @@ -400,7 +400,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> }
>
> dw_pcie_setup_rc(pp);
> - dw_pcie_msi_init(pp);
>
> if (!dw_pcie_link_up(pci) && pci->ops && pci->ops->start_link) {
> ret = pci->ops->start_link(pci);
> @@ -551,6 +550,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
> }
> }
>
> + dw_pcie_msi_init(pp);
> +
> /* Setup RC BARs */
> dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 0x0004);
> dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_1, 0x);
> --
> 2.30.1
>
Re: [PATCH RESEND] PCI: dwc: Fix MSI not work after resume
On Mon, Mar 01, 2021 at 11:10:31AM +0800, Jisheng Zhang wrote:
> After we move dw_pcie_msi_init() into core -- dw_pcie_host_init(), the
> MSI stops working after resume. Because dw_pcie_host_init() is only
> called once during probe. To fix this issue, we move dw_pcie_msi_init()
> to dw_pcie_setup_rc().
>
> Fixes: 59fbab1ae40e ("PCI: dwc: Move dw_pcie_msi_init() into core")
> Reviewed-by: Rob Herring
> Signed-off-by: Jisheng Zhang
Oops, sorry, looks like this fell through the cracks. Since
59fbab1ae40e appeared in v5.11, I think we should add:
Cc: sta...@vger.kernel.org# v5.11+
I'm sure Lorenzo will add it when applying, so no need to repost just
for that.
> ---
> Since v1:
> - collect Reviewed-by tag
>
> drivers/pci/controller/dwc/pcie-designware-host.c | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/pci/controller/dwc/pcie-designware-host.c
> b/drivers/pci/controller/dwc/pcie-designware-host.c
> index 7e55b2b66182..e6c274f4485c 100644
> --- a/drivers/pci/controller/dwc/pcie-designware-host.c
> +++ b/drivers/pci/controller/dwc/pcie-designware-host.c
> @@ -400,7 +400,6 @@ int dw_pcie_host_init(struct pcie_port *pp)
> }
>
> dw_pcie_setup_rc(pp);
> - dw_pcie_msi_init(pp);
>
> if (!dw_pcie_link_up(pci) && pci->ops && pci->ops->start_link) {
> ret = pci->ops->start_link(pci);
> @@ -551,6 +550,8 @@ void dw_pcie_setup_rc(struct pcie_port *pp)
> }
> }
>
> + dw_pcie_msi_init(pp);
> +
> /* Setup RC BARs */
> dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_0, 0x0004);
> dw_pcie_writel_dbi(pci, PCI_BASE_ADDRESS_1, 0x);
> --
> 2.30.1
>
