Re: msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up
On Wed, Jun 12, 2019 at 06:20:48PM +0200, Niklas Cassel wrote: > > Can you still reproduce the reboot? --- a/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi +++ b/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi @@ -447,12 +447,12 @@ }; pcie@608000 { - status = "okay"; + status = "disabled"; perst-gpio = <&msmgpio 130 GPIO_ACTIVE_LOW>; }; pcie@61 { - status = "okay"; + status = "disabled"; perst-gpio = <&msmgpio 114 GPIO_ACTIVE_LOW>; }; }; disabling the two other pci controllers (and leaving your previous patch applied), made the issue more frequent: i was never able to boot to user space three times in a row without experiencing a crash, previously it was more sporadic. On the other hand, now i can reproduce the issue more easily this way. -- bye, p.
Re: msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up
On 10/06/2019 15:44, Paolo Pisati wrote: > From time to time, my apq8096-db820c fails to boot to userspace, reset back to > fastboot and locks up: to easily reproduce the issue, i'm boot looping using a > cron job with a 1 min reboot entry on the board while leaving a "while 1; do > fastboot boot boot.img; done" on the host pc. > > The issue is present in mainline up to 5.2-rc4, using defconfig and: > > CONFIG_SCSI_UFS_QCOM=y > CONFIG_PHY_QCOM_QMP=y > CONFIG_PHY_QCOM_UFS=y > > but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or > mainline), where it's even easier to reproduce (e.g. takes way less reboots to > trigger it). > > These are the last lines printed out: > ... > [7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy > [7.448058] qcom-qmp-phy 741.phy: Registered Qcom-QMP phy > [7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource > [7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out > [7.538596] phy phy-34000.phy.0: phy init failed --> -110 > [7.550891] qcom-pcie: probe of 60.pcie failed with error -110 > [7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found, > using dummy regulator > > Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic > S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301 > S - IMAGE_VARIANT_STRING=M8996LAB > S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68 > S - Boot Interface: UFS > S - Secure Boot: Off > ... > > Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/ > > Any idea what is going on? Am i doing something wrong? Hmmm... this might be related to: https://patchwork.kernel.org/patch/10976217/ Regards.
Re: msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up
On Wed, Jun 12, 2019 at 04:09:11PM +0200, Paolo Pisati wrote: > On Wed, Jun 12, 2019 at 03:17:35PM +0200, Niklas Cassel wrote: > > > diff --git a/drivers/phy/qualcomm/phy-qcom-qmp.c > > > b/drivers/phy/qualcomm/phy-qcom-qmp.c > > > index cd91b4179b10..22352e3b0ec5 100644 > > > --- a/drivers/phy/qualcomm/phy-qcom-qmp.c > > > +++ b/drivers/phy/qualcomm/phy-qcom-qmp.c > > > @@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy) > > > > > > ret = qcom_qmp_phy_com_init(qphy); > > > if (ret) > > > - return ret; > > > + goto err_lane_rst; > > > > > > if (cfg->has_lane_rst) { > > > ret = reset_control_deassert(qphy->lane_rst); > > Hi Niklas, > unfortunately, it didn't help - i added a printk, to highlight when it failed: > > --- a/drivers/phy/qualcomm/phy-qcom-qmp.c > +++ b/drivers/phy/qualcomm/phy-qcom-qmp.c > @@ -1489,8 +1489,10 @@ static int qcom_qmp_phy_enable(struct phy *phy) > } > > ret = qcom_qmp_phy_com_init(qphy); > - if (ret) > - return ret; > + if (ret) { > + dev_err(qmp->dev, "qphy initialization failed\n"); > + goto err_lane_rst; > + } > > if (cfg->has_lane_rst) { > ret = reset_control_deassert(qphy->lane_rst); > > After several reboots i was able to trigger the phy init failure again: > > ... > [2.223999] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy > [2.224956] qcom-qmp-phy 741.phy: Registered Qcom-QMP phy > [2.228798] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource > [2.237271] qcom-qmp-phy 34000.phy: phy common block init timed-out > [2.240315] qcom-qmp-phy 34000.phy: qphy initialization failed > ... I still think that the above patch is correct, even though it didn't fix your problem. If you try to disable the two other PCIe controllers: diff --git a/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi b/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi index 943f69912074..95900fe99f89 100644 --- a/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi +++ b/arch/arm64/boot/dts/qcom/apq8096-db820c.dtsi @@ -455,12 +455,12 @@ }; pcie@608000 { - status = "okay"; + status = "disabled"; perst-gpio = <&msmgpio 130 GPIO_ACTIVE_LOW>; }; pcie@61 { - status = "okay"; + status = "disabled"; perst-gpio = <&msmgpio 114 GPIO_ACTIVE_LOW>; }; }; Can you still reproduce the reboot? Kind regards, Niklas > > these are the last lines printed, before rebooting in fastboot and > locking up there (as before[*]). > > So, as far as i understand there are two distinct problems: > > 1) sometimes, qcom-qmp-phy fails to initialize > > 2) and when that happens, the failure is fatal and it led to a reboot & lockup > in fastboot > > 1: https://pastebin.ubuntu.com/p/rtjVrD3yzk/ > -- > bye, > p.
Re: msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up
On Wed, Jun 12, 2019 at 03:17:35PM +0200, Niklas Cassel wrote: > > diff --git a/drivers/phy/qualcomm/phy-qcom-qmp.c > > b/drivers/phy/qualcomm/phy-qcom-qmp.c > > index cd91b4179b10..22352e3b0ec5 100644 > > --- a/drivers/phy/qualcomm/phy-qcom-qmp.c > > +++ b/drivers/phy/qualcomm/phy-qcom-qmp.c > > @@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy) > > > > ret = qcom_qmp_phy_com_init(qphy); > > if (ret) > > - return ret; > > + goto err_lane_rst; > > > > if (cfg->has_lane_rst) { > > ret = reset_control_deassert(qphy->lane_rst); Hi Niklas, unfortunately, it didn't help - i added a printk, to highlight when it failed: --- a/drivers/phy/qualcomm/phy-qcom-qmp.c +++ b/drivers/phy/qualcomm/phy-qcom-qmp.c @@ -1489,8 +1489,10 @@ static int qcom_qmp_phy_enable(struct phy *phy) } ret = qcom_qmp_phy_com_init(qphy); - if (ret) - return ret; + if (ret) { + dev_err(qmp->dev, "qphy initialization failed\n"); + goto err_lane_rst; + } if (cfg->has_lane_rst) { ret = reset_control_deassert(qphy->lane_rst); After several reboots i was able to trigger the phy init failure again: ... [2.223999] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy [2.224956] qcom-qmp-phy 741.phy: Registered Qcom-QMP phy [2.228798] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource [2.237271] qcom-qmp-phy 34000.phy: phy common block init timed-out [2.240315] qcom-qmp-phy 34000.phy: qphy initialization failed ... these are the last lines printed, before rebooting in fastboot and locking up there (as before[*]). So, as far as i understand there are two distinct problems: 1) sometimes, qcom-qmp-phy fails to initialize 2) and when that happens, the failure is fatal and it led to a reboot & lockup in fastboot 1: https://pastebin.ubuntu.com/p/rtjVrD3yzk/ -- bye, p.
Re: msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up
On Tue, Jun 11, 2019 at 07:12:25PM +0200, Niklas Cassel wrote: > On Mon, Jun 10, 2019 at 03:44:01PM +0200, Paolo Pisati wrote: > > From time to time, my apq8096-db820c fails to boot to userspace, reset back > > to > > fastboot and locks up: to easily reproduce the issue, i'm boot looping > > using a > > cron job with a 1 min reboot entry on the board while leaving a "while 1; do > > fastboot boot boot.img; done" on the host pc. > > > > The issue is present in mainline up to 5.2-rc4, using defconfig and: > > > > CONFIG_SCSI_UFS_QCOM=y > > CONFIG_PHY_QCOM_QMP=y > > CONFIG_PHY_QCOM_UFS=y > > > > but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or > > mainline), where it's even easier to reproduce (e.g. takes way less reboots > > to > > trigger it). > > Hello Paolo, > > I have a guess of what is going on. > db820c has 3 PCIe controllers, > that shares a singe QMP block (that has clocks, regulators, and resets). > The QMP block has 3 PCIe PHYs, that have their own clocks and resets. > > > > > These are the last lines printed out: > > ... > > [7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy > > [7.448058] qcom-qmp-phy 741.phy: Registered Qcom-QMP phy > > [7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource > > [7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out > > ^^ here the phy_init() called from pcie-qcom.c > which ends up to a call to qcom_qmp_phy_enable() > > which has this code: > > ret = qcom_qmp_phy_com_init(qphy); > if (ret) > return ret; > > qcom_qmp_phy_com_init() has this code: > > if (qmp->init_count++) { > mutex_unlock(&qmp->phy_mutex); > return 0; > } > > qcom_qmp_phy_com_init() later fails, > since the common block init time out, so the qmp driver > disables clocks, asserts reset, and disables regulators > > > > [7.538596] phy phy-34000.phy.0: phy init failed --> -110 > > [7.550891] qcom-pcie: probe of 60.pcie failed with error -110 > > ^^ here the first PCIe controller instance fails to probe > > > [7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not > > found, > > using dummy regulator > > ^^ here the second PCIe controller is probed. > > it will call phy_init() > > which will again call qcom_qmp_phy_enable() which will call > qcom_qmp_phy_com_init() > > where this code: > > if (qmp->init_count++) { > mutex_unlock(&qmp->phy_mutex); > return 0; > } > > now will return 0, > > so clocks will never be enabled, resets never deasserted, regulators > never enabled. > > since qcom_qmp_phy_com_init() returns success in this case, > qcom_qmp_phy_enable() will try to continue with the init, > and writes to disabled hardware is usually not a good idea. > > I think the proper fix for this is: > > diff --git a/drivers/phy/qualcomm/phy-qcom-qmp.c > b/drivers/phy/qualcomm/phy-qcom-qmp.c > index cd91b4179b10..22352e3b0ec5 100644 > --- a/drivers/phy/qualcomm/phy-qcom-qmp.c > +++ b/drivers/phy/qualcomm/phy-qcom-qmp.c > @@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy) > > ret = qcom_qmp_phy_com_init(qphy); > if (ret) > - return ret; > + goto err_lane_rst; > > if (cfg->has_lane_rst) { > ret = reset_control_deassert(qphy->lane_rst); > > > > Kind regards, > Niklas > > > > > Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic > > S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301 > > S - IMAGE_VARIANT_STRING=M8996LAB > > S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68 > > S - Boot Interface: UFS > > S - Secure Boot: Off > > ... > > > > Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/ > > > > Any idea what is going on? Am i doing something wrong? > > -- > > bye, > > p. Adding Vivek.
Re: msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up
On Mon, Jun 10, 2019 at 03:44:01PM +0200, Paolo Pisati wrote: > From time to time, my apq8096-db820c fails to boot to userspace, reset back to > fastboot and locks up: to easily reproduce the issue, i'm boot looping using a > cron job with a 1 min reboot entry on the board while leaving a "while 1; do > fastboot boot boot.img; done" on the host pc. > > The issue is present in mainline up to 5.2-rc4, using defconfig and: > > CONFIG_SCSI_UFS_QCOM=y > CONFIG_PHY_QCOM_QMP=y > CONFIG_PHY_QCOM_UFS=y > > but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or > mainline), where it's even easier to reproduce (e.g. takes way less reboots to > trigger it). Hello Paolo, I have a guess of what is going on. db820c has 3 PCIe controllers, that shares a singe QMP block (that has clocks, regulators, and resets). The QMP block has 3 PCIe PHYs, that have their own clocks and resets. > > These are the last lines printed out: > ... > [7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy > [7.448058] qcom-qmp-phy 741.phy: Registered Qcom-QMP phy > [7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource > [7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out ^^ here the phy_init() called from pcie-qcom.c which ends up to a call to qcom_qmp_phy_enable() which has this code: ret = qcom_qmp_phy_com_init(qphy); if (ret) return ret; qcom_qmp_phy_com_init() has this code: if (qmp->init_count++) { mutex_unlock(&qmp->phy_mutex); return 0; } qcom_qmp_phy_com_init() later fails, since the common block init time out, so the qmp driver disables clocks, asserts reset, and disables regulators > [7.538596] phy phy-34000.phy.0: phy init failed --> -110 > [7.550891] qcom-pcie: probe of 60.pcie failed with error -110 ^^ here the first PCIe controller instance fails to probe > [7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found, > using dummy regulator ^^ here the second PCIe controller is probed. it will call phy_init() which will again call qcom_qmp_phy_enable() which will call qcom_qmp_phy_com_init() where this code: if (qmp->init_count++) { mutex_unlock(&qmp->phy_mutex); return 0; } now will return 0, so clocks will never be enabled, resets never deasserted, regulators never enabled. since qcom_qmp_phy_com_init() returns success in this case, qcom_qmp_phy_enable() will try to continue with the init, and writes to disabled hardware is usually not a good idea. I think the proper fix for this is: diff --git a/drivers/phy/qualcomm/phy-qcom-qmp.c b/drivers/phy/qualcomm/phy-qcom-qmp.c index cd91b4179b10..22352e3b0ec5 100644 --- a/drivers/phy/qualcomm/phy-qcom-qmp.c +++ b/drivers/phy/qualcomm/phy-qcom-qmp.c @@ -1490,7 +1490,7 @@ static int qcom_qmp_phy_enable(struct phy *phy) ret = qcom_qmp_phy_com_init(qphy); if (ret) - return ret; + goto err_lane_rst; if (cfg->has_lane_rst) { ret = reset_control_deassert(qphy->lane_rst); Kind regards, Niklas > > Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic > S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301 > S - IMAGE_VARIANT_STRING=M8996LAB > S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68 > S - Boot Interface: UFS > S - Secure Boot: Off > ... > > Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/ > > Any idea what is going on? Am i doing something wrong? > -- > bye, > p.
msm8996: qcom-qmp: apq8096-db820c fails to boot, reset back to fastboot and locks up
>From time to time, my apq8096-db820c fails to boot to userspace, reset back to fastboot and locks up: to easily reproduce the issue, i'm boot looping using a cron job with a 1 min reboot entry on the board while leaving a "while 1; do fastboot boot boot.img; done" on the host pc. The issue is present in mainline up to 5.2-rc4, using defconfig and: CONFIG_SCSI_UFS_QCOM=y CONFIG_PHY_QCOM_QMP=y CONFIG_PHY_QCOM_UFS=y but was present in previous releases too (e.g. 4.14., 4.19, etc qcom-lt or mainline), where it's even easier to reproduce (e.g. takes way less reboots to trigger it). These are the last lines printed out: ... [7.407209] qcom-qmp-phy 34000.phy: Registered Qcom-QMP phy [7.448058] qcom-qmp-phy 741.phy: Registered Qcom-QMP phy [7.461859] ufs_qcom_phy_qmp_14nm 627000.phy: invalid resource [7.535434] qcom-qmp-phy 34000.phy: phy common block init timed-out [7.538596] phy phy-34000.phy.0: phy init failed --> -110 [7.550891] qcom-pcie: probe of 60.pcie failed with error -110 [7.619008] qcom-pcie 608000.pcie: 608000.pcie supply vddpe-3v3 not found, using dummy regulator Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic S - QC_IMAGE_VERSION_STRING=BOOT.XF.1.0-00301 S - IMAGE_VARIANT_STRING=M8996LAB S - OEM_IMAGE_VERSION_STRING=crm-ubuntu68 S - Boot Interface: UFS S - Secure Boot: Off ... Full boot here: https://pastebin.ubuntu.com/p/rtjVrD3yzk/ Any idea what is going on? Am i doing something wrong? -- bye, p.