Re: Ath10k reboot regression with v5.7-rc1 on dragonboard 845c
On 2020-04-30 15:57, Govind Singh wrote: -Original Message- From: Rakesh Pillai Sent: Thursday, April 30, 2020 12:26 PM To: manivannan.sadhasi...@linaro.org; Govind Singh Cc: Amit Pundir ; John Stultz ; ath10k@lists.infradead.org; bjorn.anders...@linaro.org Subject: RE: [EXT] Re: Ath10k reboot regression with v5.7-rc1 on dragonboard 845c Hi Mani, Reply inline Thanks, Rakesh Pillai -Original Message- From: ath10k On Behalf Of Manivannan Sadhasivam Sent: Thursday, April 30, 2020 11:10 AM To: Govind Singh Cc: Amit Pundir ; John Stultz ; ath10k@lists.infradead.org; bjorn.anders...@linaro.org Subject: [EXT] Re: Ath10k reboot regression with v5.7-rc1 on dragonboard 845c Hi Govind, On Thu, Apr 30, 2020 at 04:15:42AM +, Govind Singh wrote: > Hi Amit, > Seems del_server is being notified early due to qrtr-ns migration > from userspace to kernel prior remote(modem + wifi) actually went down. Sorry I don't get this. In-kernel NS just receives DEL_SERVER message from remote endpoint and it removes the entry for it and broadcasts to all observers. I think the issue is (as Bjorn suspected), previously we didn't catch the DEL_SERVER message from modem during shutdown/reboot but now we are able to do because NS is still running. Yes, with user space qrtr-ns DEL_SERVER is not revived as NS might have been terminated during reboot prior catching this from modem. > As per of del_server we are removing the MSA permission via SCM > call, but remote(wifi user pd in modem Q6) is still accessing the region. > This looks odd. Why should the remote send DEL_SERVER if it is still accessing the region? Or if that's the expected behavior, we shouldn't remove the MSA permission at this point in ath10k? Probably modem graceful shutdown is not happening in reboot case causing this issue as we don't see this issue in normal driver recovery due to FW error fatal, there also we have same code flow. Earlier we tried to register reboot notifier(devm_register_reboot_notifier) and skip the del_server for such cases, but we didn't posted patch as we never hit that case and its WAR. Probably we can have another qmi_ops->BYE to indicate this scenario if graceful shutdown is not happening or msa map/unmap can be moved in probe/remove as most of the targets except APQ/MSM8998 are using MSA carve-out region. As a part of ath10k driver unload, when we release the qmi handle, The ath10k qmi client receives a SERVER_EXIT (which has not been initiated from modem/wlan). In server_exit processing we unmap the MSA permissions. I am fixing this, to not handle the SERVER_EXIT if we have released the qmi handle. This is reboot case, your changes seems specific to rmmod case where qmi is indicating false del server due to qmi handle release. Thanks, Mani > BR, > Govind > > -Original Message- > From: ath10k On Behalf Of Amit Pundir > Sent: Wednesday, April 29, 2020 9:51 PM > To: ath10k@lists.infradead.org > Cc: bjorn.anders...@linaro.org; John Stultz > ; manivannan.sadhasi...@linaro.org > Subject: [EXT] Ath10k reboot regression with v5.7-rc1 on dragonboard > 845c > > Hi, > > I see a reboot regression with v5.7-rc1 on Dragonboard 845c > (wcn3990/ath10k_snoc) running AOSP. "reboot" or "reboot bootloader" > commands do not work as expected when the board is connected to WiFi AP. I see "ath10k_snoc 1880.wifi: firmware crashed"... dump in dmesg and board reboots into usb debug/crash mode. I do not see this reboot regression when the board is not connected to WiFi. > > It started with qrtr-ns migration from userspace to kernel which > landed in v5.7-rc1. I didn't run into this reboot issue when I was running qrtr-ns userspace tool. According to Mani, in-kernel qrtr-ns just live long enough to catch modem shutdown requests and advertise it to the modem, unlike the userspace tool. Which seem to be the case here. I further narrowed it down to ath10k_qmi_remove_msa_permission() > call in ath10k shutdown path. If I comment out that function call > then the reboot command works as expected. > > Any thoughts as to what might be going wrong? I do not understand qrtr/ath10k/qmi to see how msa permissions are getting mapped-unmapped here. I'd be happy to help debug things. > > Regards, > Amit Pundir > > ___ > ath10k mailing list > ath10k@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/ath10k ___ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k ___ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k
RE: Ath10k reboot regression with v5.7-rc1 on dragonboard 845c
Hi Mani, Reply inline Thanks, Rakesh Pillai > -Original Message- > From: ath10k On Behalf Of > Manivannan Sadhasivam > Sent: Thursday, April 30, 2020 11:10 AM > To: Govind Singh > Cc: Amit Pundir ; John Stultz > ; ath10k@lists.infradead.org; > bjorn.anders...@linaro.org > Subject: [EXT] Re: Ath10k reboot regression with v5.7-rc1 on dragonboard > 845c > > Hi Govind, > > On Thu, Apr 30, 2020 at 04:15:42AM +, Govind Singh wrote: > > Hi Amit, > > Seems del_server is being notified early due to qrtr-ns migration from > userspace to kernel prior remote(modem + wifi) actually went down. > > Sorry I don't get this. In-kernel NS just receives DEL_SERVER message from > remote endpoint and it removes the entry for it and broadcasts to all > observers. > > I think the issue is (as Bjorn suspected), previously we didn't catch the > DEL_SERVER message from modem during shutdown/reboot but now we > are able to do > because NS is still running. > > > As per of del_server we are removing the MSA permission via SCM call, but > remote(wifi user pd in modem Q6) is still accessing the region. > > > > This looks odd. Why should the remote send DEL_SERVER if it is still accessing > the region? Or if that's the expected behaviour, we shouldn't remove the > MSA > permission at this point in ath10k? As a part of ath10k driver unload, when we release the qmi handle, The ath10k qmi client receives a SERVER_EXIT (which has not been initiated from modem/wlan). In server_exit processing we unmap the MSA permissions. I am fixing this, to not handle the SERVER_EXIT if we have released the qmi handle. > > Thanks, > Mani > > > BR, > > Govind > > > > -Original Message- > > From: ath10k On Behalf Of Amit > Pundir > > Sent: Wednesday, April 29, 2020 9:51 PM > > To: ath10k@lists.infradead.org > > Cc: bjorn.anders...@linaro.org; John Stultz ; > manivannan.sadhasi...@linaro.org > > Subject: [EXT] Ath10k reboot regression with v5.7-rc1 on dragonboard 845c > > > > Hi, > > > > I see a reboot regression with v5.7-rc1 on Dragonboard 845c > > (wcn3990/ath10k_snoc) running AOSP. "reboot" or "reboot bootloader" > > commands do not work as expected when the board is connected to WiFi > AP. I see "ath10k_snoc 1880.wifi: firmware crashed"... dump in dmesg > and board reboots into usb debug/crash mode. I do not see this reboot > regression when the board is not connected to WiFi. > > > > It started with qrtr-ns migration from userspace to kernel which landed in > v5.7-rc1. I didn't run into this reboot issue when I was running qrtr-ns > userspace tool. According to Mani, in-kernel qrtr-ns just live long enough to > catch modem shutdown requests and advertise it to the modem, unlike the > userspace tool. Which seem to be the case here. I further narrowed it down > to ath10k_qmi_remove_msa_permission() > > call in ath10k shutdown path. If I comment out that function call then the > reboot command works as expected. > > > > Any thoughts as to what might be going wrong? I do not understand > qrtr/ath10k/qmi to see how msa permissions are getting mapped-unmapped > here. I'd be happy to help debug things. > > > > Regards, > > Amit Pundir > > > > ___ > > ath10k mailing list > > ath10k@lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/ath10k > > ___ > ath10k mailing list > ath10k@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/ath10k ___ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k
Re: Ath10k reboot regression with v5.7-rc1 on dragonboard 845c
Hi Govind, On Thu, Apr 30, 2020 at 04:15:42AM +, Govind Singh wrote: > Hi Amit, > Seems del_server is being notified early due to qrtr-ns migration from > userspace to kernel prior remote(modem + wifi) actually went down. Sorry I don't get this. In-kernel NS just receives DEL_SERVER message from remote endpoint and it removes the entry for it and broadcasts to all observers. I think the issue is (as Bjorn suspected), previously we didn't catch the DEL_SERVER message from modem during shutdown/reboot but now we are able to do because NS is still running. > As per of del_server we are removing the MSA permission via SCM call, but > remote(wifi user pd in modem Q6) is still accessing the region. > This looks odd. Why should the remote send DEL_SERVER if it is still accessing the region? Or if that's the expected behaviour, we shouldn't remove the MSA permission at this point in ath10k? Thanks, Mani > BR, > Govind > > -Original Message- > From: ath10k On Behalf Of Amit Pundir > Sent: Wednesday, April 29, 2020 9:51 PM > To: ath10k@lists.infradead.org > Cc: bjorn.anders...@linaro.org; John Stultz ; > manivannan.sadhasi...@linaro.org > Subject: [EXT] Ath10k reboot regression with v5.7-rc1 on dragonboard 845c > > Hi, > > I see a reboot regression with v5.7-rc1 on Dragonboard 845c > (wcn3990/ath10k_snoc) running AOSP. "reboot" or "reboot bootloader" > commands do not work as expected when the board is connected to WiFi AP. I > see "ath10k_snoc 1880.wifi: firmware crashed"... dump in dmesg and board > reboots into usb debug/crash mode. I do not see this reboot regression when > the board is not connected to WiFi. > > It started with qrtr-ns migration from userspace to kernel which landed in > v5.7-rc1. I didn't run into this reboot issue when I was running qrtr-ns > userspace tool. According to Mani, in-kernel qrtr-ns just live long enough to > catch modem shutdown requests and advertise it to the modem, unlike the > userspace tool. Which seem to be the case here. I further narrowed it down to > ath10k_qmi_remove_msa_permission() > call in ath10k shutdown path. If I comment out that function call then the > reboot command works as expected. > > Any thoughts as to what might be going wrong? I do not understand > qrtr/ath10k/qmi to see how msa permissions are getting mapped-unmapped here. > I'd be happy to help debug things. > > Regards, > Amit Pundir > > ___ > ath10k mailing list > ath10k@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/ath10k ___ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k
RE: Ath10k reboot regression with v5.7-rc1 on dragonboard 845c
Hi Amit, Seems del_server is being notified early due to qrtr-ns migration from userspace to kernel prior remote(modem + wifi) actually went down. As per of del_server we are removing the MSA permission via SCM call, but remote(wifi user pd in modem Q6) is still accessing the region. BR, Govind -Original Message- From: ath10k On Behalf Of Amit Pundir Sent: Wednesday, April 29, 2020 9:51 PM To: ath10k@lists.infradead.org Cc: bjorn.anders...@linaro.org; John Stultz ; manivannan.sadhasi...@linaro.org Subject: [EXT] Ath10k reboot regression with v5.7-rc1 on dragonboard 845c Hi, I see a reboot regression with v5.7-rc1 on Dragonboard 845c (wcn3990/ath10k_snoc) running AOSP. "reboot" or "reboot bootloader" commands do not work as expected when the board is connected to WiFi AP. I see "ath10k_snoc 1880.wifi: firmware crashed"... dump in dmesg and board reboots into usb debug/crash mode. I do not see this reboot regression when the board is not connected to WiFi. It started with qrtr-ns migration from userspace to kernel which landed in v5.7-rc1. I didn't run into this reboot issue when I was running qrtr-ns userspace tool. According to Mani, in-kernel qrtr-ns just live long enough to catch modem shutdown requests and advertise it to the modem, unlike the userspace tool. Which seem to be the case here. I further narrowed it down to ath10k_qmi_remove_msa_permission() call in ath10k shutdown path. If I comment out that function call then the reboot command works as expected. Any thoughts as to what might be going wrong? I do not understand qrtr/ath10k/qmi to see how msa permissions are getting mapped-unmapped here. I'd be happy to help debug things. Regards, Amit Pundir ___ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k ___ ath10k mailing list ath10k@lists.infradead.org http://lists.infradead.org/mailman/listinfo/ath10k
