Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Hey, > > But there is no 100% fix. Certainly we can and should improve the > probing and I'm happy to test that. But with the PPP code in place, > there's always going to be a code path where things didn't work > out quite right with timing, or some error condition was triggered > or whatever that will fall through. > Recovering some previous discussions in this email thread, I think that updating mm_base_modem_organize_ports() so that it ends up with no valid data ports flagged would probably the cleanest way to support this usecase, if we want to do that. The method would fail with "Failed to find a data port in the modem", in the very same way as QMI or MBIM modems would fail if only the control port is detected. I'm not a real fan of the fix, but we could do this by adding a new udev tag as Dan suggested, e.g. named ID_MM_PORT_TYPE_AT_NO_PPP, so that we can flag a TTY that must never be used for data (and you could apply this tag to all your TTYs, effectively making the PPP option never used). This is as dynamic as it can get; if you're happy with ModemManager not exposing a modem object in DBus upon reaching that situation, then let's add that, and it would save you from needing to patch the PPP removal everywhere. What do you think? > I don't have the luxury for waiting for improvement here, nor can > I risk putting code in the field which might do this, no matter > how tiny the chance, hence the removal of the PPP code entirely as > a config option. I'm certainly not advocating stripping such > support; that would be foolish. > Please make sure you pick the timing fix, so that the QMI ports are always detected; that is the real fix, because you will make sure the modem is exposed and usable, and you won't need any additional loop of port event emissions added anywhere. -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
On 1/18/22 11:04 AM, Dan Williams wrote: What's your desired error state when detection fails? Just mark the whole modem as failed, and you'd have some custom logic to twiddle modem power and/or reboot the machine? I don't know the ins and outs of the error mechanism, so you tell me - but what's there already seems sufficient. That is, the loop in the netfid scripts (as per my earlier patch). Failing that, our system will eventually power cycle the modem on no connectivity. This can happen already of course; I don't know what the current thinking on this is. As you say: "I'd just expose the modem as failed and let the admin/framework/whatever handle it by power-cycling the modem or machine. If we didn't expose it then it would be more complicated with timeouts or something on whatever is above MM." > > Yeah, clearly if we can fix or work around slow probing, that's > preferable. > But there is no 100% fix. Certainly we can and should improve the probing and I'm happy to test that. But with the PPP code in place, there's always going to be a code path where things didn't work out quite right with timing, or some error condition was triggered or whatever that will fall through. I don't have the luxury for waiting for improvement here, nor can I risk putting code in the field which might do this, no matter how tiny the chance, hence the removal of the PPP code entirely as a config option. I'm certainly not advocating stripping such support; that would be foolish.
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
On Tue, 2022-01-18 at 09:57 +0100, Aleksander Morgado wrote: > Hey Dan! > > > > > > If MM detects a single TTY port, it's going to default to use > > > > > PPP. > > > > > It's not a fallback to PPP, it's using whatever it has for > > > > > data > > > > > connection, if PPP is the only way forward, PPP will be. > > > > > > > > I don't want PPP, ever. I don't know how many times I can > > > > repeat > > > > that. Please stop saying this and arguing with me about it. > > > > > > > > > > I know you don't want PPP in your setup, you have made that > > > clear. > > > I'm > > > not arguing with you about that. I get it. You don't want PPP. > > > > Yeah, MM has to work for a lot of people, some who want PPP and > > many > > who don't. But there's still enough PPP going around that it must > > continue to be supported for a while. > > > > Plus, I don't think this is something we can just optionally remove > from the build, just the PPP part. This is not like QMI or MBIM where > we can disable the full QMI or MBIM protocol, both control and data. > The equivalent thing would be to fully disable AT+PPP; but just > disabling PPP is not a good idea IMO, because we could end up leaving > not-connectable AT-only modems around (unless we want to allow such a > thing) Yeah, I don't think it should be removed entirely; it's really not a lot of code over-and-above AT itself. Which is still used for a *lot* of stuff. > > > > > > > Now, the way to fix that should be by making sure ModemManager > > > gets > > > notified of the QMI/NET port before it creates a modem object > > > only > > > with TTY ports. I understand that you don't like this approach, > > > but > > > ModemManager doesn't receive a fixed list of ports to work with, > > > as > > > many other modem management setups. MM tries to dynamically > > > adjust to > > > what it finds, and that is a core feature of ModemManager that is > > > not > > > going to change. > > > > In the past IIRC we've seen bad udev rules or scripts taking > > multiple > > seconds to evaluate or do something (I think usb_modeswitch was a > > problem here once upon a time) and delaying updates for specific > > ports > > of a device. Not sure if that's a problem here, or if the system is > > just slow. > > > > The problem is the system being slow plus needing mmcli to report the > kernel events when openwrt hotplug scripts are run, which make it > even > slower. This issue won't apply to udev based systems, I think it's > applicable only to openwrt. Ah, ok. > > > > > > > So again, and don't hate me for repeating this, if MM gets > > > notified > > > of > > > one single AT capable TTY port and nothing else, it will default > > > to > > > use PPP. If you don't like that, please patch your ModemManager > > > to > > > fit > > > your needs. Or try moving to uqmi, which requires you to specify > > > which > > > is the cdc-wdm port path explicitly, although that approach has > > > its > > > own problems, but maybe not the ones you're worried about. > > > > I haven't read the whole thread in detail, so forgive me if this > > was > > covered. Perhaps a way of tagging a modem/driver/whatever with > > "never > > use PPP" would be workable. Off by default of course. > > > > The problem with that is what you do when you have a modem exposed > with one single AT port, which is what ended up happening in Peter's > setup. Do we expose the modem in DBus but not connectable? Do we > expose it in Failed state? Do we just not expose the modem in DBus? > The problem here is not a QMI+AT modem that sometimes falls back to > PPP; the problem here is an AT-only No-QMI (because it failed > probing) > modem that falls back to PPP. I'd just expose the modem as failed and let the admin/framework/whatever handle it by power-cycling the modem or machine. If we didn't expose it then it would be more complicated with timeouts or something on whatever is above MM. > > My suggestion has always been to fix the probing logic so that the > QMI > ports are properly detected always; and once that is done and is > reliable, there would be no need to take care of whether PPP is used > or not, as QMI would always be preferred. PPP is only the fallback if > QMI is unusable for any reason. Trying to remove PPP without fixing > the probing logic leads nowhere. And once probing logic is fixed, > removing PPP would not be needed. Yeah, clearly if we can fix or work around slow probing, that's preferable. Dan
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
On Tue, 2022-01-18 at 05:35 -0500, Peter Naulls wrote: > On 1/17/22 2:06 PM, Dan Williams wrote: > own problems, but maybe not the ones you're worried about. > > > > I haven't read the whole thread in detail, so forgive me if this > > was > > covered. Perhaps a way of tagging a modem/driver/whatever with > > "never > > use PPP" would be workable. Off by default of course. > > If you like. Whatever is going to work for the general use case. > > However, no matter how you finely tune it, no matter how you choose > values, > there's always a chance of detection failure (which is OK), but > falling > back to PPP is never going to be acceptable, certainly not in what > I'm > building. I don't need to be debugging such situations in the field; > there's no justification for it. What's your desired error state when detection fails? Just mark the whole modem as failed, and you'd have some custom logic to twiddle modem power and/or reboot the machine? or something else? Dan > > I'd prefer to pursue removing PPP dependency entirely as I'm doing; > that saves space, and reduces complexity. Let me know if is going > to be something that works for you. I just wish I'd done this 6 > months ago; would have saved myself a lot of grief. > > > > >
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
On 1/17/22 2:06 PM, Dan Williams wrote: own problems, but maybe not the ones you're worried about. I haven't read the whole thread in detail, so forgive me if this was covered. Perhaps a way of tagging a modem/driver/whatever with "never use PPP" would be workable. Off by default of course. If you like. Whatever is going to work for the general use case. However, no matter how you finely tune it, no matter how you choose values, there's always a chance of detection failure (which is OK), but falling back to PPP is never going to be acceptable, certainly not in what I'm building. I don't need to be debugging such situations in the field; there's no justification for it. I'd prefer to pursue removing PPP dependency entirely as I'm doing; that saves space, and reduces complexity. Let me know if is going to be something that works for you. I just wish I'd done this 6 months ago; would have saved myself a lot of grief.
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Hey Dan! > > > > If MM detects a single TTY port, it's going to default to use > > > > PPP. > > > > It's not a fallback to PPP, it's using whatever it has for data > > > > connection, if PPP is the only way forward, PPP will be. > > > > > > I don't want PPP, ever. I don't know how many times I can repeat > > > that. Please stop saying this and arguing with me about it. > > > > > > > I know you don't want PPP in your setup, you have made that clear. > > I'm > > not arguing with you about that. I get it. You don't want PPP. > > Yeah, MM has to work for a lot of people, some who want PPP and many > who don't. But there's still enough PPP going around that it must > continue to be supported for a while. > Plus, I don't think this is something we can just optionally remove from the build, just the PPP part. This is not like QMI or MBIM where we can disable the full QMI or MBIM protocol, both control and data. The equivalent thing would be to fully disable AT+PPP; but just disabling PPP is not a good idea IMO, because we could end up leaving not-connectable AT-only modems around (unless we want to allow such a thing) > > > > Now, the way to fix that should be by making sure ModemManager gets > > notified of the QMI/NET port before it creates a modem object only > > with TTY ports. I understand that you don't like this approach, but > > ModemManager doesn't receive a fixed list of ports to work with, as > > many other modem management setups. MM tries to dynamically adjust to > > what it finds, and that is a core feature of ModemManager that is not > > going to change. > > In the past IIRC we've seen bad udev rules or scripts taking multiple > seconds to evaluate or do something (I think usb_modeswitch was a > problem here once upon a time) and delaying updates for specific ports > of a device. Not sure if that's a problem here, or if the system is > just slow. > The problem is the system being slow plus needing mmcli to report the kernel events when openwrt hotplug scripts are run, which make it even slower. This issue won't apply to udev based systems, I think it's applicable only to openwrt. > > > > So again, and don't hate me for repeating this, if MM gets notified > > of > > one single AT capable TTY port and nothing else, it will default to > > use PPP. If you don't like that, please patch your ModemManager to > > fit > > your needs. Or try moving to uqmi, which requires you to specify > > which > > is the cdc-wdm port path explicitly, although that approach has its > > own problems, but maybe not the ones you're worried about. > > I haven't read the whole thread in detail, so forgive me if this was > covered. Perhaps a way of tagging a modem/driver/whatever with "never > use PPP" would be workable. Off by default of course. > The problem with that is what you do when you have a modem exposed with one single AT port, which is what ended up happening in Peter's setup. Do we expose the modem in DBus but not connectable? Do we expose it in Failed state? Do we just not expose the modem in DBus? The problem here is not a QMI+AT modem that sometimes falls back to PPP; the problem here is an AT-only No-QMI (because it failed probing) modem that falls back to PPP. My suggestion has always been to fix the probing logic so that the QMI ports are properly detected always; and once that is done and is reliable, there would be no need to take care of whether PPP is used or not, as QMI would always be preferred. PPP is only the fallback if QMI is unusable for any reason. Trying to remove PPP without fixing the probing logic leads nowhere. And once probing logic is fixed, removing PPP would not be needed. -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
On Fri, 2022-01-14 at 16:29 +0100, Aleksander Morgado wrote: > Hey Peter, > > On Fri, Jan 14, 2022 at 4:14 PM Peter Naulls > wrote: > > > > On 1/14/22 10:09 AM, Aleksander Morgado wrote: > > logic is not going to change. > > > > > > If MM detects a single TTY port, it's going to default to use > > > PPP. > > > It's not a fallback to PPP, it's using whatever it has for data > > > connection, if PPP is the only way forward, PPP will be. > > > > I don't want PPP, ever. I don't know how many times I can repeat > > that. Please stop saying this and arguing with me about it. > > > > I know you don't want PPP in your setup, you have made that clear. > I'm > not arguing with you about that. I get it. You don't want PPP. Yeah, MM has to work for a lot of people, some who want PPP and many who don't. But there's still enough PPP going around that it must continue to be supported for a while. > > Now, the way to fix that should be by making sure ModemManager gets > notified of the QMI/NET port before it creates a modem object only > with TTY ports. I understand that you don't like this approach, but > ModemManager doesn't receive a fixed list of ports to work with, as > many other modem management setups. MM tries to dynamically adjust to > what it finds, and that is a core feature of ModemManager that is not > going to change. In the past IIRC we've seen bad udev rules or scripts taking multiple seconds to evaluate or do something (I think usb_modeswitch was a problem here once upon a time) and delaying updates for specific ports of a device. Not sure if that's a problem here, or if the system is just slow. > > So again, and don't hate me for repeating this, if MM gets notified > of > one single AT capable TTY port and nothing else, it will default to > use PPP. If you don't like that, please patch your ModemManager to > fit > your needs. Or try moving to uqmi, which requires you to specify > which > is the cdc-wdm port path explicitly, although that approach has its > own problems, but maybe not the ones you're worried about. I haven't read the whole thread in detail, so forgive me if this was covered. Perhaps a way of tagging a modem/driver/whatever with "never use PPP" would be workable. Off by default of course. Dan
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Hey Peter, On Fri, Jan 14, 2022 at 4:14 PM Peter Naulls wrote: > > On 1/14/22 10:09 AM, Aleksander Morgado wrote: > logic is not going to change. > > > > If MM detects a single TTY port, it's going to default to use PPP. > > It's not a fallback to PPP, it's using whatever it has for data > > connection, if PPP is the only way forward, PPP will be. > > I don't want PPP, ever. I don't know how many times I can repeat > that. Please stop saying this and arguing with me about it. > I know you don't want PPP in your setup, you have made that clear. I'm not arguing with you about that. I get it. You don't want PPP. Now, the way to fix that should be by making sure ModemManager gets notified of the QMI/NET port before it creates a modem object only with TTY ports. I understand that you don't like this approach, but ModemManager doesn't receive a fixed list of ports to work with, as many other modem management setups. MM tries to dynamically adjust to what it finds, and that is a core feature of ModemManager that is not going to change. So again, and don't hate me for repeating this, if MM gets notified of one single AT capable TTY port and nothing else, it will default to use PPP. If you don't like that, please patch your ModemManager to fit your needs. Or try moving to uqmi, which requires you to specify which is the cdc-wdm port path explicitly, although that approach has its own problems, but maybe not the ones you're worried about. -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
On 1/14/22 10:09 AM, Aleksander Morgado wrote: logic is not going to change. If MM detects a single TTY port, it's going to default to use PPP. It's not a fallback to PPP, it's using whatever it has for data connection, if PPP is the only way forward, PPP will be. I don't want PPP, ever. I don't know how many times I can repeat that. Please stop saying this and arguing with me about it.
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Hey Peter, > > So removing the timing logic is not possible; but why not add an > > extremely longer value that will definitely be safe? E.g. why not set > > EXTRA_PROBING_TIME_MSECS to 6 to make it wait up to 1 minute in > > between port additions in the same device? Well, if we wait so long, > > the modem won't be exposed in DBus during all that time, so it's a > > delay we introduce arbitrarily. Adjusting that timing to a safe enough > > value needs to be considered. > > No. and this is my annoyance with "waiting" in general, and this issue > specifically that I've repeated here many times already. There is no > value which is "long enough". > > An arbitrarily high value is going to cause problems, but it must be > long enough that it's caught most of the time. > > *BUT* if there's some situation where there's a disconnect, external > reset or other condition where the second port is not detected or it's > simply taking "too long", (and I promise you, such situations can and > do occur), then falling back to PPP is not any kind of solution, ever. What's the solution then, when ModemManager doesn't know which kind of modem it's going to manage. You know you have a QMI modem, MM doesn't know it's a QMI modem until it has found a valid QMI+NET pair and has made sure it works. The way MM works is that it decides the type of connection setup to use based on the ports that have been detected. And for that it needs to know when all ports have been reported by the kernel, and we have timing logic associated for that. We can improve that timing logic, or we could even update MM to force a full reprobe of the device when new ports have been detected on an already existing modem, but the core logic is not going to change. If MM detects a single TTY port, it's going to default to use PPP. It's not a fallback to PPP, it's using whatever it has for data connection, if PPP is the only way forward, PPP will be. > > > > > There would be multiple ways to do that; you could modify > > mm_base_modem_organize_ports() so that it errors out when the data > > port is a TTY; > > I guess I will attempt that. > Ok. > > It really depends on what you expect to happen when PPP is assumed to be the > > only data connection mode available. > > Maye it's your English, but I don't want to get to this assumption. > English is not my main language, so it may be that I'm not explaining clearly enough, sorry if that's the case. I could also try to explain in Spanish or Basque if you prefer, but I don't think it would improve much :) > It's too late at that point. Yes, the error when the connection attempt is triggered is probably too late, I also agree with that. I was just giving you options. > Like I said, pretty tired of arguing this point. > I acknowledge we have different point of views; but please, try updating the EXTRA_PROBING_TIME_MSECS value to 3000ms, so that we don't miss the QMI port notifications. If MM detects valid QMI+NET ports, it won't try PPP. Cheers! -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
On 1/14/22 9:01 AM, Aleksander Morgado wrote: Hey, So removing the timing logic is not possible; but why not add an extremely longer value that will definitely be safe? E.g. why not set EXTRA_PROBING_TIME_MSECS to 6 to make it wait up to 1 minute in between port additions in the same device? Well, if we wait so long, the modem won't be exposed in DBus during all that time, so it's a delay we introduce arbitrarily. Adjusting that timing to a safe enough value needs to be considered. No. and this is my annoyance with "waiting" in general, and this issue specifically that I've repeated here many times already. There is no value which is "long enough". An arbitrarily high value is going to cause problems, but it must be long enough that it's caught most of the time. *BUT* if there's some situation where there's a disconnect, external reset or other condition where the second port is not detected or it's simply taking "too long", (and I promise you, such situations can and do occur), then falling back to PPP is not any kind of solution, ever. There would be multiple ways to do that; you could modify mm_base_modem_organize_ports() so that it errors out when the data port is a TTY; I guess I will attempt that. It really depends on what you expect to happen when PPP is assumed to be the only data connection mode available. Maye it's your English, but I don't want to get to this assumption. It's too late at that point. Like I said, pretty tired of arguing this point.
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Hey, > > For a quick workaround in your case, can you try modifying > > EXTRA_PROBING_TIME_MSECS in mm-plugin-manager.c and change it from > > 1500 to something bigger like e.g. 3000 or 4000? > > > > There's one thing clear though; the longer timeout should not be > > applied by default, as that would impose an extra wait time that is > > not needed in most cases. I think we can make this more specific to > > setups like yours, e.g. making the longer timeout happen only for > > openwrt setups (when udev is not available) and also maybe applicable > > only during MM daemon startup (e.g. if the port addition happened in > > the first 10s of the MM daemon startup, allow a bigger extra probing > > time). What do you think? > > > > Yes, I will try this and look at some of the typical timings and see > if something else is going on as an interim fix. > > However, it may always be the case that any timeout isn't "long enough"; But that is a problem that is inherent to how ModemManager works. MM expects ports of the same device to be notified together; with a very short time between each port notification. If this is not the case, e.g. if the ports are notified with a very long time in between each notification, MM may not grab all ports. At some point MM needs to say "I no longer wait for more in this device", and that is achieved by the timing setup hardcoded in MM itself, EXTRA_PROBING_TIME_MSECS is one of the settings associated to that logic. The best way to make this reliable so that no timeout is required would be to have the kernel tell us how many different ports are going to be exposed for the device, and then wait for the notification of each; but that is not how the kernel works. So removing the timing logic is not possible; but why not add an extremely longer value that will definitely be safe? E.g. why not set EXTRA_PROBING_TIME_MSECS to 6 to make it wait up to 1 minute in between port additions in the same device? Well, if we wait so long, the modem won't be exposed in DBus during all that time, so it's a delay we introduce arbitrarily. Adjusting that timing to a safe enough value needs to be considered. MM could also "adjust" the already existing objects if new ports are added once the modem has already been created, but that's a huge functionality change that I don't even consider; the modem object selection is done once we know the ports we have, so that we can e.g. create a QMI-capable device, or just an AT-capable one instead. We cannot change on the fly those objects if more ports in the device are suddenly notified. Another thing that MM could do is trigger a full modem object recreation if new ports are emitted for an already existing device. E.g. we got 2 TTY ports notified, we create an AT based modem, we then get notified of a QMI+NET pair, and we could then flag the AT modem as invalid and trigger a full reprobe from scratch so that the QMI port is taken into account. That may be a bit easier to do actually. > it's still critical to me that for my application I don't fall through to > PPP; this needs to be failure case that I get notified about. > > It might be in the end that changes in detection assumptions are made, > or that we have an option to strip PPP entirely from the build. There would be multiple ways to do that; you could modify mm_base_modem_organize_ports() so that it errors out when the data port is a TTY; that would prevent the modem object from being created. You could also modify the mm-broadband-bearer.c so that its creation fails during a connection attempt; that would still leave the modem object around, but it would fail if you try to connect it. It really depends on what you expect to happen when PPP is assumed to be the only data connection mode available. -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
On 1/14/22 4:28 AM, Aleksander Morgado wrote: Hey Peter, For a quick workaround in your case, can you try modifying EXTRA_PROBING_TIME_MSECS in mm-plugin-manager.c and change it from 1500 to something bigger like e.g. 3000 or 4000? There's one thing clear though; the longer timeout should not be applied by default, as that would impose an extra wait time that is not needed in most cases. I think we can make this more specific to setups like yours, e.g. making the longer timeout happen only for openwrt setups (when udev is not available) and also maybe applicable only during MM daemon startup (e.g. if the port addition happened in the first 10s of the MM daemon startup, allow a bigger extra probing time). What do you think? Yes, I will try this and look at some of the typical timings and see if something else is going on as an interim fix. However, it may always be the case that any timeout isn't "long enough"; it's still critical to me that for my application I don't fall through to PPP; this needs to be failure case that I get notified about. It might be in the end that changes in detection assumptions are made, or that we have an option to strip PPP entirely from the build.
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Hey, > So, the problem is the timing here exclusively, there is no failed QMI > protocol probing. This system you're running seems to be a bit slow, > and the events reported are taking a lot of time to complete before > the next one can be processed; we should definitely fix this in > ModemManager so that we can handle these type of systems as well. > > For a quick workaround in your case, can you try modifying > EXTRA_PROBING_TIME_MSECS in mm-plugin-manager.c and change it from > 1500 to something bigger like e.g. 3000 or 4000? > > There's one thing clear though; the longer timeout should not be > applied by default, as that would impose an extra wait time that is > not needed in most cases. I think we can make this more specific to > setups like yours, e.g. making the longer timeout happen only for > openwrt setups (when udev is not available) and also maybe applicable > only during MM daemon startup (e.g. if the port addition happened in > the first 10s of the MM daemon startup, allow a bigger extra probing > time). What do you think? > I've opened a new issue for this: https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/issues/493 -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Hey Peter, On Thu, Jan 13, 2022 at 3:22 PM Peter Naulls wrote: > > On 1/13/22 5:08 AM, Aleksander Morgado wrote: > > Hey Peter, > > > > > > > Unfortunately this log is missing the port notifications and the port > > probing phase; so we cannot investigate why the QMI port was not > > included in the modem object :/ > > We need the log including the port addition events until the modem > > object is created; the connection attempt itself is really not needed > > to debug this problem. > > > > Ok, I think I got it. > Yes, this log is perfect to understand what happened, see the flow of events below: Wed Jan 12 19:32:35 2022 user.notice ModemManager: hotplug: add network interface wwan0: event processed Wed Jan 12 19:32:35 2022 user.notice ModemManager: hotplug: add cdc interface cdc-wdm0: custom event processed Wed Jan 12 19:32:36 2022 user.notice ModemManager: hotplug: add serial interface ttyUSB0: event processed // These 3 previous events for wwan0, cdc-wdm0 and ttyUSB0 go directly to the cache exclusively, because MM is not running Wed Jan 12 19:32:36 2022 daemon.info [2709]: [1642015956.367284] ModemManager (version 1.18.4) starting in system bus... // MM is already running now, and the next events for tty USB1, ttyUSB2 and ttyUSB3 are notified to the daemon right away, and also added in the cache Wed Jan 12 19:32:37 2022 user.notice ModemManager: hotplug: add serial interface ttyUSB1: event processed Wed Jan 12 19:32:37 2022 daemon.debug [2709]: [1642015957.539903] [base-manager] adding port ttyUSB1 at sysfs path: /sys/devices/platform/1e1c.xhci/usb2/2-1/2-1:1.1/ttyUSB1/tty/ttyUSB1 Wed Jan 12 19:32:38 2022 user.notice ModemManager: hotplug: add serial interface ttyUSB2: event processed Wed Jan 12 19:32:38 2022 daemon.debug [2709]: [1642015958.580584] [base-manager] adding port ttyUSB2 at sysfs path: /sys/devices/platform/1e1c.xhci/usb2/2-1/2-1:1.2/ttyUSB2/tty/ttyUSB2 Wed Jan 12 19:32:39 2022 daemon.debug [2709]: [1642015959.106244] [ttyUSB1/probe] GPS port detected Wed Jan 12 19:32:39 2022 daemon.debug [2709]: [1642015959.648717] [ttyUSB2/probe] port is AT-capable Wed Jan 12 19:32:39 2022 daemon.debug [2709]: [1642015959.651858] [plugin-manager] task 0: all port probings completed, but not reached min probing time yet // 2 full seconds elapsed since the last event reported for ttyUSB2, so ModemManager thinks no more ports will be notified in the modem, and ends up creating the modem object with only these 2 ttyUSB1 and ttyUSB2 ports Wed Jan 12 19:32:40 2022 daemon.info [2709]: [1642015960.093395] [device /sys/devices/platform/1e1c.xhci/usb2/2-1] creating modem with plugin 'quectel' and '2' ports // A bit later, we still receive the event for ttyUSB3, notified directly to the daemon Wed Jan 12 19:32:40 2022 user.notice ModemManager: hotplug: add serial interface ttyUSB3: event processed // And some short time later on we start receiving the events from the cache for wwan0, cdc-wdm0 and ttyUSB0 Wed Jan 12 19:32:40 2022 user.notice ModemManager: hotplug: cached event found: action=add, name=wwan0, subsystem=net, sysfspath=/sys/devices/platform/1e1c.xhci/usb2/2-1/2-1:1.4/net/wwan0 Wed Jan 12 19:32:41 2022 daemon.debug [2709]: [1642015961.010150] [base-manager] additional port ttyUSB3 in device /sys/devices/platform/1e1c.xhci/usb2/2-1 Wed Jan 12 19:32:41 2022 daemon.debug [2709]: [1642015961.049996] [base-manager] additional port wwan0 in device /sys/devices/platform/1e1c.xhci/usb2/2-1 Wed Jan 12 19:32:41 2022 user.notice ModemManager: hotplug: cached event found: action=add, name=ttyUSB0, subsystem=tty, sysfspath=/sys/devices/platform/1e1c.xhci/usb2/2-1/2-1:1.0/ttyUSB0/tty/ttyUSB0 Wed Jan 12 19:32:41 2022 daemon.debug [2709]: [1642015961.121626] [base-manager] additional port cdc-wdm0 in device /sys/devices/platform/1e1c.xhci/usb2/2-1 Wed Jan 12 19:32:41 2022 daemon.debug [2709]: [1642015961.314655] [base-manager] additional port ttyUSB0 in device /sys/devices/platform/1e1c.xhci/usb2/2-1 So, the problem is the timing here exclusively, there is no failed QMI protocol probing. This system you're running seems to be a bit slow, and the events reported are taking a lot of time to complete before the next one can be processed; we should definitely fix this in ModemManager so that we can handle these type of systems as well. For a quick workaround in your case, can you try modifying EXTRA_PROBING_TIME_MSECS in mm-plugin-manager.c and change it from 1500 to something bigger like e.g. 3000 or 4000? There's one thing clear though; the longer timeout should not be applied by default, as that would impose an extra wait time that is not needed in most cases. I think we can make this more specific to setups like yours, e.g. making the longer timeout happen only for openwrt setups (when udev is not available) and also maybe applicable only during MM daemon startup (e.g. if the port addition happened in the first 10s of the MM daemon startup, allow a
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Hey Peter, > > I'm not sure there's an easy way to do that; but in this case we may > > also be interested in the QMI messages really, because we also want to > > make sure the port probing is working as expected; i.e. if this issue > > happens not only upon MM start, also when you power cycle the modem, > > it could be that the port probing of the QMI port is not completing > > successfully, so even if the ports are reported to MM, they're not > > used as MM couldn't verify it's a valid QMI port. > > > > Here we go. I may have to increase the ring buffer size; the top is > the earliest I was able to grab. > > Some of the AT commands are coming from an external application using > mmcli; i wonder if that's confouding MM. > Unfortunately this log is missing the port notifications and the port probing phase; so we cannot investigate why the QMI port was not included in the modem object :/ We need the log including the port addition events until the modem object is created; the connection attempt itself is really not needed to debug this problem. -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
On 1/12/22 9:58 AM, Aleksander Morgado wrote:
Hey,
I'm not sure there's an easy way to do that; but in this case we may
also be interested in the QMI messages really, because we also want to
make sure the port probing is working as expected; i.e. if this issue
happens not only upon MM start, also when you power cycle the modem,
it could be that the port probing of the QMI port is not completing
successfully, so even if the ports are reported to MM, they're not
used as MM couldn't verify it's a valid QMI port.
Here we go. I may have to increase the ring buffer size; the top is
the earliest I was able to grab.
Some of the AT commands are coming from an external application using
mmcli; i wonder if that's confouding MM.
Wed Jan 12 14:50:08 2022 daemon.debug [2670]: [1641999008.905414]
[modem0/ttyUSB2/at] device open count is 3 (open)
Wed Jan 12 14:50:08 2022 daemon.debug [2670]: [1641999008.905722]
[modem0/ttyUSB2/at] device open count is 2 (close)
Wed Jan 12 14:50:08 2022 daemon.debug [2670]: [1641999008.906044]
[modem0/ttyUSB2/at] --> 'AT+CLCK="PP",2'
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.044547]
[modem0/ttyUSB2/at] <-- '+CLCK: 0OK'
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.045268]
[modem0/ttyUSB2/at] device open count is 3 (open)
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.045577]
[modem0/ttyUSB2/at] device open count is 2 (close)
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.045887]
[modem0/ttyUSB2/at] --> 'AT+CLCK="PC",2'
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.184502]
[modem0/ttyUSB2/at] <-- '+CLCK: 0OK'
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.185363]
[modem0] loading IMEI...
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.185597]
[modem0/ttyUSB2/at] device open count is 3 (open)
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.185935]
[modem0/ttyUSB2/at] device open count is 2 (close)
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.186344]
[modem0] loaded IMEI: 00111913505
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.186606]
[modem0] loading UE mode of operation for EPS...
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.186806]
[modem0/ttyUSB2/at] device open count is 3 (open)
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.187086]
[modem0/ttyUSB2/at] device open count is 2 (close)
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.187382]
[modem0/ttyUSB2/at] --> 'AT+CEMODE?'
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.291833]
[modem0/ttyUSB2/at] <-- '+CEMODE: 2OK'
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.296783]
[modem0/ttyUSB2/at] device open count is 3 (open)
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.297253]
[modem0/ttyUSB2/at] device open count is 2 (close)
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.299541]
[modem0/ttyUSB2/at] device open count is 1 (close)
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.303960]
[modem0/ttyUSB2/at] device open count is 2 (open)
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.304451]
[modem0/ttyUSB2/at] --> 'AT+QGPS=?'
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.393243]
[modem0/ttyUSB2/at] <-- '+QGPS:
(1-4),(1-255),(1-3),(100-65535)OK'
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.393933]
[modem0] (quectel) GPS management with +QGPS is supported
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.399827]
[modem0/ttyUSB2/at] device open count is 3 (open)
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.400289]
[modem0/ttyUSB2/at] device open count is 2 (close)
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.400607]
[modem0/ttyUSB2/at] --> 'AT+CNMI=?'
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.490668]
[modem0/ttyUSB2/at] <-- '+CNMI:
(0-2),(0-3),(0,2),(0-2),(0,1)OK'
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.491404]
[modem0/ttyUSB2/at] device open count is 3 (open)
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.491690]
[modem0/ttyUSB2/at] device open count is 2 (close)
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.491954]
[modem0/ttyUSB2/at] --> 'AT+CPMS=?'
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.580850]
[modem0/ttyUSB2/at] <-- '+CPMS:
("ME","MT","SM","SR"),("ME","MT","SM","SR"),("ME","MT","SM","SR")OK'
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.583725]
[modem0] supported storages loaded:
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.583885]
[modem0] mem1 (list/read/delete) storages: 'me, mt, sm, sr'
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.583993]
[modem0] mem2 (write/send) storages: 'me, mt, sm, sr'
Wed Jan 12 14:50:09 2022 daemon.debug [2670]: [1641999009.584100]
[modem0] mem3 (reception) storages:'me, mt, sm, sr'
Wed Jan 12 14:50:09 2022
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Hey,
> >>> Also, please note that if you fully disable the PPP usage (maybe
> >>> making sure that data_at_primary is always NULL in
> >>> mm_base_modem_organize_ports()), what you will achieve is that the
> >>> modem ends up not usable ("Failed to find a data port in the modem").
> >>> You need to decide whether that's better than falling back to PPP. As
> >>> said in an earlier email, the key would be to understand why you don't
> >>> have a valid QMI+NET pair, not really to disable the PPP usage (IMO).
> >>
> >> Well, it took me a while to get this; it requires a specific setup. What
> >> I can
> >> say is that it most often happens at OpenWrt boot, but by no means always.
> >>
> >
> > There's a problem in that log, and the problem is exclusively
> > happening on system boots.
>
> I'll take your word for it, *but* the problem with triggering PPP happens
> other times too, but it's just most likely at boot. The end result is much
> the same.
>
As said earlier, then it could also be due to failed QMI port
probings. In the log you posted below in this email, the QMI port is
correctly detected, and correctly probed:
Tue Jan 11 18:14:05 2022 daemon.debug [2603]: [/dev/cdc-wdm0] QMI Device
supports 28 services:
> It's worth pointing out that if there's loss of connectivity on the modem
> (unplug the antenna for example), it will completely cycle the USB and the
> ports will vanish and then come back. I don't know if this is normal, but
> the effect is much like (but not identical) to that of boot.
That looks like a firmware crash in the modem; the modem goes away
from USB and comes back, and full port probing is restarted in MM.
>
> As a digression, the "45 second" issue to which I put the retries in for
> would most often happen at "first boot" where it's setting up the flash,
> but MM hadn't started yet.
>
> In any case, I have a script which power cycles the modem and the USB
> hub, which closely emulates the power on cycle/signal loss, but is obviously
> not
> 100% identical. I use this for some of my testing.
>
> Here's a log where the system does come up properly; I think this is worth
> considering to make sure this is OK:
>
It looks ok to me; as per the message I referenced above, the QMI port
is correctly detected and the port probing has succeeded.
>
>
> ue Jan 11 18:13:57 2022 user.notice ModemManager: hotplug: add network
> interface
> ip6tnl0: event processed
> Tue Jan 11 18:13:57 2022 user.notice ModemManager: hotplug: error: parent
> device
> sysfspath not found
> Tue Jan 11 18:13:57 2022 user.notice ModemManager: hotplug: add network
> interface gre0: event processed
> Tue Jan 11 18:13:57 2022 user.notice ModemManager: hotplug: error: parent
> device
> sysfspath not found
> Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.616720] mtk_soc_eth
> 1e10.ethernet: PPE started
> Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.638877] br-lan: port
> 1(eth0.1)
> entered blocking state
> Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.649726] br-lan: port
> 1(eth0.1)
> entered disabled state
> Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.661516] device eth0.1
> entered
> promiscuous mode
> Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.671169] device eth0 entered
> promiscuous mode
> Tue Jan 11 18:13:58 2022 user.notice ModemManager: hotplug: add network
> interface gretap0: event processed
> Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.686813] br-lan: port
> 1(eth0.1)
> entered blocking state
> Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.697655] br-lan: port
> 1(eth0.1)
> entered forwarding state
> Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.709392] IPv6:
> ADDRCONF(NETDEV_UP): br-lan: link is not ready
> Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'lan' is enabled
> Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'lan' is setting up
> now
> Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'lan' is now up
> Tue Jan 11 18:13:58 2022 daemon.notice netifd: bridge 'br-lan' link is up
> Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'lan' has link
> connectivity
> Tue Jan 11 18:13:58 2022 daemon.notice netifd: VLAN 'eth0.1' link is up
> Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'loopback' is enabled
> Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'loopback' is setting
> up now
> Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'loopback' is now up
> Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'wwan' is setting up
> now
> Tue Jan 11 18:13:58 2022 daemon.notice netifd: Network device 'eth0' link is
> up
> Tue Jan 11 18:13:58 2022 daemon.notice netifd: Network device 'lo' link is up
> Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'loopback' has link
> connectivity
> Tue Jan 11 18:13:58 2022 user.notice ModemManager: hotplug: error: parent
> device
> sysfspath not found
> Tue Jan 11 18:13:58 2022 daemon.notice netifd: wwan (2328): error: couldn't
> get
>
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Hey, > > > The list of available ports is decided during probing. If probing ends > > with only 2 TTY ports (as in the log you posted earlier), then QMI > > will be fully ignored and the net port will never be used. We really > > need to fix the probing phase so that that doesn't happen. If you > > could gather a MM debug log while reproducing the problem (not sure > > how easy it is to reproduce) that would be awesome and much easier to > > find where the true problem is. > > I will try, but one of the problems here is that the QMI messaging > deubg completely overwhelms the syslog. Is there a way to turn > that off but maintain all the other logging? > I'm not sure there's an easy way to do that; but in this case we may also be interested in the QMI messages really, because we also want to make sure the port probing is working as expected; i.e. if this issue happens not only upon MM start, also when you power cycle the modem, it could be that the port probing of the QMI port is not completing successfully, so even if the ports are reported to MM, they're not used as MM couldn't verify it's a valid QMI port. -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
On 1/12/22 9:16 AM, Aleksander Morgado wrote: Hey, The list of available ports is decided during probing. If probing ends with only 2 TTY ports (as in the log you posted earlier), then QMI will be fully ignored and the net port will never be used. We really need to fix the probing phase so that that doesn't happen. If you could gather a MM debug log while reproducing the problem (not sure how easy it is to reproduce) that would be awesome and much easier to find where the true problem is. I will try, but one of the problems here is that the QMI messaging deubg completely overwhelms the syslog. Is there a way to turn that off but maintain all the other logging?
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
On 1/12/22 4:25 AM, Aleksander Morgado wrote:
Hey
Also, please note that if you fully disable the PPP usage (maybe
making sure that data_at_primary is always NULL in
mm_base_modem_organize_ports()), what you will achieve is that the
modem ends up not usable ("Failed to find a data port in the modem").
You need to decide whether that's better than falling back to PPP. As
said in an earlier email, the key would be to understand why you don't
have a valid QMI+NET pair, not really to disable the PPP usage (IMO).
Well, it took me a while to get this; it requires a specific setup. What I can
say is that it most often happens at OpenWrt boot, but by no means always.
There's a problem in that log, and the problem is exclusively
happening on system boots.
I'll take your word for it, *but* the problem with triggering PPP happens
other times too, but it's just most likely at boot. The end result is much
the same.
It's worth pointing out that if there's loss of connectivity on the modem
(unplug the antenna for example), it will completely cycle the USB and the
ports will vanish and then come back. I don't know if this is normal, but
the effect is much like (but not identical) to that of boot.
As a digression, the "45 second" issue to which I put the retries in for
would most often happen at "first boot" where it's setting up the flash,
but MM hadn't started yet.
In any case, I have a script which power cycles the modem and the USB
hub, which closely emulates the power on cycle/signal loss, but is obviously not
100% identical. I use this for some of my testing.
Here's a log where the system does come up properly; I think this is worth
considering to make sure this is OK:
ue Jan 11 18:13:57 2022 user.notice ModemManager: hotplug: add network interface
ip6tnl0: event processed
Tue Jan 11 18:13:57 2022 user.notice ModemManager: hotplug: error: parent device
sysfspath not found
Tue Jan 11 18:13:57 2022 user.notice ModemManager: hotplug: add network
interface gre0: event processed
Tue Jan 11 18:13:57 2022 user.notice ModemManager: hotplug: error: parent device
sysfspath not found
Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.616720] mtk_soc_eth
1e10.ethernet: PPE started
Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.638877] br-lan: port 1(eth0.1)
entered blocking state
Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.649726] br-lan: port 1(eth0.1)
entered disabled state
Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.661516] device eth0.1 entered
promiscuous mode
Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.671169] device eth0 entered
promiscuous mode
Tue Jan 11 18:13:58 2022 user.notice ModemManager: hotplug: add network
interface gretap0: event processed
Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.686813] br-lan: port 1(eth0.1)
entered blocking state
Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.697655] br-lan: port 1(eth0.1)
entered forwarding state
Tue Jan 11 18:13:58 2022 kern.info kernel: [ 30.709392] IPv6:
ADDRCONF(NETDEV_UP): br-lan: link is not ready
Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'lan' is enabled
Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'lan' is setting up now
Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'lan' is now up
Tue Jan 11 18:13:58 2022 daemon.notice netifd: bridge 'br-lan' link is up
Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'lan' has link
connectivity
Tue Jan 11 18:13:58 2022 daemon.notice netifd: VLAN 'eth0.1' link is up
Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'loopback' is enabled
Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'loopback' is setting
up now
Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'loopback' is now up
Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'wwan' is setting up
now
Tue Jan 11 18:13:58 2022 daemon.notice netifd: Network device 'eth0' link is up
Tue Jan 11 18:13:58 2022 daemon.notice netifd: Network device 'lo' link is up
Tue Jan 11 18:13:58 2022 daemon.notice netifd: Interface 'loopback' has link
connectivity
Tue Jan 11 18:13:58 2022 user.notice ModemManager: hotplug: error: parent device
sysfspath not found
Tue Jan 11 18:13:58 2022 daemon.notice netifd: wwan (2328): error: couldn't get
bus: Could not connect: No such file or directory
Tue Jan 11 18:13:58 2022 daemon.notice netifd: wwan (2328): Device not managed
by ModemManager
Tue Jan 11 18:13:58 2022 user.notice firewall: Reloading firewall due to ifup of
lan (br-lan)
Tue Jan 11 18:13:59 2022 daemon.notice netifd: wwan (2400): stopping network
Tue Jan 11 18:13:59 2022 daemon.notice netifd: wwan (2400): error: couldn't get
bus: Could not connect: No such file or directory
Tue Jan 11 18:13:59 2022 daemon.notice netifd: wwan (2400): couldn't load bearer
path
Tue Jan 11 18:13:59 2022 daemon.notice netifd: Interface 'wwan' is now down
Tue Jan 11 18:13:59 2022 kern.info kernel: [ 31.643038] IPv6:
ADDRCONF(NETDEV_CHANGE): br-lan:
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Hey, > > In the openwrt git master branch (which I believe it's what you're > > using) that logic was updated to use a new wrapper script in > > /usr/sbin/ModemManager-wrapper, and this scripts introduces a bogus > > "sleep 2" in between the MM start and the call to > > mm_report_events_from_cache: > > > > /usr/sbin/ModemManager "$@" 1>/dev/null 2>/dev/null & > > CHILD="$!" > > sleep 2 > > mm_report_events_from_cache > > > > I think that sleep is breaking the modem detection here, because the > > realtime events are reported as soon as MM is detected, and the cached > > ones are reported more than 2s later, and MM has a strict timing of > > how long to wait for new port events since the last port event has > > happened. > > The OpenWrt I'm using is a bit of a frakenstein, due to how I got to > this point, but it's approximately 21.02 with some dev pieces. It could > probably be rebased on 21.02, but that requires some effort. > > In any event, although I'm using latest MM (along with patches > we've discussed), the packaging and scripts is very much like what's > in 21.02 - I went through and double checked that there's no changes > like this. In any case, I don't have the wrapper script or the sleep 2. > Oh well; then the issue could be in the implicit wait done inside mm_report_events_from_cache(), which has a loop with 1s in each loop. If this timing is so tight that it affects how modem objects are created, we may need to tweak the timing in the plugin probing (EXTRA_PROBING_TIME_MSECS in mm-plugin-manager.c). I truly assumed you were using the latest changes, because according to your logs the cached events were reported approx 2s after MM started. > As an aside, I went and turned off all the plugins except for quectel > (this could be done for OpenWrt menuconfig, but that's another topic) Yes. > just to avoid confusion. > The device you're using will fall into a vid:pid fillter whitelist, so there's not much room for error if you leave all the other plugins as well. > What I can see, is that even on an apparently successful connection, > is there seems to be some confusion about what's available for ports, > but I don't know enough about the mechanism to say if it's OK or not. > I'll post a log of this shortly. The list of available ports is decided during probing. If probing ends with only 2 TTY ports (as in the log you posted earlier), then QMI will be fully ignored and the net port will never be used. We really need to fix the probing phase so that that doesn't happen. If you could gather a MM debug log while reproducing the problem (not sure how easy it is to reproduce) that would be awesome and much easier to find where the true problem is. -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
On 1/12/22 4:25 AM, Aleksander Morgado wrote: In the openwrt git master branch (which I believe it's what you're using) that logic was updated to use a new wrapper script in /usr/sbin/ModemManager-wrapper, and this scripts introduces a bogus "sleep 2" in between the MM start and the call to mm_report_events_from_cache: /usr/sbin/ModemManager "$@" 1>/dev/null 2>/dev/null & CHILD="$!" sleep 2 mm_report_events_from_cache I think that sleep is breaking the modem detection here, because the realtime events are reported as soon as MM is detected, and the cached ones are reported more than 2s later, and MM has a strict timing of how long to wait for new port events since the last port event has happened. The OpenWrt I'm using is a bit of a frakenstein, due to how I got to this point, but it's approximately 21.02 with some dev pieces. It could probably be rebased on 21.02, but that requires some effort. In any event, although I'm using latest MM (along with patches we've discussed), the packaging and scripts is very much like what's in 21.02 - I went through and double checked that there's no changes like this. In any case, I don't have the wrapper script or the sleep 2. As an aside, I went and turned off all the plugins except for quectel (this could be done for OpenWrt menuconfig, but that's another topic) just to avoid confusion. What I can see, is that even on an apparently successful connection, is there seems to be some confusion about what's available for ports, but I don't know enough about the mechanism to say if it's OK or not. I'll post a log of this shortly.
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Hey
> > Also, please note that if you fully disable the PPP usage (maybe
> > making sure that data_at_primary is always NULL in
> > mm_base_modem_organize_ports()), what you will achieve is that the
> > modem ends up not usable ("Failed to find a data port in the modem").
> > You need to decide whether that's better than falling back to PPP. As
> > said in an earlier email, the key would be to understand why you don't
> > have a valid QMI+NET pair, not really to disable the PPP usage (IMO).
>
> Well, it took me a while to get this; it requires a specific setup. What I
> can
> say is that it most often happens at OpenWrt boot, but by no means always.
>
There's a problem in that log, and the problem is exclusively
happening on system boots. When the system boots, modem ports may be
detected BEFORE ModemManager has even started; what we do in the
hotplug script is to add these events in a "cache" of events that will
be processed by the ModemManager init script upon boot. The init
script will run the daemon, and right away run
"mm_report_events_from_cache" which should go over all the events in
the cache and report them one by one again to MM, as if they had just
happened. This is a bit weird, but it fits very well with the logic
applied by the normal usecase when using udev.
Now, what I can see in your logs is that the cdc-wdm port and net port
are detected way before MM has started, so they're added to the cache.
But then, MM starts, and some of the ttyUSB events don't reach the
cache because MM has already started, so they're reported to MM right
away. The final outcome is that MM creates the modem with the ttyUSB
ports that were detected during runtime and we miss all the ports that
had hit the cache, which are reported later on:
> Tue Jan 11 18:08:55 2022 daemon.info [2757]: [1641924535.408371]
> [device
/sys/devices/platform/1e1c.xhci/usb2/2-1] creating modem with plugin
'quectel' and '2' ports
...
Tue Jan 11 18:08:56 2022 user.notice ModemManager: hotplug: cached event found:
action=add, name=wwan0, subsystem=net,
sysfspath=/sys/devices/platform/1e1c.xhci/usb2/2-1/2-1:1.4/net/wwan0
Tue Jan 11 18:08:56 2022 daemon.info [2757]: [1641924536.122148]
[base-manager] modem for device '/sys/devices/platform/1e1c.xhci/usb2/2-1'
successfully created
Tue Jan 11 18:08:56 2022 user.notice ModemManager: hotplug: cached event found:
action=add, name=cdc-wdm0, subsystem=usbmisc,
sysfspath=/sys/devices/platform/1e1c.xhci/usb2/2-1/2-1:1.4/net/wwan0
I believe this is a regression in the openwrt packaging. In the
original implementation I did some time ago, we call
mm_report_events_from_cache BEFORE MM even starts, and that function
will make sure to emit the events as soon as MM is ready to receive
them (so the events in the cache are reported at the same time as
realtime events happening). This logic is the one available in the
openwrt-21.02 branch, if you want to look at it.
In the openwrt git master branch (which I believe it's what you're
using) that logic was updated to use a new wrapper script in
/usr/sbin/ModemManager-wrapper, and this scripts introduces a bogus
"sleep 2" in between the MM start and the call to
mm_report_events_from_cache:
/usr/sbin/ModemManager "$@" 1>/dev/null 2>/dev/null &
CHILD="$!"
sleep 2
mm_report_events_from_cache
I think that sleep is breaking the modem detection here, because the
realtime events are reported as soon as MM is detected, and the cached
ones are reported more than 2s later, and MM has a strict timing of
how long to wait for new port events since the last port event has
happened.
Can you edit your /usr/sbin/ModemManager-wrapper and remove that
"sleep 2" completely?
--
Aleksander
https://aleksander.es
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Also, please note that if you fully disable the PPP usage (maybe
making sure that data_at_primary is always NULL in
mm_base_modem_organize_ports()), what you will achieve is that the
modem ends up not usable ("Failed to find a data port in the modem").
You need to decide whether that's better than falling back to PPP. As
said in an earlier email, the key would be to understand why you don't
have a valid QMI+NET pair, not really to disable the PPP usage (IMO).
Well, it took me a while to get this; it requires a specific setup. What I can
say is that it most often happens at OpenWrt boot, but by no means always.
Full log below, but perhaps here's some salient pieces:
Tue Jan 11 18:09:02 2022 daemon.warn [2757]: [1641924542.939644]
[modem0/sim0] couldn't load list of preferred networks: Operation not allowed
Tue Jan 11 18:09:05 2022 daemon.info [2757]: [1641924545.174757]
[modem0] state changed (unknown -> disabled)
At the end of the log, you can see PPP failures which is of course where
I've disabled stuff (not enough, apparently, but it's too late at that
point anyway. But then it'll restart, and eventually reconnect. Anyway.
Tue Jan 11 18:08:46 2022 user.notice ModemManager: hotplug: add network
interface ip6tnl0: event processed
Tue Jan 11 18:08:46 2022 user.notice ModemManager: hotplug: error: parent device
sysfspath not found
Tue Jan 11 18:08:47 2022 user.notice ModemManager: hotplug: add network
interface gre0: event processed
Tue Jan 11 18:08:47 2022 user.notice ModemManager: hotplug: error: parent device
sysfspath not found
Tue Jan 11 18:08:47 2022 user.notice : Added device handler type: bonding
Tue Jan 11 18:08:47 2022 user.notice : Added device handler type: 8021ad
Tue Jan 11 18:08:47 2022 user.notice : Added device handler type: 8021q
Tue Jan 11 18:08:47 2022 user.notice : Added device handler type: macvlan
Tue Jan 11 18:08:47 2022 user.notice : Added device handler type: veth
Tue Jan 11 18:08:47 2022 user.notice : Added device handler type: bridge
Tue Jan 11 18:08:47 2022 user.notice : Added device handler type: Network device
Tue Jan 11 18:08:47 2022 user.notice : Added device handler type: tunnel
Tue Jan 11 18:08:47 2022 user.notice ModemManager: hotplug: add network
interface gretap0: event processed
Tue Jan 11 18:08:47 2022 user.notice ModemManager: hotplug: error: parent device
sysfspath not found
Tue Jan 11 18:08:48 2022 user.notice ModemManager: hotplug: add network
interface erspan0: event processed
Tue Jan 11 18:08:48 2022 user.notice ModemManager: hotplug: error: parent device
sysfspath not found
Tue Jan 11 18:08:49 2022 kern.info kernel: [ 30.873270] mtk_soc_eth
1e10.ethernet: PPE started
Tue Jan 11 18:08:49 2022 kern.info kernel: [ 30.893726] br-lan: port 1(eth0.1)
entered blocking state
Tue Jan 11 18:08:49 2022 kern.info kernel: [ 30.904624] br-lan: port 1(eth0.1)
entered disabled state
Tue Jan 11 18:08:49 2022 kern.info kernel: [ 30.916624] device eth0.1 entered
promiscuous mode
Tue Jan 11 18:08:49 2022 kern.info kernel: [ 30.926267] device eth0 entered
promiscuous mode
Tue Jan 11 18:08:49 2022 kern.info kernel: [ 30.944946] br-lan: port 1(eth0.1)
entered blocking state
Tue Jan 11 18:08:49 2022 kern.info kernel: [ 30.955838] br-lan: port 1(eth0.1)
entered forwarding state
Tue Jan 11 18:08:49 2022 kern.info kernel: [ 30.967643] IPv6:
ADDRCONF(NETDEV_UP): br-lan: link is not ready
Tue Jan 11 18:08:49 2022 daemon.notice netifd: Interface 'lan' is enabled
Tue Jan 11 18:08:49 2022 daemon.notice netifd: Interface 'lan' is setting up now
Tue Jan 11 18:08:49 2022 daemon.notice netifd: Interface 'lan' is now up
Tue Jan 11 18:08:49 2022 daemon.notice netifd: bridge 'br-lan' link is up
Tue Jan 11 18:08:49 2022 daemon.notice netifd: Interface 'lan' has link
connectivity
Tue Jan 11 18:08:49 2022 daemon.notice netifd: VLAN 'eth0.1' link is up
Tue Jan 11 18:08:49 2022 daemon.notice netifd: Interface 'loopback' is enabled
Tue Jan 11 18:08:49 2022 daemon.notice netifd: Interface 'loopback' is setting
up now
Tue Jan 11 18:08:49 2022 daemon.notice netifd: Interface 'loopback' is now up
Tue Jan 11 18:08:49 2022 daemon.notice netifd: Interface 'wwan' is setting up
now
Tue Jan 11 18:08:49 2022 daemon.notice netifd: Network device 'eth0' link is up
Tue Jan 11 18:08:49 2022 daemon.notice netifd: Network device 'lo' link is up
Tue Jan 11 18:08:49 2022 daemon.notice netifd: Interface 'loopback' has link
connectivity
Tue Jan 11 18:08:49 2022 user.notice ModemManager: hotplug: add network
interface ip6gre0: event processed
Tue Jan 11 18:08:49 2022 user.notice ModemManager: hotplug: error: parent device
sysfspath not found
Tue Jan 11 18:08:49 2022 daemon.notice netifd: wwan (2500): error: couldn't get
bus: Could not connect: No such file or directory
Tue Jan 11 18:08:49 2022 daemon.notice netifd: wwan (2500): Device not managed
by ModemManager
Tue Jan 11 18:08:49 2022 daemon.notice netifd: wwan (2564): stopping network
Tue Jan 11 18:08:49
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Hey Peter,
> >> I see. So the problem is that I have to keep port 02 open for mmcli
> >> commands,
> >> but if I do it'll try to use this for PPP still sometimes.
> >>
> >
> > Please note that the reason to use PPP is not that MM sometimes
> > prefers it over QMI+NET. If PPP is being used, it's likely because the
> > QMI port failed to be probed successfully, so AT+PPP is the only
> > option left. The failure in the QMI probing could be a limitation of
> > the probing logic, or the modem firmware getting fully stuck, or
> > something in between (e.g. maybe a given modem boot took longer than
> > usual and we failed probing while waiting for it to work properly).
> > Debug logs will tell you in detail why PPP is chosen instead of
> > QMI+NET.
> >
>
> Well, I didn't dig into the debug, but I understand why it's falling back
> and that QMI/driver is not ready yet - it doesn't matter though; I need this
> to be a failure case (and then fall through to retry). PPP is not a credible
> option in my setup, and will just cause a longer connect cycle, since it'll
> fail later on.
>
> On the flip side, I understand the rationale for the general case here.
>
> This is what I ended up using as a local patch, which should be obvious:
>
> --- a/src/mm-base-modem.c.orig2022-01-04 09:50:31.631768015 -0500
> +++ b/src/mm-base-modem.c 2022-01-04 09:46:08.183807331 -0500
> @@ -387,9 +387,9 @@
> } else if (mm_kernel_device_get_property_as_boolean
> (kernel_device, ID_MM_PORT_TYPE_AT_SECONDARY)) {
> mm_obj_dbg (port, "AT port flagged as secondary");
> at_pflags = MM_PORT_SERIAL_AT_FLAG_SECONDARY;
> -} else if (mm_kernel_device_get_property_as_boolean
> (kernel_device,
> ID_MM_PORT_TYPE_AT_PPP)) {
> +/*} else if (mm_kernel_device_get_property_as_boolean
> (kernel_device, ID_MM_PORT_TYPE_AT_PPP)) {
> mm_obj_dbg (port, "AT port flagged as PPP");
> -at_pflags = MM_PORT_SERIAL_AT_FLAG_PPP;
> +at_pflags = MM_PORT_SERIAL_AT_FLAG_PPP;*/
> }
> }
> /* The plugin may specify NONE_NO_GENERIC to avoid the generic
>
>
> Maybe this should be a command line option or something, up to you.
>
Please note, that patch is not completely disabling the PPP usage,
it's just disabling the PPP port type hint. If ModemManager detects AT
ports and no valid QMI+NET port, it will still use PPP over the tty on
the detected AT port. By disabling the port type hint, you're removing
the association of which explicit port should be used for PPP data,
and that means MM may try PPP over a serial port that it's not
prepared for that.
Also, please note that if you fully disable the PPP usage (maybe
making sure that data_at_primary is always NULL in
mm_base_modem_organize_ports()), what you will achieve is that the
modem ends up not usable ("Failed to find a data port in the modem").
You need to decide whether that's better than falling back to PPP. As
said in an earlier email, the key would be to understand why you don't
have a valid QMI+NET pair, not really to disable the PPP usage (IMO).
> As for ID_MM_PORT_IGNORE, I had to pull those out - it seems that
> once ModemManager connects, it's no longer possible to use mmcli - I didn't
> try to dig down into this detail, but it was problematic in our testing.
>
Not sure what you mean with this, mmcli is completely independent from
the connection state. What mmcli command failed?
--
Aleksander
https://aleksander.es
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
On 1/1/22 2:33 PM, Aleksander Morgado wrote:
Hey Peter,
I see. So the problem is that I have to keep port 02 open for mmcli commands,
but if I do it'll try to use this for PPP still sometimes.
Please note that the reason to use PPP is not that MM sometimes
prefers it over QMI+NET. If PPP is being used, it's likely because the
QMI port failed to be probed successfully, so AT+PPP is the only
option left. The failure in the QMI probing could be a limitation of
the probing logic, or the modem firmware getting fully stuck, or
something in between (e.g. maybe a given modem boot took longer than
usual and we failed probing while waiting for it to work properly).
Debug logs will tell you in detail why PPP is chosen instead of
QMI+NET.
Well, I didn't dig into the debug, but I understand why it's falling back
and that QMI/driver is not ready yet - it doesn't matter though; I need this
to be a failure case (and then fall through to retry). PPP is not a credible
option in my setup, and will just cause a longer connect cycle, since it'll
fail later on.
On the flip side, I understand the rationale for the general case here.
This is what I ended up using as a local patch, which should be obvious:
--- a/src/mm-base-modem.c.orig 2022-01-04 09:50:31.631768015 -0500
+++ b/src/mm-base-modem.c 2022-01-04 09:46:08.183807331 -0500
@@ -387,9 +387,9 @@
} else if (mm_kernel_device_get_property_as_boolean
(kernel_device, ID_MM_PORT_TYPE_AT_SECONDARY)) {
mm_obj_dbg (port, "AT port flagged as secondary");
at_pflags = MM_PORT_SERIAL_AT_FLAG_SECONDARY;
-} else if (mm_kernel_device_get_property_as_boolean (kernel_device,
ID_MM_PORT_TYPE_AT_PPP)) {
+/*} else if (mm_kernel_device_get_property_as_boolean
(kernel_device, ID_MM_PORT_TYPE_AT_PPP)) {
mm_obj_dbg (port, "AT port flagged as PPP");
-at_pflags = MM_PORT_SERIAL_AT_FLAG_PPP;
+at_pflags = MM_PORT_SERIAL_AT_FLAG_PPP;*/
}
}
/* The plugin may specify NONE_NO_GENERIC to avoid the generic
Maybe this should be a command line option or something, up to you.
As for ID_MM_PORT_IGNORE, I had to pull those out - it seems that
once ModemManager connects, it's no longer possible to use mmcli - I didn't
try to dig down into this detail, but it was problematic in our testing.
Thanks.
Re: FCC unlock surprises after upgrading (was: Re: ANN: ModemManager 1.18.4 released)
Hey, > > * A new FCC unlock operation management via external scripts is > >introduced, which will avoid to automatically unlock FCC locked > >devices unless the user has configured the operation manually, or > >unless an official vendor-provided FCC unlock tool is found in the > >system. > > Sorry for late comments again, but I'm worrying about the user > experience for those upgrading a previously working system. > > I did read this announcement and the discussion about the feature back > in november, and thought it sounded all fine. And I upgraded to 1.18.4 > right before christmas when the Debian package showed up. Still didn't > think much about this, and obviously didn't read out of the NEWS file > that I was supposed to do something... > > Now I don't suspend or reboot my laptop often with Corona office and all > that. But today I did. And was very surprised when MM failed to > connect after resuming. Manually trying to enable the modem failed with > a message which didn't immediately ring any bells: > > root@miraculix:/home/bjorn# mmcli -m any -e > error: couldn't enable the modem: > 'GDBus.Error:org.freedesktop.ModemManager1.Error.Core.Retry: Invalid > transition' > > I must admit that I had to take a detour via AT commands, looking at > AT!PCINFO? to realize that this was a failure to do the FCC unlock. Only > after that did it occur to me that I should read those instructions > again. But that was only because I had seen then before only a few > weeks ago. I'm not sure everyone else would make the connection. Or > maybe they would, and I'm the only slow one here :-) > > Fixing the problem was of course as simple as > > cd /etc/ModemManager/fcc-unlock.d > ln -s /usr/share/ModemManager/fcc-unlock.available.d/1199\:9079 > > and everything worked as expected after that. > Please note that you can use one single command to link all known scripts, it's probably easier than going to the directory and looking up which is your exact vid:pid: $ sudo ln -sft /etc/ModemManager/fcc-unlock.d /usr/share/ModemManager/fcc-unlock.available.d/* > I understand that you want/need to make this a concious decision for new > users and modems. But I wonder if we really have to make it this > difficult for existing users? After all, I am spoiled by 5+ years of > fully automatic unlock on this laptop/modem. Why should I have to > manually re-enable it now? Maybe we could accept a script adding the > symlink on upgrades from older MM versions, on systems with an affected > modem? Or at least a subset of older modems? When I started to work on this, I definitely didn't want to have different approaches for different modems. Because, where to draw the line? In the modems using the old Sierra approach? In the SDX55 that we already had included to autounlock? In the new EM120 unlock logic that is already available in libmbim/mbimcli? If we had some to autounlock, but not all, why not add all the ones we know how to unlock? My approach is to say that ModemManager has been doing it all wrong since the very beginning, we shouldn't have done any auto unlock. MM has tried to solve (circumvent) an issue introduced by modem manufacturers, and it shouldn't have even tried it. So, all modules require the manual user configuration. This is a one-time operation to be done by the user, effectively selecting which fcc unlock procedure to run (the ones known by MM, or the ones installed by manufacturers). > > Or maybe this isn't a big problem? After all, it's just a one time > thing. I only hope people figure it out quicker than me... > I'd like to think that any user complaining about 1.18.4 not connecting will be able to find info about the FCC unlock change on the Internet. I've written info in the release notes, I wrote the whole https://modemmanager.org/docs/modemmanager/fcc-unlock/ page in the doc site, and I've also blogged about it. -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Hey Peter,
> >> # AWC EM12-AW
> >> # ttyUSB0 (if #0): QCDM/DIAG port
> >> # ttyUSB1 (if #1): GPS data port
> >> # ttyUSB2 (if #2): AT primary port
> >> # ttyUSB3 (if #3): AT secondary port
> >> ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="00",
> >> SUBSYSTEM=="tty", ENV{ID_MM_PORT_TYPE_QCDM}="1"
> >> ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="01",
> >> SUBSYSTEM=="tty", ENV{ID_MM_DEVICE_IGNORE}="1"
> >> ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="02",
> >> SUBSYSTEM=="tty", ENV{ID_MM_DEVICE_IGNORE}="1
> >> ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="03",
> >> SUBSYSTEM=="tty", ENV{ID_MM_DEVICE_IGNORE}="1"
> >>
> >
> > Should be ID_MM_PORT_IGNORE in all those (not DEVICE_iGNORE).
> >
>
> I see. So the problem is that I have to keep port 02 open for mmcli commands,
> but if I do it'll try to use this for PPP still sometimes.
>
Please note that the reason to use PPP is not that MM sometimes
prefers it over QMI+NET. If PPP is being used, it's likely because the
QMI port failed to be probed successfully, so AT+PPP is the only
option left. The failure in the QMI probing could be a limitation of
the probing logic, or the modem firmware getting fully stuck, or
something in between (e.g. maybe a given modem boot took longer than
usual and we failed probing while waiting for it to work properly).
Debug logs will tell you in detail why PPP is chosen instead of
QMI+NET.
--
Aleksander
https://aleksander.es
FCC unlock surprises after upgrading (was: Re: ANN: ModemManager 1.18.4 released)
Aleksander Morgado writes: > * A new FCC unlock operation management via external scripts is >introduced, which will avoid to automatically unlock FCC locked >devices unless the user has configured the operation manually, or >unless an official vendor-provided FCC unlock tool is found in the >system. Sorry for late comments again, but I'm worrying about the user experience for those upgrading a previously working system. I did read this announcement and the discussion about the feature back in november, and thought it sounded all fine. And I upgraded to 1.18.4 right before christmas when the Debian package showed up. Still didn't think much about this, and obviously didn't read out of the NEWS file that I was supposed to do something... Now I don't suspend or reboot my laptop often with Corona office and all that. But today I did. And was very surprised when MM failed to connect after resuming. Manually trying to enable the modem failed with a message which didn't immediately ring any bells: root@miraculix:/home/bjorn# mmcli -m any -e error: couldn't enable the modem: 'GDBus.Error:org.freedesktop.ModemManager1.Error.Core.Retry: Invalid transition' I must admit that I had to take a detour via AT commands, looking at AT!PCINFO? to realize that this was a failure to do the FCC unlock. Only after that did it occur to me that I should read those instructions again. But that was only because I had seen then before only a few weeks ago. I'm not sure everyone else would make the connection. Or maybe they would, and I'm the only slow one here :-) Fixing the problem was of course as simple as cd /etc/ModemManager/fcc-unlock.d ln -s /usr/share/ModemManager/fcc-unlock.available.d/1199\:9079 and everything worked as expected after that. I understand that you want/need to make this a concious decision for new users and modems. But I wonder if we really have to make it this difficult for existing users? After all, I am spoiled by 5+ years of fully automatic unlock on this laptop/modem. Why should I have to manually re-enable it now? Maybe we could accept a script adding the symlink on upgrades from older MM versions, on systems with an affected modem? Or at least a subset of older modems? Or maybe this isn't a big problem? After all, it's just a one time thing. I only hope people figure it out quicker than me... Bjørn
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
On 12/30/21 8:28 AM, Aleksander Morgado wrote:
Hey,
# AWC EM12-AW
# ttyUSB0 (if #0): QCDM/DIAG port
# ttyUSB1 (if #1): GPS data port
# ttyUSB2 (if #2): AT primary port
# ttyUSB3 (if #3): AT secondary port
ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="00",
SUBSYSTEM=="tty", ENV{ID_MM_PORT_TYPE_QCDM}="1"
ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="01",
SUBSYSTEM=="tty", ENV{ID_MM_DEVICE_IGNORE}="1"
ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="02",
SUBSYSTEM=="tty", ENV{ID_MM_DEVICE_IGNORE}="1
ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="03",
SUBSYSTEM=="tty", ENV{ID_MM_DEVICE_IGNORE}="1"
Should be ID_MM_PORT_IGNORE in all those (not DEVICE_iGNORE).
I see. So the problem is that I have to keep port 02 open for mmcli commands,
but if I do it'll try to use this for PPP still sometimes.
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
Hey,
> ACTION!="add|change|move|bind", GOTO="mm_awc_port_types_end"
> SUBSYSTEMS=="usb", ATTRS{idVendor}=="2c7c", GOTO="mm_awc_port_types"
> GOTO="mm_awc_port_types_end"
>
> LABEL="mm_awc_port_types"
>
> SUBSYSTEMS=="usb", ATTRS{bInterfaceNumber}=="?*",
> ENV{.MM_USBIFNUM}="$attr{bInterfaceNumber}"
>
> # AWC EM12-AW
> # ttyUSB0 (if #0): QCDM/DIAG port
> # ttyUSB1 (if #1): GPS data port
> # ttyUSB2 (if #2): AT primary port
> # ttyUSB3 (if #3): AT secondary port
> ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="00",
> SUBSYSTEM=="tty", ENV{ID_MM_PORT_TYPE_QCDM}="1"
> ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="01",
> SUBSYSTEM=="tty", ENV{ID_MM_DEVICE_IGNORE}="1"
> ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="02",
> SUBSYSTEM=="tty", ENV{ID_MM_DEVICE_IGNORE}="1
> ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="03",
> SUBSYSTEM=="tty", ENV{ID_MM_DEVICE_IGNORE}="1"
>
Should be ID_MM_PORT_IGNORE in all those (not DEVICE_iGNORE).
--
Aleksander
https://aleksander.es
Re: ANN: ModemManager 1.18.4 released / Avoiding PPP
On 12/3/21 9:52 AM, Aleksander Morgado wrote:
Hey,
And two, that we never, ever try to use PPP. To that end,
I may need to fashion a patch to make this so - maybe just a runtime
option to stop it being used. Actually removing it from the build is
more work than I care to take on.
You can always ID_MM_PORT_IGNORE the TTYs :D
Sorry for the delay. This is what I ended up using. It still
seems that sometimes it'll attempt PPP. Is this correct?
ACTION!="add|change|move|bind", GOTO="mm_awc_port_types_end"
SUBSYSTEMS=="usb", ATTRS{idVendor}=="2c7c", GOTO="mm_awc_port_types"
GOTO="mm_awc_port_types_end"
LABEL="mm_awc_port_types"
SUBSYSTEMS=="usb", ATTRS{bInterfaceNumber}=="?*",
ENV{.MM_USBIFNUM}="$attr{bInterfaceNumber}"
# AWC EM12-AW
# ttyUSB0 (if #0): QCDM/DIAG port
# ttyUSB1 (if #1): GPS data port
# ttyUSB2 (if #2): AT primary port
# ttyUSB3 (if #3): AT secondary port
ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="00",
SUBSYSTEM=="tty", ENV{ID_MM_PORT_TYPE_QCDM}="1"
ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="01",
SUBSYSTEM=="tty", ENV{ID_MM_DEVICE_IGNORE}="1"
ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="02",
SUBSYSTEM=="tty", ENV{ID_MM_DEVICE_IGNORE}="1
ATTRS{idVendor}=="2c7c", ATTRS{idProduct}=="0512", ENV{.MM_USBIFNUM}=="03",
SUBSYSTEM=="tty", ENV{ID_MM_DEVICE_IGNORE}="1"
Re: ANN: ModemManager 1.18.4 released
Hey, > >> However, and by all means convince me otherwise, I could still see > >> the fall though case to PPP happening when things are not quite ready > >> with the USB stack, QMI driver, etc. And that's a real problem for me. > > > > Here we're relying on two things: the kernel exposes all ports > > (control and data) of a given single device in a certain amount of > > time; plus, the modem makes those exposed ports functional in a > > certain amount of time. If for any reason those things go wrong, the > > MM probing will fail. > > No argument here. I understand that timeouts must happen and things > take different amounts of time. > > However, for my setup, two things are critical - one, that it keeps > trying indefinitely. Hence my suggested OpenWrt patches earlier > this year. I don't know if those were integrated or not; they > were met with some derision at the time. > Do you have links to those patches? > And two, that we never, ever try to use PPP. To that end, > I may need to fashion a patch to make this so - maybe just a runtime > option to stop it being used. Actually removing it from the build is > more work than I care to take on. You can always ID_MM_PORT_IGNORE the TTYs :D -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released
On 12/3/21 3:48 AM, Aleksander Morgado wrote: Hey, However, and by all means convince me otherwise, I could still see the fall though case to PPP happening when things are not quite ready with the USB stack, QMI driver, etc. And that's a real problem for me. Here we're relying on two things: the kernel exposes all ports (control and data) of a given single device in a certain amount of time; plus, the modem makes those exposed ports functional in a certain amount of time. If for any reason those things go wrong, the MM probing will fail. No argument here. I understand that timeouts must happen and things take different amounts of time. However, for my setup, two things are critical - one, that it keeps trying indefinitely. Hence my suggested OpenWrt patches earlier this year. I don't know if those were integrated or not; they were met with some derision at the time. And two, that we never, ever try to use PPP. To that end, I may need to fashion a patch to make this so - maybe just a runtime option to stop it being used. Actually removing it from the build is more work than I care to take on.
Re: ANN: ModemManager 1.18.4 released
Hey, > > Aleksander Morgado wrote on Fri, Dec 03, 2021 at 11:02:12AM +0100: > > I'll give it some time before a new release I think; this issue should > > affect openwrt only, and there was no bump to 1.18.4 there yet. > > I've missed why these udev rules should only affect openwrt? > procd doesn't parse patterns the same way as udev/eudev do? it seems to > be globbing is fairly standard from the links you pointed at in the > commit message? procd doesn't parse those rules when in openwrt; it's ModemManager itself the one doing that. > > If that is indeed specific to openwrt I think that's fine, they can > probably pick a single patch easily. > The fix in the udev rule is to avoid treating the pattern as a regex while it really is a shell globbing pattern. For non-openwrt distributions that was not an issue, the matches were still happening even if the pattern given was not exactly what we thought. For openwrt this was an issue, because the changes done in the custom udev parsing rules during 1.18.4 couldn't handle the pattern properly. So, I just simplified the udev rule pattern, which still will work when using the real udev, but which also makes it work properly with the custom udev parsing logic. -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released
Thanks for the quick reply! Aleksander Morgado wrote on Fri, Dec 03, 2021 at 11:02:12AM +0100: > I'll give it some time before a new release I think; this issue should > affect openwrt only, and there was no bump to 1.18.4 there yet. I've missed why these udev rules should only affect openwrt? procd doesn't parse patterns the same way as udev/eudev do? it seems to be globbing is fairly standard from the links you pointed at in the commit message? If that is indeed specific to openwrt I think that's fine, they can probably pick a single patch easily. Cheers, -- Dominique
Re: ANN: ModemManager 1.18.4 released
Hey Dominique > > > > Could you please test the following single patch on top of 1.18.4? > > > > https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/700 > > > > > > I ran through a couple of test cases and this appear to be doing the right > > > thing. Thanks. > > > > > > > Thanks for testing, will merge and backport to 1.18.x > > Will there also be a new tag quickly, or is that considered too rare of > a bug and will wait until the next cycle? > > (wondering if I should wait to upgrade the package on $distro; I > personally don't think my usecase will have a problem with these rules > but better play it safe when it involves more people...) > I'll give it some time before a new release I think; this issue should affect openwrt only, and there was no bump to 1.18.4 there yet. @Nick B when we do that bump, we should provide the fix as an extra patch. -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released
Hello, Aleksander Morgado wrote on Thu, Dec 02, 2021 at 04:24:39PM +0100: > > > Could you please test the following single patch on top of 1.18.4? > > > https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/700 > > > > I ran through a couple of test cases and this appear to be doing the right > > thing. Thanks. > > > > Thanks for testing, will merge and backport to 1.18.x Will there also be a new tag quickly, or is that considered too rare of a bug and will wait until the next cycle? (wondering if I should wait to upgrade the package on $distro; I personally don't think my usecase will have a problem with these rules but better play it safe when it involves more people...) -- Dominique Martinet
Re: ANN: ModemManager 1.18.4 released
Hey, > > The udev rules themselves are robust. The problem may be the time > > required by the module to expose the ports in the system, and the time > > required by the module to actually reply anything in those ports. > > Those two last things are handled by timeouts in ModemManager, and if > > there are race conditions leading to falling back to PPP, the things > > to fix would be those timeouts usually. > > Certainly. And that's precisely what I meant. There's various systems > in play, with different timeouts and what not. > > > The issue we've seen here has been a bad interpretation of how the > > udev rules are written, and on top of that, the fact that for openwrt > > we have a custom rules parser that is far from perfect. A solution to > > solve all this would be to fully avoid using udev rules, and provide > > our own setup rules for both udev and non-udev systems. The own setup > > could be equivalent to what we already do with the custom parser, just > > specifying which are the very specific types of matches we support, > > for example. If someone wants to work cleaning that up let me know :) > > Well yes. And I agree, an integrated udev solution is usually better, > and avoids steps and extra processes and what not. This is probably > non-trivial though, given the system has to support non-udev usage too > (default OpenWrt setups need a bunch of stuff turned on for udev to even > work in my experience). What I mean is, in OpenWRT we already have a "fake udev rules processor" that reads the udev rules and applies them in runtime when the port is detected. That's already working. My suggestion here is to remove the "real udev rules support" so that we just rely on our custom rules processor everywhere. If we have to make it work in openwrt, there's no reason why it should not also work in the same way for setups with real udev. > > However, and by all means convince me otherwise, I could still see > the fall though case to PPP happening when things are not quite ready > with the USB stack, QMI driver, etc. And that's a real problem for me. Here we're relying on two things: the kernel exposes all ports (control and data) of a given single device in a certain amount of time; plus, the modem makes those exposed ports functional in a certain amount of time. If for any reason those things go wrong, the MM probing will fail. The only thing we could do to avoid this is to extend the timeouts, but by doing so, we're also imposing a penalty on other devices for which we don't have port type hints and require full port probing. In other words, we should not increase the timeouts to extremely high values just to wait enough for the modem to reply or for the kernel to expose the ports, as that will break other use cases. We've seen the newer PCI based devices take up to 40s to reply anything in the control ports since the ports are exposed, which is really bad, and forces us to have long timeouts :/ So, yes, it's a real problem. The setup cannot be fully reliable because we assume behavior from other components that is also not reliable, we just try to mitigate those behaviors in MM as much as possible. But hey, open to suggestions on how to improve this, it's been an ongoing process in the past 10 years. If you ask me, the current setup is very stable, apart from unfortunate bugs like the one that did bite you :) -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released
On 12/2/21 4:46 PM, Aleksander Morgado wrote: The udev rules themselves are robust. The problem may be the time required by the module to expose the ports in the system, and the time required by the module to actually reply anything in those ports. Those two last things are handled by timeouts in ModemManager, and if there are race conditions leading to falling back to PPP, the things to fix would be those timeouts usually. Certainly. And that's precisely what I meant. There's various systems in play, with different timeouts and what not. The issue we've seen here has been a bad interpretation of how the udev rules are written, and on top of that, the fact that for openwrt we have a custom rules parser that is far from perfect. A solution to solve all this would be to fully avoid using udev rules, and provide our own setup rules for both udev and non-udev systems. The own setup could be equivalent to what we already do with the custom parser, just specifying which are the very specific types of matches we support, for example. If someone wants to work cleaning that up let me know :) Well yes. And I agree, an integrated udev solution is usually better, and avoids steps and extra processes and what not. This is probably non-trivial though, given the system has to support non-udev usage too (default OpenWrt setups need a bunch of stuff turned on for udev to even work in my experience). However, and by all means convince me otherwise, I could still see the fall though case to PPP happening when things are not quite ready with the USB stack, QMI driver, etc. And that's a real problem for me.
Re: ANN: ModemManager 1.18.4 released
> >>> Could you please test the following single patch on top of 1.18.4? > >>> https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/700 > >>> > >> > >> I ran through a couple of test cases and this appear to be doing the right > >> thing. Thanks. > >> > > > > Thanks for testing, will merge and backport to 1.18.x > > > > It does occur to me that, udev rules not being the most robust things (In my > experience anyway), that there could be conditions where devices being > reported > are not quite ready, leading to certain race conditions, or at least the > occasional, in this case, still deciding to use PPP. > > This is circumstantial speculation on my part of course, but it fits with > what I've seen. You'll have to comment on the exact behavior. > The udev rules themselves are robust. The problem may be the time required by the module to expose the ports in the system, and the time required by the module to actually reply anything in those ports. Those two last things are handled by timeouts in ModemManager, and if there are race conditions leading to falling back to PPP, the things to fix would be those timeouts usually. The issue we've seen here has been a bad interpretation of how the udev rules are written, and on top of that, the fact that for openwrt we have a custom rules parser that is far from perfect. A solution to solve all this would be to fully avoid using udev rules, and provide our own setup rules for both udev and non-udev systems. The own setup could be equivalent to what we already do with the custom parser, just specifying which are the very specific types of matches we support, for example. If someone wants to work cleaning that up let me know :) -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released
On 12/2/21 10:24 AM, Aleksander Morgado wrote: Hey Could you please test the following single patch on top of 1.18.4? https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/700 I ran through a couple of test cases and this appear to be doing the right thing. Thanks. Thanks for testing, will merge and backport to 1.18.x It does occur to me that, udev rules not being the most robust things (In my experience anyway), that there could be conditions where devices being reported are not quite ready, leading to certain race conditions, or at least the occasional, in this case, still deciding to use PPP. This is circumstantial speculation on my part of course, but it fits with what I've seen. You'll have to comment on the exact behavior.
Re: ANN: ModemManager 1.18.4 released
Hey > > Could you please test the following single patch on top of 1.18.4? > > https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/700 > > > > I ran through a couple of test cases and this appear to be doing the right > thing. Thanks. > Thanks for testing, will merge and backport to 1.18.x -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released
On 12/1/21 4:11 PM, Aleksander Morgado wrote: Could you please test the following single patch on top of 1.18.4? https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/700 I ran through a couple of test cases and this appear to be doing the right thing. Thanks.
Re: ANN: ModemManager 1.18.4 released
Hey Peter, > > > Wondering, would you be able to also test 1.18.4 but with these 2 > > > changes reverted? The regex related fix was a big one, maybe it broke > > > something else. > > > > > > 2beae71a6 kerneldevice,generic: simplify DEVPATH matching logic > > > 1bd70df8a kerneldevice,generic: input pattern to string match is not regex > > > > > > If you could get debug logs for both tests, that would be best > > > > It doesn't work with only the newer patch reversed, but with both it does. > > Both > > logs, respectively below. > > > > Thanks for testing this; I'll try to give it a run myself and see if I > can reproduce the problem. > Could you please test the following single patch on top of 1.18.4? https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/700 -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released
Hey Peter, > > Wondering, would you be able to also test 1.18.4 but with these 2 > > changes reverted? The regex related fix was a big one, maybe it broke > > something else. > > > > 2beae71a6 kerneldevice,generic: simplify DEVPATH matching logic > > 1bd70df8a kerneldevice,generic: input pattern to string match is not regex > > > > If you could get debug logs for both tests, that would be best > > It doesn't work with only the newer patch reversed, but with both it does. > Both > logs, respectively below. > Thanks for testing this; I'll try to give it a run myself and see if I can reproduce the problem. -- Aleksander https://aleksander.es
Re: ANN: ModemManager 1.18.4 released
On 12/1/21 5:11 AM, Aleksander Morgado wrote: Hey Peter Wondering, would you be able to also test 1.18.4 but with these 2 changes reverted? The regex related fix was a big one, maybe it broke something else. 2beae71a6 kerneldevice,generic: simplify DEVPATH matching logic 1bd70df8a kerneldevice,generic: input pattern to string match is not regex If you could get debug logs for both tests, that would be best It doesn't work with only the newer patch reversed, but with both it does. Both logs, respectively below. I'm just using syslog output here; if you need me to run on command line or whatever to get different debug, let me know. Mon Nov 29 15:40:56 2021 user.notice ModemManager: hotplug: add network interface ip6tnl0: event processed Mon Nov 29 15:40:56 2021 user.notice ModemManager: hotplug: error: parent device sysfspath not found Mon Nov 29 15:40:56 2021 user.notice ModemManager: hotplug: add network interface gre0: event processed Mon Nov 29 15:40:56 2021 user.notice : Added device handler type: bonding Mon Nov 29 15:40:56 2021 user.notice : Added device handler type: 8021ad Mon Nov 29 15:40:56 2021 user.notice : Added device handler type: 8021q Mon Nov 29 15:40:56 2021 user.notice : Added device handler type: macvlan Mon Nov 29 15:40:56 2021 user.notice : Added device handler type: veth Mon Nov 29 15:40:56 2021 user.notice : Added device handler type: bridge Mon Nov 29 15:40:56 2021 user.notice : Added device handler type: Network device Mon Nov 29 15:40:56 2021 user.notice : Added device handler type: tunnel Mon Nov 29 15:40:56 2021 user.notice ModemManager: hotplug: error: parent device sysfspath not found Mon Nov 29 15:40:57 2021 user.notice ModemManager: hotplug: add network interface gretap0: event processed Mon Nov 29 15:40:57 2021 user.notice ModemManager: hotplug: error: parent device sysfspath not found Mon Nov 29 15:40:58 2021 user.notice ModemManager: hotplug: add network interface erspan0: event processed Mon Nov 29 15:40:58 2021 user.notice ModemManager: hotplug: error: parent device sysfspath not found Mon Nov 29 15:40:59 2021 daemon.notice netifd: Interface 'lan' is enabled Mon Nov 29 15:40:59 2021 daemon.notice netifd: Interface 'lan' is setting up now Mon Nov 29 15:40:59 2021 daemon.notice netifd: Interface 'lan' is now up Mon Nov 29 15:40:59 2021 daemon.notice netifd: bridge 'br-lan' link is up Mon Nov 29 15:40:59 2021 daemon.notice netifd: Interface 'lan' has link connectivity Mon Nov 29 15:40:59 2021 daemon.notice netifd: VLAN 'eth0.1' link is up Mon Nov 29 15:40:59 2021 daemon.notice netifd: Interface 'loopback' is enabled Mon Nov 29 15:40:59 2021 daemon.notice netifd: Interface 'loopback' is setting up now Mon Nov 29 15:40:59 2021 daemon.notice netifd: Interface 'loopback' is now up Mon Nov 29 15:40:59 2021 daemon.notice netifd: Interface 'wwan' is setting up now Mon Nov 29 15:40:59 2021 daemon.notice netifd: Network device 'eth0' link is up Mon Nov 29 15:40:59 2021 daemon.notice netifd: Network device 'lo' link is up Mon Nov 29 15:40:59 2021 daemon.notice netifd: Interface 'loopback' has link connectivity Mon Nov 29 15:40:59 2021 user.notice ModemManager: hotplug: add network interface ip6gre0: event processed Mon Nov 29 15:40:59 2021 daemon.notice netifd: wwan (2493): error: couldn't get bus: Could not connect: No such file or directory Mon Nov 29 15:40:59 2021 daemon.notice netifd: wwan (2493): Device not managed by ModemManager Mon Nov 29 15:40:59 2021 user.notice ModemManager: hotplug: error: parent device sysfspath not found Mon Nov 29 15:40:59 2021 daemon.notice netifd: wwan (2531): stopping network Mon Nov 29 15:40:59 2021 user.notice firewall: Reloading firewall due to ifup of lan (br-lan) Mon Nov 29 15:40:59 2021 daemon.notice netifd: wwan (2531): error: couldn't find the ModemManager process in the bus Mon Nov 29 15:40:59 2021 daemon.notice netifd: wwan (2531): couldn't load bearer path Mon Nov 29 15:40:59 2021 daemon.notice netifd: Interface 'wwan' is now down Mon Nov 29 15:40:59 2021 daemon.err procd: failed to open pidfile for writing: : No such file or directory Mon Nov 29 15:41:00 2021 user.notice ModemManager: hotplug: add network interface wwan0: event processed Mon Nov 29 15:41:00 2021 user.notice ModemManager: hotplug: interface 'wwan' is set to configure device '/sys/devices/platform/1e1c.xhci/usb2/2-1' Mon Nov 29 15:41:00 2021 user.notice ModemManager: hotplug: now waiting for modem at sysfs path /sys/devices/platform/1e1c.xhci/usb2/2-1 Mon Nov 29 15:41:00 2021 user.notice ModemManager: hotplug: add cdc interface cdc-wdm0: custom event processed Mon Nov 29 15:41:00 2021 user.notice ucitrack: Setting up /etc/config/network reload dependency on /etc/config/dhcp Mon Nov 29 15:41:00 2021 user.notice ucitrack: Setting up /etc/config/wireless reload dependency on /etc/config/network Mon Nov 29 15:41:00 2021 daemon.err odhcpd[2086]: Failed to send to ff02::1%lan@br-lan (Address not available) Mon Nov 29
Re: ANN: ModemManager 1.18.4 released
On 11/29/21 11:12 AM, Dan Williams wrote: Do you have debug logs from MM? At the very least MM has to find a netdevice for the modem, otherwise it would fall back to an available AT port for PPP. Does MM find one? Here are two logs to compare. In both cases the kernel is 4.14.256; the only difference in the OpenWrt builds is the version of ModemManager. Correctly working in 1.18.2 is first, then not working - i.e, making a PPP connection in 1.18.4. Apologies for wrap, etc; hopefully this is sufficient debug. Wed Nov 24 13:17:54 2021 user.notice ModemManager: hotplug: add network interface ip6tnl0: event processed Wed Nov 24 13:17:54 2021 user.notice ModemManager: hotplug: error: parent device sysfspath not found Wed Nov 24 13:17:54 2021 user.notice ModemManager: hotplug: add network interface gre0: event processed Wed Nov 24 13:17:54 2021 user.notice : Added device handler type: bonding Wed Nov 24 13:17:54 2021 user.notice : Added device handler type: 8021ad Wed Nov 24 13:17:54 2021 user.notice : Added device handler type: 8021q Wed Nov 24 13:17:54 2021 user.notice : Added device handler type: macvlan Wed Nov 24 13:17:54 2021 user.notice : Added device handler type: veth Wed Nov 24 13:17:54 2021 user.notice : Added device handler type: bridge Wed Nov 24 13:17:54 2021 user.notice : Added device handler type: Network device Wed Nov 24 13:17:54 2021 user.notice : Added device handler type: tunnel Wed Nov 24 13:17:54 2021 user.notice ModemManager: hotplug: error: parent device sysfspath not found Wed Nov 24 13:17:55 2021 user.notice ModemManager: hotplug: add network interface gretap0: event processed Wed Nov 24 13:17:55 2021 user.notice ModemManager: hotplug: error: parent device sysfspath not found Wed Nov 24 13:17:56 2021 user.notice ModemManager: hotplug: add network interface erspan0: event processed Wed Nov 24 13:17:56 2021 user.notice ModemManager: hotplug: error: parent device sysfspath not found Wed Nov 24 13:17:57 2021 daemon.notice netifd: Interface 'lan' is enabled Wed Nov 24 13:17:57 2021 daemon.notice netifd: Interface 'lan' is setting up now Wed Nov 24 13:17:57 2021 daemon.notice netifd: Interface 'lan' is now up Wed Nov 24 13:17:57 2021 daemon.notice netifd: bridge 'br-lan' link is up Wed Nov 24 13:17:57 2021 daemon.notice netifd: Interface 'lan' has link connectivity Wed Nov 24 13:17:57 2021 daemon.notice netifd: VLAN 'eth0.1' link is up Wed Nov 24 13:17:57 2021 daemon.notice netifd: Interface 'loopback' is enabled Wed Nov 24 13:17:57 2021 daemon.notice netifd: Interface 'loopback' is setting up now Wed Nov 24 13:17:57 2021 daemon.notice netifd: Interface 'loopback' is now up Wed Nov 24 13:17:57 2021 daemon.notice netifd: Interface 'wwan' is setting up now Wed Nov 24 13:17:57 2021 daemon.notice netifd: Network device 'eth0' link is up Wed Nov 24 13:17:57 2021 daemon.notice netifd: Network device 'lo' link is up Wed Nov 24 13:17:57 2021 daemon.notice netifd: Interface 'loopback' has link connectivity Wed Nov 24 13:17:57 2021 user.notice ModemManager: hotplug: add network interface ip6gre0: event processed Wed Nov 24 13:17:57 2021 user.notice ModemManager: hotplug: error: parent device sysfspath not found Wed Nov 24 13:17:57 2021 daemon.notice netifd: wwan (2494): error: couldn't get bus: Could not connect: No such file or directory Wed Nov 24 13:17:57 2021 daemon.notice netifd: wwan (2494): Device not managed by ModemManager Wed Nov 24 13:17:57 2021 daemon.notice netifd: wwan (2544): stopping network Wed Nov 24 13:17:57 2021 user.notice firewall: Reloading firewall due to ifup of lan (br-lan) Wed Nov 24 13:17:57 2021 daemon.err procd: failed to open pidfile for writing: : No such file or directory Wed Nov 24 13:17:57 2021 daemon.notice netifd: wwan (2544): error: couldn't find the ModemManager process in the bus Wed Nov 24 13:17:57 2021 daemon.notice netifd: wwan (2544): couldn't load bearer path Wed Nov 24 13:17:57 2021 daemon.notice netifd: Interface 'wwan' is now down Wed Nov 24 13:17:58 2021 user.notice ModemManager: hotplug: add network interface wwan0: event processed Wed Nov 24 13:17:58 2021 user.notice ModemManager: hotplug: interface 'wwan' is set to configure device '/sys/devices/platform/1e1c.xhci/usb2/2-1' Wed Nov 24 13:17:58 2021 user.notice ModemManager: hotplug: now waiting for modem at sysfs path /sys/devices/platform/1e1c.xhci/usb2/2-1 Wed Nov 24 13:17:58 2021 user.notice ModemManager: hotplug: add cdc interface cdc-wdm0: custom event processed Wed Nov 24 13:17:58 2021 user.notice ucitrack: Setting up /etc/config/network reload dependency on /etc/config/dhcp Wed Nov 24 13:17:58 2021 user.notice ucitrack: Setting up /etc/config/wireless reload dependency on /etc/config/network Wed Nov 24 13:17:58 2021 daemon.err odhcpd[2093]: Failed to send to ff02::1%lan@br-lan (Address not available) Wed Nov 24 13:17:58 2021 user.notice ucitrack: Setting up /etc/config/firewall reload dependency on /etc/config/luci-splash Wed Nov 24 13:17:58
Re: ANN: ModemManager 1.18.4 released
On Mon, 2021-11-29 at 10:52 -0500, Peter Naulls wrote: > On 11/26/21 4:52 AM, Aleksander Morgado wrote: > > Hey hey, > > > > This is the second bugfix release in the 1.18.x series, built from > > the mm-1-18 > > branch. > > > > Thanks! > > I've been chasing, as I mentioned a while ago, where "sometimes" the > modem > connection will be brought up as PPP, not qmi. This kind of > connection fails > in our setup for a bunch of reasons, and is not desirable. > > I know that PPP is woven into ModemManager for several reasons, and > building > without is some work - I've seen several prior discussions on this > topic. > > I'm using a custom version of OpenWrt which is approximately the > current > release. I've mitigated the problem somewhat by removing the PPP > connection > scripts (/lib/netif/ppp*), which helps a little. I'm using > ModemManager > 1.18.2 presently, which seems to work pretty well against kernel > 4.14.256. > Some versions of the kernel (perhaps due to timing or whatever) seem > to > trigger the problem more often, although it's hard to confirm that > one > way or another. > > The QMI driver I have is highly customized, and may well be the root > of the > problem - recent kernels seem to have some validation around USB > devices > and whatnot. > > In any case, I tested 1.18.4 today, and 100% of the time the > connection > now comes up in PPP. It's not clear from the changes why this might > now happen - perhaps there's extra validation or something. But I'd > really like to get to the bottom of this - any suggestions in the > current > changes that I should look at? Do you have debug logs from MM? At the very least MM has to find a netdevice for the modem, otherwise it would fall back to an available AT port for PPP. Does MM find one? Dan
Re: ANN: ModemManager 1.18.4 released
On 11/26/21 4:52 AM, Aleksander Morgado wrote: Hey hey, This is the second bugfix release in the 1.18.x series, built from the mm-1-18 branch. Thanks! I've been chasing, as I mentioned a while ago, where "sometimes" the modem connection will be brought up as PPP, not qmi. This kind of connection fails in our setup for a bunch of reasons, and is not desirable. I know that PPP is woven into ModemManager for several reasons, and building without is some work - I've seen several prior discussions on this topic. I'm using a custom version of OpenWrt which is approximately the current release. I've mitigated the problem somewhat by removing the PPP connection scripts (/lib/netif/ppp*), which helps a little. I'm using ModemManager 1.18.2 presently, which seems to work pretty well against kernel 4.14.256. Some versions of the kernel (perhaps due to timing or whatever) seem to trigger the problem more often, although it's hard to confirm that one way or another. The QMI driver I have is highly customized, and may well be the root of the problem - recent kernels seem to have some validation around USB devices and whatnot. In any case, I tested 1.18.4 today, and 100% of the time the connection now comes up in PPP. It's not clear from the changes why this might now happen - perhaps there's extra validation or something. But I'd really like to get to the bottom of this - any suggestions in the current changes that I should look at? Thanks again.
ANN: ModemManager 1.18.4 released
Hey hey,
This is the second bugfix release in the 1.18.x series, built from the mm-1-18
branch.
ModemManager 1.18.4
---
* A new FCC unlock operation management via external scripts is introduced,
which will avoid to automatically unlock FCC locked devices unless the user has
configured the operation manually, or unless an official vendor-provided FCC
unlock tool is found in the system.
Please refer to the following URL for full details:
https://modemmanager.org/docs/modemmanager/fcc-unlock/
The following changes should be taken into account by distribution packagers:
** A set of FCC unlock scripts named as the specific vendor 'vid' will be
installed in ${datadir}/ModemManager/fcc-unlock.available.d.
** A set of symlinks is created named as the specific device 'vid:pid', and
pointing to the per-vendor 'vid' files, in the same location inside ${datadir}.
** A new ${sysconfdir}/ModemManager/fcc-unlock.d directory is created, where
users will manually install additional symlinks to the scripts shipped in
${datadir}.
** A new ${libdir}/ModemManager/fcc-unlock.d directory is created, where
vendors will install their own official FCC unlock tools.
** Both fcc-unlock.d directories should be empty on a new install, and their
contents (if any) should not be removed on ModemManager upgrades.
* build,meson:
** All optional settings in the build (e.g. introspection, systemd
suspend/resume support, udev...) are now booleans (true/false) instead of
features. Having dependencies as 'auto' has given problems since the very
beginning when using autotools, because sometimes the build goes on even if all
the features required by the user were not enabled. This change in the meson
build now requires the user to explicitly enable or disable features. This is
for consistency across all libqrtr-glib, libqmi, libmbim and ModemManager.
** If the build relies on enabling specific plugins, though, these are still
features (enable/disable/auto). The user may disable all plugins with
`-Dauto_features=disabled` and then enable specific plugins independently.
** Fixed the plugin dependency on build options.
* libmm-glib:
** The license text in several source files of the library is fixed to
reflect that they are LGPLv2+, as the whole library. This is not a relicense of
any kind, just a fix in the wording.
* modem interface:
** Always set error if creating device identifier fails.
* firmware interface:
** Fix missing context initialization leading to crash.
* profile manager interface:
** Fix incorrect GError ownership management.
* mbim:
** Fix crash when processing DNS addresses on a connection attempt.
** Fix incorrect GError ownership management when reloading connection
status.
* qmi:
** Fix non-initialized GError.
* sms:
** Merge WDP multipart CDMA WAP messages.
** Increase default send timeout from 180s to 300s.
* kerneldevice:
** Fix pattern matching logic in the custom udev rules parser, required to
correctly handle devices in the WWAN subsystem in kernel 5.13.
* plugins:
** qcom-soc: this plugin is now enabled by default, so that it can be used
on several new laptops based on Qualcomm SoCs.
** qcom-soc: updated to require the explicit ID_MM_QCOM_SOC udev tag.
** telit: fix critical message when attempting to unref a NULL object.
** telit: allow unlock retries loading for unknown error csim reply.
** telit: avoid sim hot swap procedure if #QSS is not supported.
** telit: add port type hints for LE910S1 0x7010, 0x7011 compositions.
** quectel: fix non-initialized GError.
** foxconn: added support for T99W265 modules.
* Several other improvements and fixes.
About ModemManager:
https://www.freedesktop.org/wiki/Software/ModemManager
Download here:
https://www.freedesktop.org/software/ModemManager/ModemManager-1.18.4.tar.xz
https://www.freedesktop.org/software/ModemManager/ModemManager-1.18.4.tar.xz.asc
Verify it:
$ sha256sum ModemManager-1.18.4.tar.xz
11fb970f63e2da88df4b6d8759e4ee649944c515244b979bf50a7a6df1d7f199
ModemManager-1.18.4.tar.xz
$ gpg --verify ModemManager-1.18.4.tar.xz.asc ModemManager-1.18.4.tar.xz
APIs and manpages here:
https://www.freedesktop.org/software/ModemManager/doc/1.18.0/ModemManager
https://www.freedesktop.org/software/ModemManager/doc/1.18.0/libmm-glib
https://www.freedesktop.org/software/ModemManager/man/1.18.0/ModemManager.8.html
https://www.freedesktop.org/software/ModemManager/man/1.18.0/mmcli.1.html
Please report bugs either to:
modemmanager-devel@lists.freedesktop.org
Or to gitlab:
https://gitlab.freedesktop.org/mobile-broadband/ModemManager/issues
--
Aleksander
https://aleksander.es
OpenPGP_0x3CAD53398973FFFA.asc
Description: OpenPGP public key
OpenPGP_signature
Description: OpenPGP digital signature
