Re: Issues regarding multiple MBIM bearer support
Aleksander Morgado writes: >> With master (83ac82470589a3672092a0ba0be855093b1cf5e2) I had problems. MM >> did want to enable multiplexing, but it failed for some reason, probably at >> sub-interface creation: >> >> 2021-06-16 10:04:05.630664 +00:00 test-host NetworkManager[457]: >> [1623837845.6302] modem["cdc-wdm0"]: modem state changed, >> 'registered' --> 'connecting' (reason: user-r >> equested) >> 2021-06-16 10:04:05.636959 +00:00 test-host ModemManager[455]: >> Using dynamic session ID 0 >> 2021-06-16 10:04:05.637483 +00:00 test-host ModemManager[455]: >> [modem0/bearer4] connection attempt #1 failed: failed to create net >> link for device: failed to add lin >> k for device: Could not allocate link: Failed to add link with session id 0: >> Netlink message with transaction 5 failed: Unknown error -1 >> 2021-06-16 10:04:05.638678 +00:00 test-host ModemManager[455]: >> [modem0] state changed (connecting -> registered) >> >> On the device I have 5.11.19 kernel with 802.1Q enabled. Again, I can >> provide you with detailed logs if this wasn't fixed already. >> > > Please, provide detailed logs, we need to understand why this failed > in the kernel. @Bjørn Mork any idea why it may have failed like this? > does the kernel need any additional thing on top of MBIM and VLAN > support? I have no idea what happened here. If that's an ENOENT error then I don't see any way it could be trggered. Bjørn ___ ModemManager-devel mailing list ModemManager-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/modemmanager-devel
Re: Issues regarding multiple MBIM bearer support
Hey, > > Just to let you know - I cherry-picked > e5305498b7ddbc5e419cf51fc0c30aa6593a40d2 over 1.16.6 tag, and it did fix the > VLAN issue - new bearers objects appear, but all use session 0. > Without the fix it didn't - the issue was caught exactly on vanilla 1.16.6, > but we didn't gather the logs at the time. > I've cherry-picked that commit for MM 1.16. > I prefer this over updating configuration on units deployed in the field. I > can still provide you with the logs if you wish. > What configuration would you need to update? > With master (83ac82470589a3672092a0ba0be855093b1cf5e2) I had problems. MM did > want to enable multiplexing, but it failed for some reason, probably at > sub-interface creation: > > 2021-06-16 10:04:05.630664 +00:00 test-host NetworkManager[457]: > [1623837845.6302] modem["cdc-wdm0"]: modem state changed, > 'registered' --> 'connecting' (reason: user-r > equested) > 2021-06-16 10:04:05.636959 +00:00 test-host ModemManager[455]: Using > dynamic session ID 0 > 2021-06-16 10:04:05.637483 +00:00 test-host ModemManager[455]: > [modem0/bearer4] connection attempt #1 failed: failed to create net > link for device: failed to add lin > k for device: Could not allocate link: Failed to add link with session id 0: > Netlink message with transaction 5 failed: Unknown error -1 > 2021-06-16 10:04:05.638678 +00:00 test-host ModemManager[455]: > [modem0] state changed (connecting -> registered) > > On the device I have 5.11.19 kernel with 802.1Q enabled. Again, I can provide > you with detailed logs if this wasn't fixed already. > Please, provide detailed logs, we need to understand why this failed in the kernel. @Bjørn Mork any idea why it may have failed like this? does the kernel need any additional thing on top of MBIM and VLAN support? Also, by default the multiplexing setup is "requested" (not "required") and so, if for any reason it fails to setup multiplexing, we should allow setting it up as a non-multiplexed session. In your case, we should have fallen back without breaking the connection attempt. I need to work on that in the next few weeks, so if it's not much to ask, I may ping you again once I have a fix to see if you could retest it... :) -- Aleksander https://aleksander.es ___ ModemManager-devel mailing list ModemManager-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/modemmanager-devel
Re: Issues regarding multiple MBIM bearer support
Hi Aleksander, Just to let you know - I cherry-picked e5305498b7ddbc5e419cf51fc0c30aa6593a40d2 over 1.16.6 tag, and it did fix the VLAN issue - new bearers objects appear, but all use session 0. Without the fix it didn't - the issue was caught exactly on vanilla 1.16.6, but we didn't gather the logs at the time. I prefer this over updating configuration on units deployed in the field. I can still provide you with the logs if you wish. With master (83ac82470589a3672092a0ba0be855093b1cf5e2) I had problems. MM did want to enable multiplexing, but it failed for some reason, probably at sub-interface creation: 2021-06-16 10:04:05.630664 +00:00 test-host NetworkManager[457]: [1623837845.6302] modem["cdc-wdm0"]: modem state changed, 'registered' --> 'connecting' (reason: user-r equested) 2021-06-16 10:04:05.636959 +00:00 test-host ModemManager[455]: Using dynamic session ID 0 2021-06-16 10:04:05.637483 +00:00 test-host ModemManager[455]: [modem0/bearer4] connection attempt #1 failed: failed to create net link for device: failed to add lin k for device: Could not allocate link: Failed to add link with session id 0: Netlink message with transaction 5 failed: Unknown error -1 2021-06-16 10:04:05.638678 +00:00 test-host ModemManager[455]: [modem0] state changed (connecting -> registered) On the device I have 5.11.19 kernel with 802.1Q enabled. Again, I can provide you with detailed logs if this wasn't fixed already. Kind regards, Lech W dniu 15.06.2021 o 21:26, Aleksander Morgado pisze: > Hey Lech, > > > > > Recently, I was investigating connectivity issues with Telit LE910-EU V2 in > > MBIM mode, when working in roaming. At the time, I was using ModemManager > > 1.16.5. > > > > In order to go further while waiting for fix on the modem side - I made a > > workaround, which led me to discovery of a few issues regarding multiple > > MBIM bearer support. > > > > Long story short: > > - At time of ModemManager 1.6.x, the modem was establishing connection > > properly most of the time - it was hard to reproduce the connection problem > > at all > > - With ModemManager 1.16.x, the modem would sometimes fail at "packet > > service attach" step, sometimes at "connect" step. > > Can you confirm that when using MM 1.6.x you didn't see the "packet > service attach" error? That would be interesting. I don't truly recall > how many changes in the MBIM connection sequence were done between 1.6 > and 1.16, but that could give us some hints to debug that further. > > > - Discussion with operator and Telit yielded information, that connections > > failed at "connect" step, because a second PDN session was requested with > > the same APN > > - ModemManager 1.6.x always shown only one bearer in the list of beares > > through mmcli. > > - ModemManager 1.16x (which gained default EPS bearer setting - which is > > supposedly not supported in this modem) sometimes shown multiple bearers - > > always a "bearer 0" in addition to some other bearer, often greater than > > one. > > That should totally be unrelated to any of your issues. The initial > EPS bearer status object is maintained out of the "bearer list" object > that tracks the different connection attempts with different settings. > > I recall having fixed some issues more or less recently that solved a > problem like yours, but those fixes were definitely already included > in 1.16.4, so maybe there is some other issue somewhere. The fixes > were 4f7ee18d1cf1ce21bba3f2701e408ae83108b3de and > 99bffc03cff060a752c0c9a4af0e7d06847c0c00, both in libmm-glib; due to > those problems comparing the bearer properties would always fail even > if they were equal, and that would have triggered creating a new > bearer with the same settings, instead of reusing the last one. > > > - The cause of failures at "Packet service attach" step is still not known > > at this step, but issue is also reproducible using pure mbimcli commands. > > Issuing "detach packet service" would restore the modem to operation, > > - I tried to implement this workaround inside ModemManager (which seems to > > be exactly the same as [1], despite being different modem manufacturer - > > but with exactly same Intel chipset), but it would not work for 100% time. > > That issue rings a bell; i don't think we had a proper solution out of > that CGACTT=0 hack. > > > - I implemented another dirty workaround - to re-register the modem in the > > network, by executing 'AT+COPS=2' and 'AT+COPS=0,0" through mmcli in > > succession - it came out during debugging, and trying to gather as many > &g
Re: Issues regarding multiple MBIM bearer support
Hey Lech, > > Recently, I was investigating connectivity issues with Telit LE910-EU V2 in > MBIM mode, when working in roaming. At the time, I was using ModemManager > 1.16.5. > > In order to go further while waiting for fix on the modem side - I made a > workaround, which led me to discovery of a few issues regarding multiple MBIM > bearer support. > > Long story short: > - At time of ModemManager 1.6.x, the modem was establishing connection > properly most of the time - it was hard to reproduce the connection problem > at all > - With ModemManager 1.16.x, the modem would sometimes fail at "packet service > attach" step, sometimes at "connect" step. Can you confirm that when using MM 1.6.x you didn't see the "packet service attach" error? That would be interesting. I don't truly recall how many changes in the MBIM connection sequence were done between 1.6 and 1.16, but that could give us some hints to debug that further. > - Discussion with operator and Telit yielded information, that connections > failed at "connect" step, because a second PDN session was requested with the > same APN > - ModemManager 1.6.x always shown only one bearer in the list of beares > through mmcli. > - ModemManager 1.16x (which gained default EPS bearer setting - which is > supposedly not supported in this modem) sometimes shown multiple bearers - > always a "bearer 0" in addition to some other bearer, often greater than one. That should totally be unrelated to any of your issues. The initial EPS bearer status object is maintained out of the "bearer list" object that tracks the different connection attempts with different settings. I recall having fixed some issues more or less recently that solved a problem like yours, but those fixes were definitely already included in 1.16.4, so maybe there is some other issue somewhere. The fixes were 4f7ee18d1cf1ce21bba3f2701e408ae83108b3de and 99bffc03cff060a752c0c9a4af0e7d06847c0c00, both in libmm-glib; due to those problems comparing the bearer properties would always fail even if they were equal, and that would have triggered creating a new bearer with the same settings, instead of reusing the last one. > - The cause of failures at "Packet service attach" step is still not known at > this step, but issue is also reproducible using pure mbimcli commands. > Issuing "detach packet service" would restore the modem to operation, > - I tried to implement this workaround inside ModemManager (which seems to be > exactly the same as [1], despite being different modem manufacturer - but > with exactly same Intel chipset), but it would not work for 100% time. That issue rings a bell; i don't think we had a proper solution out of that CGACTT=0 hack. > - I implemented another dirty workaround - to re-register the modem in the > network, by executing 'AT+COPS=2' and 'AT+COPS=0,0" through mmcli in > succession - it came out during debugging, and trying to gather as many logs > from modem at the time, that this actually helps to establish the connection > at the PDN level. That would make the connection attempt quite slow, doesn't it? > - When this is executed, but the modem is already registered in home network, > this expectedly causes a dropped connection, but then MM detects this > situation and reacts accordingly, quickly re-establishing the connection. > This is very racey, but a similar situation could be triggered by an external > factor, like the operator dropping a PDN connection. > MM does not automatically reconnect; you probably have NM doing that autoreconnection for you. > And finally, we come to the issues in ModemManager, that I discovered because > of this: > - When reconnecting after triggering the issue, ModemManager always ends up > with two bearers associated with the modem, one of which is disconnected and > has no IP configuration. > MM 1.6.x did not, and this is first issue - I suspect that a bearer cleanup > in 1.16.x is missing on modem de-registration forced by AT command. Fixing > this would most probably fix our usecase, because we use only a single PDN > connection. > I will try with master as well, as it might be already fixed there. Please open a bug in gitlab and provide steps to reproduce and a full debug log of the daemon: https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/issues/ Also, please make sure you attempt to reproduce with 1.16.6 at least (and git master if possible) Also, please note one thing. If you ask NetworkManager to request a connection without specifying IP type, it will try IPv4v6 first, if that fails it will try IPv4-only, and if that fails it will try IPv6-pnly. If your connection attempt ends up with one bearer disconnected and one connected, maybe t
Issues regarding multiple MBIM bearer support
Hello, Recently, I was investigating connectivity issues with Telit LE910-EU V2 in MBIM mode, when working in roaming. At the time, I was using ModemManager 1.16.5. In order to go further while waiting for fix on the modem side - I made a workaround, which led me to discovery of a few issues regarding multiple MBIM bearer support. Long story short: - At time of ModemManager 1.6.x, the modem was establishing connection properly most of the time - it was hard to reproduce the connection problem at all - With ModemManager 1.16.x, the modem would sometimes fail at "packet service attach" step, sometimes at "connect" step. - Discussion with operator and Telit yielded information, that connections failed at "connect" step, because a second PDN session was requested with the same APN - ModemManager 1.6.x always shown only one bearer in the list of beares through mmcli. - ModemManager 1.16x (which gained default EPS bearer setting - which is supposedly not supported in this modem) sometimes shown multiple bearers - always a "bearer 0" in addition to some other bearer, often greater than one. - The cause of failures at "Packet service attach" step is still not known at this step, but issue is also reproducible using pure mbimcli commands. Issuing "detach packet service" would restore the modem to operation, - I tried to implement this workaround inside ModemManager (which seems to be exactly the same as [1], despite being different modem manufacturer - but with exactly same Intel chipset), but it would not work for 100% time. - I implemented another dirty workaround - to re-register the modem in the network, by executing 'AT+COPS=2' and 'AT+COPS=0,0" through mmcli in succession - it came out during debugging, and trying to gather as many logs from modem at the time, that this actually helps to establish the connection at the PDN level. - When this is executed, but the modem is already registered in home network, this expectedly causes a dropped connection, but then MM detects this situation and reacts accordingly, quickly re-establishing the connection. This is very racey, but a similar situation could be triggered by an external factor, like the operator dropping a PDN connection. And finally, we come to the issues in ModemManager, that I discovered because of this: - When reconnecting after triggering the issue, ModemManager always ends up with two bearers associated with the modem, one of which is disconnected and has no IP configuration. MM 1.6.x did not, and this is first issue - I suspect that a bearer cleanup in 1.16.x is missing on modem de-registration forced by AT command. Fixing this would most probably fix our usecase, because we use only a single PDN connection. I will try with master as well, as it might be already fixed there. - Whether the connection actually works, depends on which bearer gets actually connected, and this is random. If it is "Bearer 0", then all is fine, IP connectivity works. If it is the other bearer, then connection does not work. The randomness here is the second issue. - Connectivity does not work in case of bearer different than 0, because, at MBIM level, ModemManager establishes a PDN session at ID different than "session 0", if not using "Bearer 0". According to Linux kernel documentation for cdc_mbim driver, such sessions are mapped as 802.1Q VLANs at the network interface level. [2] I confirmed this by manually creating VLAN sub-interface for the modem, with session number extracted through mbimcli, and moving the IP configuration there - this allowed the connectivity to work again. Here is a CLI dump confirming my hypothesis regarding erroneous VLAN mapping: # mmcli -m 0 General | path: /org/freedesktop/ModemManager1/Modem/0 | device id: b49ab6d9407e88df9ca805de471af5ac7e066589 Hardware | manufacturer: Telit | model: FIH7160 | firmware revision: 20.00.406 | h/w revision: XMM7160_V1.2_HWID665_MBIM_NAND | supported: gsm-umts, lte | current: gsm-umts, lte | equipment id: System | device: /sys/devices/platform/soc/3080.bus/30b2.usb/ci_hdrc.1/usb1/1-1/1-1.3 | drivers: cdc_acm, cdc_mbim | plugin: telit | primary port: cdc-wdm0 | ports: cdc-wdm0 (mbim), ttyACM0 (at), ttyACM1 (ignored), | ttyACM2 (ignored), ttyACM3 (at), ttyACM4 (ignored), ttyACM5 (ignored), | wwan0 (net) Status | unlock retri
