TLDR: I guess most of my problems are from calling ifup in the wrong state (CONNECTING instead of CONNECTED). More inline. I might bring some follow-up questions next week.
On 20 March 2017 at 19:33, Dan Williams <d...@redhat.com> wrote: > On Mon, 2017-03-20 at 12:14 +0100, Einar Jón wrote: >> Hello, >> > > Let me preface my response by saying I have no idea what ifup/ifdown do > on your platform, and whether they correctly handle teardown of the > interface and DHCP. > >> Our main issues: >> Operstate seems to be always UNKNOWN > > You can basically ignore operstate. You care about interface flags > instead (UP, DOWN, LOWER_UP, etc). I should have also mentioned that the interface flags are always UP, LOWER_UP with an operstate UNKNOWN. So ip addr/ifconfig is always the same for me. When using the Quectel UC20 with ModemManager - no matter what the state is: $ ip a show dev wwan0 4: wwan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000 link/ether 2a:eb:68:af:9e:1b brd ff:ff:ff:ff:ff:ff inet6 fe80::28eb:68ff:feaf:9e1b/64 scope link valid_lft forever preferred_lft forever Different OS using the Quectel UC20 without ModemManager (Using custom C code with AT commands): $ ip a show dev wwan0 5: wwan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 0a:f6:61:71:71:e4 brd ff:ff:ff:ff:ff:ff $ sudo rild -q & # this connects the modem [1] 1417 $ ip a show dev wwan0 5: wwan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 1000 link/ether 0a:f6:61:71:71:e4 brd ff:ff:ff:ff:ff:ff inet XX.YY.153.249/30 brd XX.YY.153.251 scope global wwan0 valid_lft forever preferred_lft forever > >> "ip link set dev wwan0 up" and "ip link set dev wwan0 down" seem to >> do nothing. > > Right. That's normal Linux. These set/clear the IFF_UP flag which > means "administratively up". Which means that when the interface is > UP, it can do IP and other stuff. When it's down, it can't. That's > all. It doesn't affect *what* addressing is done on the interface, > it's simply a pre-requisite for doing anything with the interface. Good to know... >> We need to reset the connection once a day (with mmcli --disable), >> because of external reasons. > > I presume the provider terminates the packet data session after 24h or > something like that? Not uncommon. Exactly why we need that. > >> This mostly works, but sometimes we lose connectivity after the >> reconnect. >> In those cases I have 2 IP addresses in the interface, and ip route >> tells me that both of them are gateways. > > Since MM doesn't do any IP address setup/teardown on the interface > itself, I wouldn't expect '--disable' alone to clean anything up. When > disabling, you'll need to either have a process monitor the modem state > and call "ifdown" or do some other kind of cleanup, or do the ifdown > along with the --disable. > >> "ifup wwan0 && ifdown wwan0" fixes the connection again without any >> action from the python script (this automatically starts a new >> dhclient and updates ip route). >> This seems to be just a timing issue, or a question of doing things >> in >> the right order. > > The order would probably be to disconnect the bearer (mmcli -m X -b Y > --disconnect), "ifdown wwan0", "mmcli -m X --disable". That's worth a try. > >> My main questions: >> 1) what do I need to do to have a reliable connection? > > That depends on quite a few things, many of them not ModemManager > related, so let's try to address them individually as they come up. > >> 2) when do I need to do those things? >> - in what state should I call ifup ? > > After the modem state becomes CONNECTED, or after the bearer > activation/connect returns successfully and you can retrieve the bearer > IP details from MM. CONNECTED state requires that the bearer > activation be successful, so if you see CONNECTED you should be fine. > >> - in what state should I call connect_bearer()? Any exceptions? > > Bearer-interface specific operations require the modem to be in the > right state (eg, something like REGISTERED). I would wait to connect a > bearer until the modem has found the network and registered. So CONNECTING is too soon. We currently try to connect_bearer when we reach REGISTERED. At least that's right... > > If you use the Simple interface instead, you can call Simple.Connect() > at any time other than FAILED state, as the Simple interface will do > everything it can to get connected, including enabling the modem, > waiting for registration, and then starting the bearer. That could > take a few minutes, but it's meant as a dead-simple interface. I would like to stick with what we have. > >> 3) I switch modem states really fast (from DISABLING to CONNECTING in >> under a second - see syslog below). Do some of these states need to >> wait a bit? > > ModemManager won't return from the D-Bus requests for > disabling/enabling/connecting/etc until it's ready; if it does so and > the modem hasn't fully entered that state, I'd consider it a bug. > > But I don't think that's your problem, see below. Probably just incorrect use. > >> 4) Is ModemManager 1.4.10 too old? Are there any major improvements >> I'm missing from staying on Ubuntu 12.04? > > Obviously we always recommend the most recent released version > (currently 1.6.x) but I can't think of anything specific that 1.4.x has > that you'd need in terms of bug fixes or features. Good to know. Upgrading would be a pain. >> NOTE: To find out the status of the connection, I check if the >> IfState >> is up or down (basically "grep wwan0 /run/network/ifstate"), instead >> of checking operstate (which is UNKNOWN no matter what I do). >> If ifstate is down I call "ifup wwan0" manually. >> When disconnecting (and on the pyhon startup), I call "ifdown wwan0" >> manually. > > This could be part of the problem. The network stack of your distro > (especially ifup/ifdown) has literally no idea what state the modem is > in. There is no real connection between the modem firmware state and > the Linux-side network interface; it's just a packet tunnel. Its > Linux-side state doesn't represent anything about the data connection > in the modem firmware itself. Right. I as an admin have literally no idea what state the modem is in. So this seemed to make sense at the time. > > While perhaps unfortunate, there are various reasons why this is the > case and I'm happy to explain further if you'd like. > > The point being: > > 1) when the modem enters the CONNECTED state, call "ifup wwan0" > 2) when the mode leaves the CONNECTED state, call "ifdown wwan0" and > wait for that to complete before doing anything else I guess I'm calling it too soon (in CONNECTING state), and I wait for ifup to finish before going to CONNECTED. This may be causing many of my issues. I did that because we have had a bearer_connect failure with PolicyMismatch that does an endless CONNECTING -> REGISTERED -> CONNECTING -> REGISTERED loop until the modem is reset. So I thought ifup would be needed for that. > don't bother checking the existing ifup/ifdown state because it may not > be accurate, because ifup/ifdown don't know anything about the actual > modem state. True. But it seems to be the only way to know if something is actually happening or not, since the And calling "ifdown wwan0; ifup wwan0" manually seems to fix the double IP issue. So it seemed like a good idea at the time. > > One last comment about the double-address thing; it's in the logs below > but I'll say it here instead. > > I don't think your process is waiting for "ifdown wwan0" to complete > before attempting to re-enable the modem. ifdown may take a short > period of time to clean things up, including letting DHCP release the > address. > > Mar 19 06:35:52 ifdown > Mar 19 06:35:52 re-enable > Mar 19 06:35:52 registered > Mar 19 06:35:53 dhclient is calling DISCOVER > Mar 19 06:36:53 dhclient gets a lease > Mar 19 06:37:16 ifup > Mar 19 06:37:17 finally CONNECTED state > > I think the old dhclient process is sticking around requesting a lease > long before you're at the point where you call 'ifup'. I'd suggest > waiting until the 'ifdown' completes before re-enabling the modem. > That is likely where your duplicate addresses are coming from. I don't think you are correct. What is actually happening: Mar 19 06:35:52 Modem state: CONNECTED -> DISABLING --- ifdown is called when we reach state DISABLING Mar 19 06:35:52 ifdown RETURNS and logs everything that happened since it was called Mar 19 06:35:52 Modem state: DISABLING -> DISABLED Mar 19 06:35:52 Modem state: DISABLED -> ENABLING Mar 19 06:35:52 Modem state: ENABLING -> REGISTERED Mar 19 06:35:52 Modem state: REGISTERED -> CONNECTING --- ifup is called when we reach state CONNECTING - this surely too soon Mar 19 06:35:53 dhclient is calling DISCOVER Mar 19 06:36:53 dhclient gets a lease Mar 19 06:37:16 ifup RETURNS and logs everything that happened since it was called at 06:35:52 Mar 19 06:37:17 finally CONNECTED state -- I should be calling ifup here Thanks for the detailed response. Einar > > Dan > >> Relevant python code: >> def handle_modem_state(self, oldState, state): >> logging.debug('Handle modem state, old: ' + >> self.modem_state_str(oldState) + ' new: ' + >> self.modem_state_str(state)) >> if state == ModemManager.ModemState.DISABLED: >> self.enable_modem() # calls modemProxy.Enable(1, ...) >> elif state == ModemManager.ModemState.ENABLED: >> self.register_modem() # calls modemProxy.Register(...) >> elif state == ModemManager.ModemState.SEARCHING: >> pass >> elif state == ModemManager.ModemState.REGISTERED: >> if oldState != ModemManager.ModemState.CONNECTING and >> oldState >> != ModemManager.ModemState.DISCONNECTING: >> self.connect_bearer() # get bearer from DBUS and connect >> elif state == ModemManager.ModemState.CONNECTED: >> self.triedPreferredOperator = False >> elif state == ModemManager.ModemState.CONNECTING: >> if not self.is_network_interface_up('wwan0'): # check >> operstate/ifstate >> self.set_network_state('wwan0', 'up') # calls ifup >> wwan0 in a new process >> elif state == ModemManager.ModemState.DISABLING: >> if self.is_network_interface_up('wwan0'): # check >> operstate/ifstate >> self.set_network_state('wwan0', 'down') # calls ifdown >> wwan0 in a new process >> ... snip ... -- Regards Einar Jón +31 610 957234 _______________________________________________ ModemManager-devel mailing list ModemManager-devel@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/modemmanager-devel