[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
Changed Status to FixCommitted, as this has landed in -proposed. ** Changed in: network-manager (Ubuntu) Status: Confirmed = Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
lxc-android-config (0.163) utopic; urgency=medium * add /etc/NetworkManager/dispatcher.d/02default_route_workaround to make sure the default route is always reset when an interface change is detected (LP: #1307981) ** Also affects: lxc-android-config (Ubuntu) Importance: Undecided Status: New ** Changed in: lxc-android-config (Ubuntu) Status: New = Fix Released ** Changed in: lxc-android-config (Ubuntu) Importance: Undecided = High ** Changed in: network-manager (Ubuntu) Status: Fix Committed = Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc-android-config/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
So a couple more comments... 1. Do we know if NM sees the mobile connection drop, and thus it re- activates the context again, or does the context appear active to NM the whole time and the connection drops and re-establishes itself without action from NM? My guess based upon Oliver's description is that the latter is happening. 2. Android is obviously able to handle devices that configure the routing table ( although I have seen references in the code that rild implementation should *not* do this ). Do we know how Android handles this? 3. How difficult would it be to change the NM policy so that we never have concurrent active mobile and Wi-Fi connections? Our current model is that we leave the mobile connection active when Wi-Fi is activated, and then just switch the default route to Wi-Fi once connected. One final thought along these lines, does this leave us in a situation where an app could have an active network session and continue to use mobile data when Wi-Fi is activated? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
on boot my routing is usually fine ... what i see today is: - boot with proper default route over WLAN, mobile is enabled but not owning the default route. - leave the phone next to me in the office or living room (both covered by the same WLAN) and randomly pick up the device to check G+ or read some news in a webapp - two or three times throughout the day notice that suddenly the app pops up the internal error page, checking the routing then via adb shows the two default routes, disabling and re-enabling WLAN then sets the route properly again. - while i *have* seen the routing being broken from boot on once or twice this is a very rare case, usually routing is fine on boot and breaks later (though pats comments above indicate that this is different for him) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
After some discussion with Mathieu yesterday, there's one addition to my previous comment. On some phones, and in this case mako, rild actually configures the routing table for the mobile network connection. As rild is a binary blob, and there's no parameters available to control this behavior, the NM ofono code was designed with this in mind. In the normal bringup case, this shouldn't be an issue as mobile data will be activated, the routes setup, followed by WiFi if enabled, and NM should be able to fixup the routing table correctly. I suspect what maybe happening is that the mobile connection might be dropping and then re-establishing itself, resulting in rild re-adding the default route when one already exists for Wi-Fi. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
That routing code is a red herring, it's actually never reached in the case where wifi is already activated, because wifi remains the best IP4 connection available (given that it has a higher priority than modems). From what I see here, it seems like rild is just *always* setting the route, and this is basically just a race between NM and rild: if rild happens to write the route before NM has activated the primary; then you'll get a valid routing table (because the rild route gets replaced by NM's route via wifi), but if it happens after, NM rightfully no longer cares about what happens to the routing. This *is* proper behavior since we can't go trash routes that could be added after the connection by a sysadmin or some other process... I'd very much like to understand why this suddently became an issue -- I don't think either code bases changed much in relation to routing, so something ought to have changed in the environment on mako. I see two options, listed in order of preference (not necessarily feasibility): 1) Convince rild to never set up the default route. I would have hope this to be doable via an android property that can be set, I tried to find a property that looked related, but without results yet. It could well not be feasible at all, sadly. 2) Ship a custom dispatcher script in lxc-android-config; to, on purpose, trash the boot routing table which rild abuses to do this. I would rather do this as a dispatcher script we ship only on touch, because I think it's explicitly Very Very Wrong to muck with the routing table arbitrarily for other images. I did already test this as a workaround with the expected successful result, see http://paste.ubuntu.com/7376744/. 3) Find a way to catch the precise, exact route using netlink in the ofono modem plugin of NM; and explicitly RTM_DELROUTE it. This is the safest to ship everywhere, but requires the most effort. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
I'm not sure what you mean by the routing code is a red herring? Also, I'd like to get the exact scenario nailed down. Oliver's description said that he had Wi-Fi enabled the whole time. If so, was the routing table *ever* correct, or was it broken from boot onwards? So which of the following two scenarios do we think is happening? After the initial device boot, it has a valid default route pointing at wlan and network access is enabled, then at some point rild comes along and blindly adds a new default route due to the data call dropping and being re-established. OR At boot, since WiFi is enabled, both the mobile context and the WiFi connection are started by NM, but the rild activation takes longer than WiFi, so rild winning the race and adds it's additional default route. As for the options listed above... 1. I could find no way to do this... 2. I'm not sure what you mean by trash the 'boot' routing table...? What what would trigger this script? Would this work for both scenarios mentioned above ( ie. race at initial boot, or mobile link dropping and connection being re-established )? 3. So the basic idea is that you'd add a routing table listener to the NM ofono code, and that this listener would delete a default route added for mobile data if a default route already existed? Isn't there an inherent race ( albeit a small one ) in this approach? I guess one final question is there anyway to configure our system routing policy to prevent this from being possible ( ie. prevent two default routes from being configured )? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
After the initial device boot, it has a valid default route pointing at wlan and network access is enabled, then at some point rild comes along and blindly adds a new default route due to the data call dropping and being re-established. This is only a problem when wifi is enabled, because otherwise you don't care what creates the default route for mobile, just that there is one. There are a few candidate properties for this kind of stuff. Digging more, it's not even quite rild that applies the route, but really more netmgrd (at least, it seems to me like it's the case). This doesn't change the fact that both are binary blobs from qualcomm or wherever, so not much to do about it. There are lots of rild and netmgrd properties though, it's just not well documented what each does. As for trashing the 'boot' table, it's a one-liner that would be triggered by NM directly on every device activation. NM doesn't touch the 'ip route list proto boot' routes, just kernel routes, so flushing that whole list is safe as long as we know there is nothing else that is expected to use it for a good reason -- that's something we can only say with any degree of confidence on Touch. As for doing this programmatically directly in NM, it's not at all a listener. I want to avoid doing any such kind of route listening, because NM already does it for other things elsewhere, and it's just not necessary. When the modem is activated, we can just try to delete a route defined to be a default gateway via the gateway address we already got from ofono, whether it exists or not. All of these options remain racy because there is no way to control rild/netmgrd. You can only hope they do their stuff right and don't take too long to run. You can't specify a routing policy that prevents two default routes, because this is something that is actually valid in some circumstances. The best you can do is either remove the routes after they are added, or prevent rild/netmgrd from adding the routes in the first place, and there currently doesn't seem to be a way to do the latter. I'm still looking at whether it's possible to change security policies to deal with this. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
The routing and in particular the switching of the default route is handled soley by NM. This was an issue last Summer during initial development of the NM ofono code, but Mathieu believed he'd resolved the issue. Apparently there's still some sort of race and it's possible that other recent system changes may have caused it to be triggered more easily. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: network-manager (Ubuntu) Status: New = Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
** Changed in: network-manager (Ubuntu) Assignee: (unassigned) = Mathieu Trudel-Lapierre (mathieu-tl) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
maybe red herring, but see this entry which probably indicates that stuff doesn't go as expected: Apr 28 16:55:42 ubuntu-phablet NetworkManager[1212]: error [1398718542.745922] [nm-system.c:965] add_ip4_route_to_gateway(): (rmnet_usb0): failed to add IPv4 route to gateway (-12) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
pat: the syslog you attached was captured when you were in that state? or after reboot/fixing? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
could be red herring, but guess that error could really make nm not do
proper default route replace...
in nm-system.c:1043 you see the code that will give up trying to replace
the gateway in case the add_ip4_route_to_gateway call fails...
/* Try adding a direct route to the gateway first */
gw_route = add_ip4_route_to_gateway (parent_ifindex, ext_gw,
parent_mss);
if (!gw_route)
goto out;
/* Try adding the original route again */
err = replace_default_ip4_route (ifindex, int_gw, mss);
if (err != 0) {
nm_netlink_route_delete (gw_route);
nm_log_err (LOGD_DEVICE | LOGD_VPN | LOGD_IP4,
(%s): failed to set IPv4 default route (pass #2):
%d,
iface, err);
} else
success = TRUE;
out:
if (gw_route)
rtnl_route_put (gw_route);
g_free (iface);
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1307981
Title:
[touch] randomly messed up routing with recent trusty images
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
sorry above i referenced wrong linenumber inside the vpn specific code ... but the same pattern is also around 1094... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
cypher/tony: maybe worth checking/debugging this code and making it more robust? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
ignore my posts in #10 and #11... i misread that code... however, the error in #9 seems to be in code that is only run if something with the normal default route replace didn't work... someone with device and closer to code probably should check this out. Maybe ofono or the modem driver also tries to do route magic which causes a racy situation here? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
Happened here after first flash and boot to u3 on mako phablet@ubuntu-phablet:~$ route Kernel IP routing table Destination Gateway Genmask Flags Metric RefUse Iface default 10.185.115.145 0.0.0.0 UG0 00 rmnet_usb0 default 10.0.1.10.0.0.0 UG0 00 wlan0 10.0.1.0* 255.255.255.0 U 9 00 wlan0 10.185.115.144 * 255.255.255.252 U 13 00 rmnet_usb0 phablet@ubuntu-phablet:~$ sudo ping www.google.com PING www.google.com (74.125.226.244) 56(84) bytes of data. ^C --- www.google.com ping statistics --- 13 packets transmitted, 0 received, 100% packet loss, time 12003ms phablet@ubuntu-phablet:~$ sudo ip route default via 10.185.115.145 dev rmnet_usb0 default via 10.0.1.1 dev wlan0 proto static 10.0.1.0/24 dev wlan0 proto kernel scope link src 10.0.1.39 metric 9 10.185.115.144/30 dev rmnet_usb0 proto kernel scope link src 10.185.115.144 metric 13 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
Same symptoms on 5 straight boots Turning wifi off allowed 3g to work Turning 3g off allowed wifi to work Disabled 3g data and then reboot wlan worked fine Enabled 3g data and reboot and back to the network not being reachable -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
** Attachment added: syslog https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+attachment/4100752/+files/syslog -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
i just noticed bug 1314410 do not rely on a webapp for testing if your connectivity is back (restarting the app works fine indeed, but the button on the error page nor the refresh option from the hud does not) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
ogra@styx:~/apps$ adb shell ip route default via 37.83.18.26 dev rmnet_usb1 default via 192.168.2.1 dev wlan0 proto static 37.83.18.24/30 dev rmnet_usb1 proto kernel scope link src 37.83.18.25 192.168.2.0/24 dev wlan0 proto kernel scope link src 192.168.2.78 metric 9 ogra@styx:~/apps$ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1307981] Re: [touch] randomly messed up routing with recent trusty images
If this happens again, could you get the same routing table but with the 'ip route' command instead? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1307981 Title: [touch] randomly messed up routing with recent trusty images To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1307981/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
