Hi Thomas, Glad you could provoke a very interesting discussion! But I'm still confused -- how is "sticky IP-based routing" going to work after the client roams to a new IP address (or to a new UDP source port)? When your system seems an incoming UDP datagram from a previously unseen source IP:port, how does it know which mosh-server (on which server machine) to send it to?
With off-the-shelf Mosh, you basically need a load-balancing strategy that allows a destination IP:port to uniquely identify a particular mosh-server. You can do this with multiple DNS A/AAAA records (where the client picks the winning one -- maybe you permute the list), or with a smart DNS server that serves *one* A or AAAA record to the client at the time of resolution (like a CDN would use). Instead of using the mosh wrapper script, you could have your users use some other scheme to figure out the IP:port of the server, but the point is that once you launch the mosh-client, it's going to keep sending datagrams to the IP:port of the mosh-server, and those datagrams need to get to the same mosh-server process even if the client roams to a different publicly-visible IP address or port. You could imagine writing a very smart mosh proxy that has the keys to all the sessions and can figure out (for an incoming datagram coming from an unknown source IP:port) which session it actually belongs to, and then makes a sticky mapping and routes it to the proper mosh-server. But I don't think anybody has actually done this yet and of course there's a challenge in making this reliable/replicated. -Keith On Mon, Jun 25, 2018 at 3:10 AM, Thomas Buckley-Houston <t...@tombh.co.uk> wrote: > Thanks so much for the clarification. > > > UDP is connectionless > > That's the key here. So I have no choice but to use sticky IP-based > routing. Round-robin DNS isn't an option I don't think, because I hope > one day to be able to scale to thousands of servers. > > And thanks so much for the heads up about my DNSSEC records. I've sent > a request for them to be deleted. I'd added them and some SSHFP > records to explore automatically passing the StrictHostKey warning. > But it's not entirely straight forward. Even with correct DNS records > the SSH user still has to have VerifyHostKeyDNS enabled, which as I > understand most people don't. And then on top of that my DNS provider > (DNSSimple) automatically rotate the keys every 3 months, which means > I have to manually send a request to my registrars by email to update > the DNSSEC records. Is it all worth it do you think? > > On 24 June 2018 at 13:36, Anders Kaseorg <ande...@mit.edu> wrote: > > You may have a misunderstanding about how a Mosh session is set up. The > > mosh script launches a mosh-server on the remote system via SSH; > > mosh-server picks a port number and a random encryption key, and writes > > them to stdout, where they go back over SSH to the mosh script; then the > > mosh script launches mosh-client passing the IP address, port number, and > > encryption key. The newly launched mosh-client and mosh-server processes > > exchange UDP packets encrypted with the shared key; communication is > > successful if the packets can be decrypted. > > > > There’s no separate “key checking” step to be disabled. And it doesn’t > > make sense to “refuse more than 1 connection on the same port”, both > > because UDP is connectionless, and because a new mosh-server is launched > > on a new port for each Mosh session (it is not a daemon like sshd). > > > > The easiest way to put Mosh servers behind a load balancer is with > > round-robin DNS where a single hostname resolves to many addresses, or to > > different addresses for different clients and/or at different times. > > We’ve already gone out of our way to make the mosh script resolve the > > hostname only once and use the same address for the SSH connection and > the > > UDP packets, because that’s needed for MIT’s athena.dialup.mit.edu pool. > > > > If that’s not an option and you really need all connections to go through > > a single load balancer address, you could try wrapping mosh-server in a > > script that passes different disjoint port ranges (-p) on different > > backends, and forwarding those ranges to the corresponding backends from > > the load balancer. > > > > Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled resolvers like > > 1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS records set > > with the registrar: https://dnssec-debugger.verisignlabs.com/brow.sh. > > > > Anders > > _______________________________________________ > mosh-devel mailing list > mosh-devel@mit.edu > http://mailman.mit.edu/mailman/listinfo/mosh-devel >
_______________________________________________ mosh-devel mailing list mosh-devel@mit.edu http://mailman.mit.edu/mailman/listinfo/mosh-devel