How about a semi-smart (but mostly Mosh-oblivious) server-side proxy/NAT that works like this:
- The proxy service has one public IP address and like 65,000 available UDP ports. - The proxy service can itself be redundant with failover... - When a user wants to open a new Mosh connection, they Mosh to a single hostname (which resolves to the IP address of the proxy service). - Your code allocates the necessary container, etc., and also allocates a unique UDP port on the proxy. - Your code runs the new mosh-server process in the target container. - The proxy intercepts the mosh-server's "MOSH CONNECT <port> <key>" message, replacing the port number with the unique public-facing UDP port (and remembering the container's IP address and the original port number). - When the proxy receives an incoming UDP datagram destined to a particular UDP port, it forwards it to the appropriate container at its IP address and at the original port number. It *preserves* the source IP and port of the datagram when forwarding. - When the container wants to send an outgoing UDP datagram, it sends it normally (to whatever IP:port is associated with the client), except the containers are not directly connected to the Internet; they use the proxy/NAT as their next-hop router. - For the outgoing UDP datagram, the proxy/NAT rewrites the container's source IP:port to its own IP and the public port number. I think this will allow you to serve like 65,000 separate mosh connections from a single public IP address... The added latency in forwarding a datagram is probably <1 ms, and you don't really have to change anything about Mosh itself or its internals. Unfortunately there are no unencrypted identifying marks to a Mosh connection, except the incrementing sequence numbers (which start at 0 for every connection). -Keith On Fri, Jun 29, 2018 at 12:17 AM, Thomas Buckley-Houston <t...@tombh.co.uk> wrote: > Hey Keith, John, everyone, > > Yeah the more this is looking like a quite a big hurdle. Especially > your point Keith about roaming IPs (which I'd forgotten), it's a > central feature of Mosh I don't want to lose. > > So the only 2 options seems to be exposing multiple IPs for Round > Robin (or other smart DNS routing) or writing a new Mosh proxy that > already has knowledge of the existing keys. Both seem like quite a > challenge. Round Robin DNS seems more approachable and I can imagine > integrating it with the Google Cloud DNS API I'm already using, but I > just wonder how expensive Google (or anyone for that matter) will make > thousands of static IP addresses? Apart from me having to learn Mosh > internals, one difficulty that strikes me about a Mosh proxy is that > it might introduce a non-trivial delay to each datagram arriving? > Though surely only ever in the order of a handful of milliseconds I > suppose. > > Are there not any other identifying marks to a datagram, I don't know > much about low level networking, but maybe something like a MAC > address for example? > > Thanks, > Tom > > On 27 June 2018 at 04:50, Keith Winstein <kei...@cs.stanford.edu> wrote: > > Hi Thomas, > > > > Glad you could provoke a very interesting discussion! But I'm still > confused > > -- how is "sticky IP-based routing" going to work after the client roams > to > > a new IP address (or to a new UDP source port)? When your system seems an > > incoming UDP datagram from a previously unseen source IP:port, how does > it > > know which mosh-server (on which server machine) to send it to? > > > > With off-the-shelf Mosh, you basically need a load-balancing strategy > that > > allows a destination IP:port to uniquely identify a particular > mosh-server. > > You can do this with multiple DNS A/AAAA records (where the client picks > the > > winning one -- maybe you permute the list), or with a smart DNS server > that > > serves *one* A or AAAA record to the client at the time of resolution > (like > > a CDN would use). > > > > Instead of using the mosh wrapper script, you could have your users use > some > > other scheme to figure out the IP:port of the server, but the point is > that > > once you launch the mosh-client, it's going to keep sending datagrams to > the > > IP:port of the mosh-server, and those datagrams need to get to the same > > mosh-server process even if the client roams to a different > publicly-visible > > IP address or port. > > > > You could imagine writing a very smart mosh proxy that has the keys to > all > > the sessions and can figure out (for an incoming datagram coming from an > > unknown source IP:port) which session it actually belongs to, and then > makes > > a sticky mapping and routes it to the proper mosh-server. But I don't > think > > anybody has actually done this yet and of course there's a challenge in > > making this reliable/replicated. > > > > -Keith > > > > On Mon, Jun 25, 2018 at 3:10 AM, Thomas Buckley-Houston <t...@tombh.co.uk > > > > wrote: > >> > >> Thanks so much for the clarification. > >> > >> > UDP is connectionless > >> > >> That's the key here. So I have no choice but to use sticky IP-based > >> routing. Round-robin DNS isn't an option I don't think, because I hope > >> one day to be able to scale to thousands of servers. > >> > >> And thanks so much for the heads up about my DNSSEC records. I've sent > >> a request for them to be deleted. I'd added them and some SSHFP > >> records to explore automatically passing the StrictHostKey warning. > >> But it's not entirely straight forward. Even with correct DNS records > >> the SSH user still has to have VerifyHostKeyDNS enabled, which as I > >> understand most people don't. And then on top of that my DNS provider > >> (DNSSimple) automatically rotate the keys every 3 months, which means > >> I have to manually send a request to my registrars by email to update > >> the DNSSEC records. Is it all worth it do you think? > >> > >> On 24 June 2018 at 13:36, Anders Kaseorg <ande...@mit.edu> wrote: > >> > You may have a misunderstanding about how a Mosh session is set up. > The > >> > mosh script launches a mosh-server on the remote system via SSH; > >> > mosh-server picks a port number and a random encryption key, and > writes > >> > them to stdout, where they go back over SSH to the mosh script; then > the > >> > mosh script launches mosh-client passing the IP address, port number, > >> > and > >> > encryption key. The newly launched mosh-client and mosh-server > >> > processes > >> > exchange UDP packets encrypted with the shared key; communication is > >> > successful if the packets can be decrypted. > >> > > >> > There’s no separate “key checking” step to be disabled. And it > doesn’t > >> > make sense to “refuse more than 1 connection on the same port”, both > >> > because UDP is connectionless, and because a new mosh-server is > launched > >> > on a new port for each Mosh session (it is not a daemon like sshd). > >> > > >> > The easiest way to put Mosh servers behind a load balancer is with > >> > round-robin DNS where a single hostname resolves to many addresses, or > >> > to > >> > different addresses for different clients and/or at different times. > >> > We’ve already gone out of our way to make the mosh script resolve the > >> > hostname only once and use the same address for the SSH connection and > >> > the > >> > UDP packets, because that’s needed for MIT’s athena.dialup.mit.edu > pool. > >> > > >> > If that’s not an option and you really need all connections to go > >> > through > >> > a single load balancer address, you could try wrapping mosh-server in > a > >> > script that passes different disjoint port ranges (-p) on different > >> > backends, and forwarding those ranges to the corresponding backends > from > >> > the load balancer. > >> > > >> > Unrelatedly, brow.sh doesn’t resolve with DNSSEC-enabled resolvers > like > >> > 1.1.1.1 or 8.8.8.8, seemingly due to some problem with the DS records > >> > set > >> > with the registrar: https://dnssec-debugger.verisignlabs.com/brow.sh. > >> > > >> > Anders > >> > >> _______________________________________________ > >> mosh-devel mailing list > >> mosh-devel@mit.edu > >> http://mailman.mit.edu/mailman/listinfo/mosh-devel > > > > > > _______________________________________________ > mosh-devel mailing list > mosh-devel@mit.edu > http://mailman.mit.edu/mailman/listinfo/mosh-devel >
_______________________________________________ mosh-devel mailing list mosh-devel@mit.edu http://mailman.mit.edu/mailman/listinfo/mosh-devel