Savannah Users,
Everyone who operates a version control server on the Internet has
been suffering from the endless botnet and AI Web Crawler abuse that
has been hitting all of the online systems. It's a problem! Today
the Savannah Hackers have activated a change that will improve things
significantly.
Up until now all git services have been served by one solo virtual
machine system. And this was limited by all uses using the one single
git.savannah.gnu.org hostname, and the .sv and .nongnu aliases,
putting everything on that address as a practical matter. (Yes there
are ways of doing port forwarding and load balancers in front but
compute resources and free software options for us are limited.)
We have assembled a small collection of volunteer contributed and
maintained read-only mirrors servers. This is very similar to the
long history of using contributed download mirrors. These have been
getting "tinkered together" and now are ready for production use.
Today we activated an HTTP Redirect on the primary system to redirect
all CGIT and GITWEB traffic from the primary to the secondaries. The
secondaries use Round-Robin-DNS to distribute the load among the
collective of them. Also this means that if one of the systems is
offline web browsers will automatically fallback to one of the other
secondaries automatically. This should improve reliability hugely!
The new URLs.
https://gitweb.git.savannah.gnu.org/gitweb/
https://cgit.git.savannah.gnu.org/cgit/
When this change was made live the load average on the primary fell
from its typical 38-40 that it plateaus at, it would be much higher if
we had the compute resources to support it, down to 1-2 almost
immediately. That's a huge improvement! This should improve the
reliability of the primary for the services that are still on that
machine and specifically the member ssh access to it.
Clearly this is a new configuration just now being brought online.
There are almost certainly going to be snags that we don't know about
yet. Please let us know if you run into problems.
As foreshadowing this is leading to more future changes to manage the
load problems of operating services on the current hostile Internet.
We are aiming at using Divide And Conquer to split services out into a
scaled-out distributed set of servers. This requires URL name changes
on the client side. There is no way to avoid it forever. But for the
moment client side name changes are not yet required. It will be
needed at some point however. You have foreshadowing of this now.
Bob
