On Sun, 17 Nov 2019 01:26:19 +0100 Andreas Grapentin <[email protected]> wrote:
> > Hi, > > it's been almost two weeks since the first announcement was made, and > I believe we have now a good understanding of our infrastructure and > our hosting needs, and are ready to move forward. > > We have discussed internally, and have come to the conclusion that a > good course of action could be to keep winston running, but start > migrating some of the essential services off it one by one, > streamlining their deplopment in the process. > > To get started, we are currently looking at getting a single VPS, with > specs comparable to, or slightly improved upon, the specs of winston > itself. > > Winston currently has: > > - (nominally) about 4GiB of RAM. > > we believe that 4GiB is currently the lower bound of what our > services need to function. Winston is currently struggling with > RAM, but this seems to be mainly because he's not having the nominal > amount actually available for use. > > To get some room to breathe, we're hoping for an upgrade here > towards the region of 6 or 8GiB of ram, mainly to make sure the > machine doesn't need to swap as much as it does now. > > - 4 QEMU virtual CPUS @ 2.2GhZ > > winston is never really computationally busy. iirc, vikings is > allowing only for 1 CPU VPS's (please correct me if I'm wrong) so Wrong :) Vikings can do anything between 1 and 32 CPUs. When we say "CPU" we refer to the physical core and not some over-subscribed, mass-hosting shenanigans. > we're not looking for an upgrade here. Realistically, anything > that's not a potato should be sufficient to deal with our workload. > > - 500GiB of storage > > storage is getting a bit tight. We are planning to extend our repos > to more architectures, chiefly ppc64le and riscv64, and we would > not be able to host all the built packages due to limited storage > capacity. > > Ideally, we would like to see an extended capacity of at least 1TiB > here, in order to work towards our medium-term goals. Even more > storage would allow us to move forward with other projects, such as > providing a source packgaes repository, or maintaining a larger > archive of older packages. But we can discuss these separately once > the respective projects move forward. > > - virtually unlimited network traffic > > as far as I know, winston is not limited in terms of network > traffic. And we do incur a LOT of network traffic. The order of > magnitude should be somewhere in the single-digit TiB's per month, > although it's hard to be sure since we don't have a monitoring in > place. > > Most of this traffic is because of a growing number of busily > syncing parabola mirrors, and we could cut that down by better > structuring our network of mirrors, and giving that more of a > hierarchy instead of having everyone pull from the root. > > We should talk to the vikings people here, what sort of traffic is > allowed on their network, to get a feeling how much we need to cut > down here. Also, we should start putting better traffic monitoring > in place. > > So much for the rationale. In summary, we believe that we currently > need a single VPS with: > > - a single core > - at least 8GiB of RAM > - at least 1TiB of storage > - a reasonable agreement on traffic limits > > If I have missed or forgot anything, let me know. Otherwise, we should > probably get the vikings people in the loop and discuss next steps! > > Best, > Andreas (oaken-source) > > > > On Tue, Nov 05, 2019 at 11:28:36PM +0100, Andreas Grapentin wrote: > > > > Hi, > > > > As you know, a good chunk of the infrastructure that makes parabola > > is currently hosted on a virtual machine called winston, which is > > generously provided by the hosting provider 1984. We used to have > > proton as well, but when that died, the things that it hosted were > > migrated over to winston. > > > > It's come to our attention that this machine is currently > > struggling to cope with the load it has to bear, as is evident by > > the fact that it is as of this writing at ~70% swap utilization, > > and requires more and more frequent reboots just to keep our basic > > infrastructure operational. As you can probably imagine, this is > > not a healthy mode of operation for a distribution, and is > > something that is eating a bunch of hacker time to keep on top of. > > > > These developments have prompted freemor, bill-auger and me to keep > > an eye open for other hosting options that would maybe give our > > services and bit more breathing room, and maybe some space to grow > > and expand. > > > > bill-auger had floated the idea to get in touch with the people > > over at vikings (vikings.net) -- in CC -- who offer (among other > > things) hosting services on exclusively libre-friendly server > > hardware, which would be perfect for parabola. We did already have > > a brief chat with their team to figure out what their available > > hosting options are, and now we need to discuss amongs ourselves > > what sort of infrastructure we want going forwards. > > > > I think it would make sense to start by listing all the services > > that the new infrastructure should host, and estimating from there > > what sort of physical or virtual machine or machines we need, and > > how we distribute the services. > > > > Off the top of my head, these are the services that I know we run: > > > > - webserver for parabola.nu, including > > - wiki > > - main website > > - git repositories > > - issue tracker (redmine?) > > - forum > > - Tier 0 package repositories > > - repo importer > > - mailman > > > > But I'm probably missing a bunch of things. This is where I'd like > > someone more experienced with our infrastructure to step in and > > explain what we have and need. I would suggest we start maintaining > > this information in the wiki as well, assuming no such article > > exists already. This might come in handy down the line. > > > > My hunch as a Software Engineer and Architect would be that it could > > make sense to partition these services in two, or maybe three VMs, > > just to have a more fine-grained control over fault tolerance and > > recovery in case something does go belly-up, and to be able to > > separate the critical from the non-critical infrastructure. But > > this is something we need to discuss down the line, if we have a > > clear idea about our service landscape. > > > > While we do this, I would also suggest we take another look at which > > part of our infrastructure could and / or should be converted to > > packages, instead of unaccounted for files. We could conceivably > > add a separate undocumented repo for server-only packages. And > > while we're at that, I would also like to take a gander at > > automatic provisioners, such as ansible, that would allow us to > > describe our server roles and then automatically deploy them as > > needed, to have a self-documenting deployment of our services. > > > > But first, let's complete the list of services that parabola needs > > to function, and figure out how we want to partition these and what > > sort of resources they need. Everything beyond that is still far > > away. :) > > > > > > I have to say; I am personally very excited to see these changes > > unfold. I believe that parabola is currently at a place where we > > need to stabilize our infrastructure and establish ourselves as a > > serious distribution. And a collaboration with vikings could be a > > great chance to take a leap toward that direction. > > > > However, I also appreciate that parabola is an adhocracy, and that > > each of you has their own unique views on these issues and options. > > If you have any concerns about these developments, please bring > > them to our attention as well. > > > > I'm looking forwards to everyones input. > > > > Best, > > Andreas > > > > oaken-source -- Best regards/Mit freundlichen Grüßen/Cordialement/Met vriendelijke groet The Vikings Team Web: https://www.vikings.net/ Store: https://store.vikings.net/ Phone: +49 69 247 54 91 0 Fax: +49 69 247 50 339 Follow us on Mastodon: https://social.vikings.net/@vikings Follow us on Twitter: https://twitter.com/vikingslibre Vikings GmbH Sauerackerweg 14, 60529 Frankfurt/Main, Germany Register Court: AG Frankfurt am Main Register No.: HR B 84497, CEO: Thomas Umbach Tax Office: Frankfurt am Main, VATIN: DE254094338 GPG key fingerprint: FD5E 2543 EE62 7395 00A6 9FBB F228 7A48 7CA2 3FA5
pgpeRvORbJspE.pgp
Description: OpenPGP digital signature
_______________________________________________ Dev mailing list [email protected] https://lists.parabola.nu/mailman/listinfo/dev
