Re: [OpenStack-Infra] tarballs.openstack.org to AFS publishing gameplan
Ian Wienand writes: > On Wed, Jan 29, 2020 at 05:21:49AM +, Jeremy Stanley wrote: >> Of course I meant from /(.*) to tarballs.opendev.org/openstack/$1 so >> that clients actually get directed to the correct files. ;) > > Ahh yes, sorry you mentioned that in IRC and I should have > incorporated that. I'm happy with that; we can also have that > in-place and test it by overriding our hosts files before any > cut-over. The overall plan sounds good to me, as does the follow-up. I'm ambivalent about when we put the redirects in place (during or after the host move). Whichever is easiest (but my guess is that due to the additional testing we would be able to do, *during* might be easiest). -Jim ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] tarballs.openstack.org to AFS publishing gameplan
On Wed, Jan 29, 2020 at 05:21:49AM +, Jeremy Stanley wrote: > Of course I meant from /(.*) to tarballs.opendev.org/openstack/$1 so > that clients actually get directed to the correct files. ;) Ahh yes, sorry you mentioned that in IRC and I should have incorporated that. I'm happy with that; we can also have that in-place and test it by overriding our hosts files before any cut-over. -i ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] tarballs.openstack.org to AFS publishing gameplan
On 2020-01-29 05:18:55 + (+), Jeremy Stanley wrote: > On 2020-01-29 15:56:24 +1100 (+1100), Ian Wienand wrote: > [...] > > 3) I propose we make tarballs.openstack.org a vhost on > > static.opendev.org that serves the > > /afs/openstack.org/project/tarballs.opendev.org/openstack/ > > directory. > > > > Because > > > > * This is transparent for tarballs.openstack.org; all URLs work with > >no redirection, etc. > > * anyone hitting tarballs.opendev.org will see top-level project > >directories (openstack, zuul, airship, etc.) which makes sense. > [...] > > While it could be a later step, if the OpenStack project leadership > agrees I think I'd rather see the tarballs.openstack.org just host a > permanent redirect from /(.*) to tarballs.opendev.org/$1 so that we > might eventually be able to consider dropping the > tarballs.openstack.org hostname entirely. Also continuing to host > that tree as a separate white-labeled site may encourage other > projects to request similar vanity tarball sites. Of course I meant from /(.*) to tarballs.opendev.org/openstack/$1 so that clients actually get directed to the correct files. ;) -- Jeremy Stanley signature.asc Description: PGP signature ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] tarballs.openstack.org to AFS publishing gameplan
On 2020-01-29 15:56:24 +1100 (+1100), Ian Wienand wrote: [...] > 3) I propose we make tarballs.openstack.org a vhost on > static.opendev.org that serves the > /afs/openstack.org/project/tarballs.opendev.org/openstack/ > directory. > > Because > > * This is transparent for tarballs.openstack.org; all URLs work with >no redirection, etc. > * anyone hitting tarballs.opendev.org will see top-level project >directories (openstack, zuul, airship, etc.) which makes sense. [...] While it could be a later step, if the OpenStack project leadership agrees I think I'd rather see the tarballs.openstack.org just host a permanent redirect from /(.*) to tarballs.opendev.org/$1 so that we might eventually be able to consider dropping the tarballs.openstack.org hostname entirely. Also continuing to host that tree as a separate white-labeled site may encourage other projects to request similar vanity tarball sites. -- Jeremy Stanley signature.asc Description: PGP signature ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] tarballs.openstack.org to AFS publishing gameplan
Hello, We're at the point of implementing the tarballs.openstack.org publishing changes from [1], and I would just like to propose some low-level plans for feedback that exceeded the detail in the spec. We currently have tarballs.opendev.org which publishes content from /afs/openstack.org/project/opendev.org/tarballs. This is hosted on the physical server files02.openstack.org and managed by puppet [2]. 1) I propose we move tarballs.opendev.org to be served by static01.opendev.org and configured via ansible Because: * it's one less thing running on a Xenial host with puppet we don't want to maintain. * it will be alongside tarballs.openstack.org per below The /afs/openstack.org/project/opendev.org directory is currently a single AFS volume "projet.opendev" and contains subdirectories: docs tarballs www opendev.org jobs currently write their tarball content into the AFS location, which is periodically "vos released" by [3]. 2) I propose we make a separate volume, with separate quota, and mount it at /afs/openstack.org/project/tarballs.opendev.org. We copy the current data to that location and modify the opendev.org tarball publishing jobs to use that location, and setup the same periodic release. Because: * Although currently the volume is tiny (<100mb), it will become quite large when combined with ~140gb of openstack repos * this seems distincly separate from docs and www data * we have content for other hosts at /afs/openstack.org/project like this, it fits logically. The next steps are described in the spec; with this in place, we copy the current openstack tarballs from static.openstack.org:/srv/static/tarballs to /afs/openstack.org/project/tarballs.opendev.org/openstack/ We then update the openstack tarball publishing jobs to publish to this new location via AFS (we should be able to make this happen in parallel, initially). Finally, we need to serve these files. 3) I propose we make tarballs.openstack.org a vhost on static.opendev.org that serves the /afs/openstack.org/project/tarballs.opendev.org/openstack/ directory. Because * This is transparent for tarballs.openstack.org; all URLs work with no redirection, etc. * anyone hitting tarballs.opendev.org will see top-level project directories (openstack, zuul, airship, etc.) which makes sense. I think this will get us where we want to be. Any feedback welcome, thanks. We will keep track of things in [4]. [1] https://docs.opendev.org/opendev/infra-specs/latest/specs/retire-static.html [2] https://opendev.org/opendev/system-config/src/branch/master/manifests/site.pp#L441 [3] https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/files/openafs/release-volumes.py [4] https://storyboard.openstack.org/#!/story/2006598 ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
