2012/9/9 R. Tyler Croy <[email protected]>: > > On Fri, 07 Sep 2012, Kohsuke Kawaguchi wrote: > >> >> Jenkins handles a lot of download requests for core as well as >> plugins. So we rely on our mirrors to actually deliver bits. >> >> Unfortunately, by the nature of our mirrors, there's some time lag >> between the time we stage our master copy of files (which gets >> created as a part of the update center metadata generation process) >> to the time mirrors pick up those files. >> >> But as soon as bits are staged in the master copy, those URLs get >> advertised (download link from http://jenkins-ci.org/ and update >> center metadata.) Right now, download requests that happened during >> this window will result in 404. >> >> What can we do to fix this? Does anyone have any thoughts? > > > Technically the OSUOSL mirror system is one we push to, I think we try to only > run rsync's on a poll based interval but there's no reason we can't "push" > more.
If I remember correctly, the actual push to OSUOSL mirror system is still asynchronous --- we rsync files to the staging area, then run a script, which pokes their system that eventually delivers bits to the actual mirror servers. So we still have no way of knowing when files actually become downloadable via mirrors. We can rework the way files are generated (by pushing files first then page scrape apache index or check with mirrorbrain to figure out what files are downloadable), but I think it's easier to buy a disk and have a fallback that we can control synchronously. > Both cabbage and lettuce have LVM-managed disks, so we should be able expand > those volumes if needed. MirrorBrain allows for setting a fallback mirror > which > we currently set to the ftp-nyc OSUOSL mirror (IIRC). We can easily set that > to > cabbage or lettuce if we chose those as a dedicated mirror master. OK, this is good to know. What is the process involved in getting a disk added? I guess I should ask in #osuosl? Do you know what interface these servers take? ... actually, you know what? looks like cabbage has a 800GB VG that's pretty much unused! So there's no need for additional disk. Should I create a LV out of it and mount it to /srv like we do elsewhere? > - R. Tyler Croy > -------------------------------------- > Code: https://github.com/rtyler > Chatter: https://twitter.com/agentdero -- Kohsuke Kawaguchi
