On Sun, Sep 9, 2012 at 10:14 PM, Lloyd Chang <[email protected]> wrote:
> If we can't reliably force flush DNS propagation on client sides, > then another option's a dedicated HTTP that redirects. > > Per your conversation with R. Tyler Croy, it seems like we'd have to poll > for status of rsync from mirrors in order to change the HTTP redirect from > master back to mirror. > > Alternatively, perhaps continue pursuing the route of expanding storage, > then you can transparently rotate volume snapshots (before and after rsync > finishes) using a cluster or high availability failover via LVM. > > Another option's using a snapshot-ready file system like BTRFS, then you > wouldn't need LVM. > It adds a bit more latency, but couldn't the metadata change be delayed until all the mirrors report back that they have completed the sync > > Cheers, > Lloyd > > > On Sun, Sep 9, 2012 at 2:23 PM, Kohsuke Kawaguchi <[email protected]> wrote: > >> I believe DNS propagation delay makes this approach unworkable. >> >> 2012/9/9 Lloyd Chang <[email protected]>: >> > Kohsuke asked, "What can we do to fix this? Does anyone have any >> thoughts?" >> > >> > Proposal with steps >> > 1. Rename host jenkins-ci.org to master.jenkins-ci.org >> > 2. Change jenkins-ci.org to a CNAME (if not already CNAME) >> > 3. Set jenkins-ci.org CNAME to master.jenkins-ci.org >> > 4. Before staging job, set jenkins-ci.org CNAMfE to >> mirrors.jenkins-ci.org >> > (or cucumber.jenkins-ci.org, etc.) >> > 5. mirrors.jenkins-ci.org push notification to staging job process, or >> > staging job process pull status from mirrors.jenkins-ci.org >> > 6. After staging job recognizes that mirrors have pick up files from >> > master, set jenkins-ci.org CNAME back to master.jenkins-ci.org >> > >> > Thanks, >> > Lloyd >> > >> > On Fri, Sep 7, 2012 at 7:05 PM, Kohsuke Kawaguchi < >> [email protected]> >> > wrote: >> >> >> >> >> >> Jenkins handles a lot of download requests for core as well as >> plugins. So >> >> we rely on our mirrors to actually deliver bits. >> >> >> >> Unfortunately, by the nature of our mirrors, there's some time lag >> between >> >> the time we stage our master copy of files (which gets created as a >> part of >> >> the update center metadata generation process) to the time mirrors >> pick up >> >> those files. >> >> >> >> But as soon as bits are staged in the master copy, those URLs get >> >> advertised (download link from http://jenkins-ci.org/ and update >> center >> >> metadata.) Right now, download requests that happened during this >> window >> >> will result in 404. >> >> >> >> What can we do to fix this? Does anyone have any thoughts? >> >> >> >> >> >> We used to serve binaries from our master server in this situation >> without >> >> forwarding requests to mirrors, but this was the root cause why we >> ended up >> >> with the $5000 hole in our budget last year. So naturally I suspect >> Tyler >> >> will be afraid of enabling this again. >> >> >> >> If someone is willing to be a mirror that allows us to push files, >> that'd >> >> be ideal, as we can ensure that there's always at least one mirror >> that has >> >> all the copies all the time. We can set the priority very low to make >> sure >> >> that it only gets used very rarely. >> >> >> >> Another possibility is maybe to run some kind of bandwidth-restricted >> HTTP >> >> server. In that way, maybe we can convince ourselves that even in the >> worst >> >> case we won't go bankrupt. >> >> >> >> The third possibility might be to buy a bigger disk to >> >> cabbage.jenkins-ci.org. My understanding is that this machine is in >> OSUOSL, >> >> so we can use it to serve bytes without worrying about incurring >> overcharge. >> >> >> >> >> >> BTW, right now, we have about 58GB of data that needs to be mirrored. >> >> >> >> -- >> >> Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ >> >> Try Nectar, our professional version of Jenkins >> > >> > >> >> >> >> -- >> Kohsuke Kawaguchi >> > > -- -- Andrew Melo
