If we can't reliably force flush DNS propagation on client sides, then another option's a dedicated HTTP that redirects.
Per your conversation with R. Tyler Croy, it seems like we'd have to poll for status of rsync from mirrors in order to change the HTTP redirect from master back to mirror. Alternatively, perhaps continue pursuing the route of expanding storage, then you can transparently rotate volume snapshots (before and after rsync finishes) using a cluster or high availability failover via LVM. Another option's using a snapshot-ready file system like BTRFS, then you wouldn't need LVM. Cheers, Lloyd On Sun, Sep 9, 2012 at 2:23 PM, Kohsuke Kawaguchi <[email protected]> wrote: > I believe DNS propagation delay makes this approach unworkable. > > 2012/9/9 Lloyd Chang <[email protected]>: > > Kohsuke asked, "What can we do to fix this? Does anyone have any > thoughts?" > > > > Proposal with steps > > 1. Rename host jenkins-ci.org to master.jenkins-ci.org > > 2. Change jenkins-ci.org to a CNAME (if not already CNAME) > > 3. Set jenkins-ci.org CNAME to master.jenkins-ci.org > > 4. Before staging job, set jenkins-ci.org CNAMfE to > mirrors.jenkins-ci.org > > (or cucumber.jenkins-ci.org, etc.) > > 5. mirrors.jenkins-ci.org push notification to staging job process, or > > staging job process pull status from mirrors.jenkins-ci.org > > 6. After staging job recognizes that mirrors have pick up files from > > master, set jenkins-ci.org CNAME back to master.jenkins-ci.org > > > > Thanks, > > Lloyd > > > > On Fri, Sep 7, 2012 at 7:05 PM, Kohsuke Kawaguchi < > [email protected]> > > wrote: > >> > >> > >> Jenkins handles a lot of download requests for core as well as plugins. > So > >> we rely on our mirrors to actually deliver bits. > >> > >> Unfortunately, by the nature of our mirrors, there's some time lag > between > >> the time we stage our master copy of files (which gets created as a > part of > >> the update center metadata generation process) to the time mirrors pick > up > >> those files. > >> > >> But as soon as bits are staged in the master copy, those URLs get > >> advertised (download link from http://jenkins-ci.org/ and update center > >> metadata.) Right now, download requests that happened during this window > >> will result in 404. > >> > >> What can we do to fix this? Does anyone have any thoughts? > >> > >> > >> We used to serve binaries from our master server in this situation > without > >> forwarding requests to mirrors, but this was the root cause why we > ended up > >> with the $5000 hole in our budget last year. So naturally I suspect > Tyler > >> will be afraid of enabling this again. > >> > >> If someone is willing to be a mirror that allows us to push files, > that'd > >> be ideal, as we can ensure that there's always at least one mirror that > has > >> all the copies all the time. We can set the priority very low to make > sure > >> that it only gets used very rarely. > >> > >> Another possibility is maybe to run some kind of bandwidth-restricted > HTTP > >> server. In that way, maybe we can convince ourselves that even in the > worst > >> case we won't go bankrupt. > >> > >> The third possibility might be to buy a bigger disk to > >> cabbage.jenkins-ci.org. My understanding is that this machine is in > OSUOSL, > >> so we can use it to serve bytes without worrying about incurring > overcharge. > >> > >> > >> BTW, right now, we have about 58GB of data that needs to be mirrored. > >> > >> -- > >> Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ > >> Try Nectar, our professional version of Jenkins > > > > > > > > -- > Kohsuke Kawaguchi >
