Marton, Thanks for the tip. (Too bad the docs <https://reference.apache.org/pmc/mirror_scripts> referenced from the issue I opened with INFRA <https://issues.apache.org/jira/browse/INFRA-10999> make no mention of mirrors.cgi.)
Matei, A Requester Pays bucket is a good idea. I was trying to avoid having to maintain a repository of assets, but I suppose it's ultimately unavoidable given that Apache does not partner with a CDN. I will look into this for Flintrock. Nick On Wed, Feb 28, 2018 at 7:21 AM Marton, Elek <e...@apache.org> wrote: > > > 2. *Apache mirrors are inconvenient to use.* When you download > > something from an Apache mirror, you get a link like this one > > < > https://www.apache.org/dyn/closer.lua/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz > >. > > Instead of automatically redirecting you to your download, though, > > you need to process the results you get back > > < > https://github.com/nchammas/flintrock/blob/67bf84a1b7cfa1c276cf57ecd8a0b27613ad2698/flintrock/scripts/download-hadoop.py#L21-L42 > > > > to find your download target. And you need to handle the high > > download failure rate, since sometimes the mirror you get doesn’t > > have the file it claims to have. > > It's not a full answer, just a note: > > You can also use mirrors.cgi instead of parsing the json from closer.lua: > > > https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz > > (Unfortunatelly it doesn't check the availibility of the file. If it's > moved to the archive you will be redirected to a 404) > > Marton > > --------------------------------------------------------------------- > To unsubscribe e-mail: dev-unsubscr...@spark.apache.org > >
