potiuk opened a new issue #14989:
URL: https://github.com/apache/airflow/issues/14989
Every now and then our docs builds start to fail because of external
dependency (latest example here #14985). And while we are doing caching now of
that information, it does not help when the initial retrieval fails. This
information does not change often but with the number of dependencies we have
it will continue to fail regularly simply because many of those depenencies are
not very reliable - they are just a web page hosted somewhere. They are nowhere
near the stabilty of even PyPI or Apt sources and we have no mirroring in case
of problem.
Maybe we could
a) see if we can use some kind of mirroring scheme (do those sites have
mirrrors ? )
b) if not, simply write a simple script that will dump the cached content
for those to S3, refresh it in the CI scheduled (nightly) master builds ad have
a fallback mechanism to download that from there in case of any problems in CI?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
