potiuk commented on PR #21323: URL: https://github.com/apache/airflow/pull/21323#issuecomment-1091513183
> @potiuk that sounds reasonable so in this particular case, are you saying that we do want to actually test that we can run a container workload? > > Just bear in mind that what motivated this was that an external container registry was unable to provide the container image in time so the test timed out. Yeah. Those external images fail occassionally - but not often enough to make it a problem. Our tests will anyhow refuse to work if a number of images are not available. We actually have quite a good sytem in-place to prevent from depending on "external images" to break our tests - rather than rely on those external images we either build, or simply push such an image to our Github Registry - this way we are free from dependence of the external images in a location that is not ours and speed up the builds as Github Registry is generally "close" and "more stable" for Github actions. Also it has the nice benefit that we do not fall into DockerHub limits of pulling images (DockerHub images have very aggressive rate limiting for unauthenticated pulls (based on outgoing IP address). Github Actions Public runners are exempted from it agreement of DockerHub and GitHub, similarly as Apache DockerHub Images (ASF has an agreement with DockerHub about it) but our self-hosted runners pulling regular "DockerHub" images are only protected by the limit for authenticated users (we do authenticcate in our self-hosted runners). Those rate limits are better but still we might be rate limited at times. GHCR.IO images have no such limit - you can see example image here, but there are few more: https://github.com/orgs/apache/packages?tab=packages&q=airflow-trino. In this case I think we are using `quay.io/bitnami/python:3.9` for that reason (the official image of Python might make us hit rate limits). So the solution here is to push "airlfow-python:3.9" image to ghcr.io and use it from there. This will be faster and more reliable. I will push it in a moment and you might switch to it in a new PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
