potiuk commented on pull request #13551: URL: https://github.com/apache/airflow/pull/13551#issuecomment-758112488
> Wonder what has happened to CI... @ashb: Just in case it happens again so that you know. This is a random error of the Github Packages infrastructure. 1) The image was correctly build and failed the first time it was pushed: https://github.com/apache/airflow/runs/1670775083?check_suite_focus=true#step:10:443 ``` manifest invalid: Only schema version 2 is supported WARNING:: Error 1 when pushing image on 1 try docker push docker.pkg.github.com/apache/airflow/master-python3.6-ci:e3eaf5160cc4a694afb05debf3ef81bf56694230 ```` ^^ This is retry and it succeeded. 2) The image is present (in theory) so "wait for image succeds". Wait for image (to save bandwith and time) only checks if the manifest exists using the registry API: https://github.com/apache/airflow/pull/13551/checks?check_run_id=1670693749#step:5:221 3) But when you try to pull it (you can try it even now): ``` docker pull docker.pkg.github.com/apache/airflow/master-python3.6-ci:472552156 Error response from daemon: mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not '' ``` The problem is that GitHub has not 100% unstable architecture for "Github Packages" - looks like sometimes randomly the proxy is redirecting the requests wrongly and they get to a wrong backend (this manifest itself as v1 vs. v2 manifest version problem). This is one of the main reasons they introduced GitHub Container Registry in September and deprecated GitHub Packages : https://github.blog/2020-09-01-introducing-github-container-registry/ (without the sunset time yet). Apparenly there are cases where such image gets corupted (and there is not much we can do). When I opened (Oct 10) Personal ticket https://support.github.com/ticket/personal/0/865404 (so you won't see) but I also documented it (last point in https://github.com/apache/airflow/issues/11294) they fixed "unknown blob" apparently but now we have 'invalid manifest" Another ticket - where I verified that we can continue with Packages without the possibility of deleting the packages - I asked if their system is capable of handling our many images without deleting: https://support.github.com/ticket/personal/0/861667 - GH support confirmed that this will be no problem, but recommended us to switch to Container Registry (where old image retention can be implemented). I think switching to container registry is the only way to solve it. I opened (7th of October) a JIRA ticket to infra so that they consider enabling Container Registry for the projects so that we can switch to it: https://issues.apache.org/jira/projects/INFRA/issues/INFRA-20959 This ticket was waiting for the INFRA-organized meeting with GitHub Account, where we were supposed to discuss it but the meeting got cancelled and we are waiting for the new date (I asked for it several times). Anything unclear - let me know @ashb -> this ticket needs reb ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
