potiuk commented on pull request #14531: URL: https://github.com/apache/airflow/pull/14531#issuecomment-803296288
Hey @ashb - I had experienced quite a few problems with "pull limits with Docker" again. I have no idea where the credentials are deleted. I have a few candidates but I cannot pin-point any of them: * system prune of docker sometimes deleting the credentials (when they expired maybe) ? * unit tests of docker operator * cancelling the test while we are adding experimental flag My most probable hypothesis is now: * temporary failure to login when the runner gets initialized For me the most likely cause is the last one. I believe the problem comes from a temporary failure of logging to docker registry when runner gets initialized. It is rather plausible, we have no error generated when it happens. We do not have set -e in the init scripts, so this sounds very much like this. For now I did a very simple thing - i simply repeat logging in in the CI scripts. It seems it is not necessary in the runner - we can do it only when we have RUNS_ON = self-hosted (the awscli is still installed and we have the same permissions that the runner code to retrieve the password from the secret). I am testing this. Eventually we will be able to remove this line from the runner init I believe. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
