It does appear to be broken for every project we create on the cluster. Containers that are from outside sources pull and run fine, but all fail when we try to push to the registry.
The ansible playbook is the standard openshift origin one found at https://github.com/openshift/openshift-ansible. I am unable to login to the registry using "docker login -u "$(oc whoami)" -p "$(oc whoami -t)" 172.30.241.53:5000". It spins a while and then returns a "Error response from daemon: Get https://172.30.241.53:5000/v1/users/: dial tcp 172.30.241.53:5000: i/o timeout". I confirmed the IP of the docker-registry with "oc -n default get svc/docker-registry" as 172.30.241.53, which is correct. There are 5 pods in the registry. I checked the logs for the registry and all he shows are successful health checks as well. The strange portion of this is that when I try to push to the registry, I can't see attempted connections in the docker-registry logs at all. We did see once a permissions error, but we have looked at our storage and the permissions appear to be correct. Is it possible that there are some holdover files from a previous install that are for some reason interfering with openshift as it tries to write? On Tue, Sep 19, 2017 at 3:25 AM, Alexey Gladkov <[email protected]> wrote: > On Mon, Sep 18, 2017 at 11:46:40PM -0400, Ben Parees wrote: > > On Mon, Sep 18, 2017 at 12:47 PM, Graigry Henrie <[email protected]> > > wrote: > > > > > We recently installed openshift origin using an ansible playbook. We > had > > > used the same one before without issue, except that now when projects > > > attempt to push to the internal registry, it results in a 500 error. > > > > > > Attached is the log from attempting to build the Django Example > project. > > > The build itself proceeds perfectly, but fails with a "500 Internal > Server > > > Error" when attempting to push to the internal registry. Checking the > logs > > > of the registry itself only shows health checks, indicating that > container > > > has come up properly. > > > > > > What makes it more confusing is that we have installed with this same > file > > > multiple times and sometimes it works and sometime it doesn't. We > believe > > > that it is a permissions issue. We have checked and rechecked our > ansible > > > hosts file, but we cannot see what the problem is. I have included it > as > > > well. > > > > > > > can you try manually pushing to the registry using the service account's > > token? (ie docker login to the registry w/ the token and then tag an > > image+push to it). > > > > There is a known issue that the registry returns 500 errors when it > should > > return 403s, so your theory that it is a permission issue is plausible > but > > it is strange that some of your clusters work. When it is not working, > it > > sounds like it is broken for all projects in the cluster, not just some > or > > fails randomly? > > > > Adding our registry gurus to CC. > > Can we see this ansible playbook ? > > How many registry pods in the cluster (maybe the logs from the requests on > the other pod) ? > > Is there a router in front of him and what's in his logs ? > > -- > Rgrds, agladkov > >
_______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
