Thanks for your response Clayton, It's not the v4auth, as I've experimented disabling it and the error is the same. If the s3 driver, this issue would likely be affecting the 3.2 enterprise release too? I'm planning on launching an enterprise 3.2 stack to confirm this.
Do you know if the code base is the same for the docker-registry? Might save some time... Cheers, Lew On 20 July 2016 at 14:19, Clayton Coleman <[email protected]> wrote: > Is S3 v4auth supported in the sydney region? That is the first thing that > comes to mind. The public docker image for the registry is a bit newer > than the registry in Origin, so it's possible as well that the S3 driver > was updated between that registry and v1.2.0. No registry changes were > made in origin between v1.2.0-rc1 and v1.2.0, so I suspect the change is > related to S3 config, not the registry code. > > On Tue, Jul 19, 2016 at 11:21 PM, Lewis Shobbrook < > [email protected]> wrote: > >> Hi Guys, >> >> Spent quite a bit more time on this and have been able to determine that >> the problem is specifically with the openshift registry image. >> Using the docker.io/registry:2 to create a registry on the same nodes, >> using the same configuration works. >> The service is able to write to the designated S3 folder. >> >> I've adjusted the registry build to use the latest image >> oadm registry --config=/etc/origin/master/admin.kubeconfig >> --service-account=registry >> --credentials=/etc/origin/master/openshift-registry.kubeconfig >> --latest-images=true >> Which has pulled a newer image, but alas the error is much the same... >> >> http.request.method=HEAD http.request.remoteaddr="10.1.1.1:51518" >> http.request.uri="/v2/bnz-uat/buzybox/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" >> http.request.useragent="docker/1.9.1 go/go1.4.2 >> kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" >> instance.id=745d924f-3a9d-4613-bd1d-dd9c826d52e0 >> vars.digest="sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" >> vars.name="bnz-uat/buzybox" >> time="2016-07-20T02:03:20.167566754Z" level=error msg="response completed >> with error" err.code=unknown err.detail="s3aws: RequestError: send request >> failed\ncaused by: Get >> https://os3master-prod-os-aws-XXX-com-au-docker.s3-ap-southeast-2.amazonaws.com/?max-keys=1&prefix=registry%2Fdocker%2Fregistry%2Fv2%2Fblobs%2Fsha256%2Fa3%2Fa3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4%2Fdata: >> dial tcp 54.66.155.60:443: getsockopt: connection refused" >> err.message="unknown error" go.version=go1.6.2 http.request.host=" >> 172.30.160.215:5000" http.request.id=0cf8999f-21b6-4c8c-9ffb-258e607e4914 >> http.request.method=HEAD http.request.remoteaddr="10.1.1.1:51518" >> http.request.uri="/v2/bnz-uat/buzybox/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" >> http.request.useragent="docker/1.9.1 go/go1.4.2 >> kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" >> http.response.contenttype="application/json; charset=utf-8" >> http.response.duration=308.013092ms http.response.status=500 >> http.response.written=104 instance.id=745d924f-3a9d-4613-bd1d-dd9c826d52e0 >> vars.digest="sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" >> vars.name="bnz-uat/buzybox" >> >> Is there anyone willing to help out here. Banging my head. >> >> Cheers, >> >> Lew >> >> On 13 July 2016 at 00:13, Lewis Shobbrook < >> [email protected]> wrote: >> >>> Hi Guys, >>> >>> I've spent the past two days looking at the problem of s3 backed storage >>> from Sydney based instances. The problem now appears to be more generally >>> associated with S3 backed registries in the ap-southeast-2 region. >>> >>> With configurations identical other than keys/bucket/region the results >>> are considerably different. >>> >>> In Sydney I see this... >>> >>> time="2016-07-12T13:03:33.859868304Z" level=error msg="response >>> completed with error" err.code=UNKNOWN err.detail="s3: Get >>> https://s3-external-1.amazonaws.com/os3master-prod-os-aws-XXX-com-au-docker-registry/registry/docker/registry/v2/repositories/bnz-uat/busybox/_layers/sha256/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4/link: >>> dial tcp 54.66.155.60:443: getsockopt: connection refused" >>> err.message="unknown error" go.version=go1.6 http.request.host=" >>> 172.30.27.237:5000" http.request.id=f5a2a1cb-be54-4f31-a13d-1294fc74a3f8 >>> http.request.method=HEAD http.request.remoteaddr="10.1.0.1:45204" >>> http.request.uri="/v2/bnz-uat/busybox/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" >>> http.request.useragent="docker/1.9.1 go/go1.4.2 >>> kernel/3.10.0-327.22.2.el7.x86_64 os/linux arch/amd64" >>> http.response.contenttype="application/json; charset=utf-8" >>> http.response.duration=4.863003309s http.response.status=500 >>> http.response.written=487 instance.id >>> =542dd6da-2901-4cc7-975c-4afd71077e19 >>> >>> Configuration as follows... >>> >>> version: 0.1 >>> log: >>> level: debug >>> http: >>> addr: :5000 >>> storage: >>> cache: >>> layerinfo: inmemory >>> s3: >>> accesskey: XXX >>> secretkey: XXX >>> region: us-east-1 >>> bucket: os3master-prod-os-aws-XX-com-au-docker-registry >>> encrypt: true >>> secure: true >>> v4auth: true >>> rootdirectory: /registry >>> auth: >>> openshift: >>> realm: openshift >>> middleware: >>> repository: >>> - name: openshift >>> >>> While the working registry in us-east >>> time="2016-07-12T13:45:08.168492162Z" level=info msg="response >>> completed" go.version=go1.6 http.request.host="172.30.1.78:5000" >>> http.request.id=66047be1-0f4e-4045-b710-c94fdaca10d6 >>> http.request.method=GET http.request.remoteaddr="10.1.0.1:56051" >>> http.request.uri="/v2/paycorp-pty-ltd/paycorp-service-auth1501/manifests/sha256:78a3af175b9d7c653cd4fdb42a3ccf44095c6e07b912c2719a85d5568179129f" >>> http.request.useragent="docker/1.9.1 go/go1.4.2 >>> kernel/3.10.0-327.13.1.el7.x86_64 os/linux arch/amd64" >>> http.response.contenttype="application/json; charset=utf-8" >>> http.response.duration=49.599539ms http.response.status=200 >>> http.response.written=66957 instance.id >>> =04de8cf0-2b4e-4546-b623-43f05c771805 >>> >>> version: 0.1 >>> log: >>> level: debug >>> http: >>> addr: :5000 >>> storage: >>> cache: >>> layerinfo: inmemory >>> s3: >>> accesskey: XXX >>> secretkey: XXX >>> region: us-east-1 >>> bucket: os3master-test-openshift-XXX-com-au-docker >>> encrypt: true >>> secure: true >>> v4auth: true >>> rootdirectory: /registry >>> auth: >>> openshift: >>> realm: openshift >>> middleware: >>> repository: >>> - name: openshif >>> >>> Looking inside the containers the only diffrence appears to be the image >>> id which is openshift/origin-docker-registry:v1.2.0-rc1 in the working >>> region & 1.2.0 in the failing. >>> >>> I'm hoping someone can assist here. >>> >>> Cheers, >>> >>> Lew >>> >> >> >> _______________________________________________ >> users mailing list >> [email protected] >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >> >> >
_______________________________________________ users mailing list [email protected] http://lists.openshift.redhat.com/openshiftmm/listinfo/users
