[jira] [Commented] (MESOS-10092) Cannot pull image from docker registry which does not reply with 'scope'/'service' in WWW-Authenticate header
[
https://issues.apache.org/jira/browse/MESOS-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044770#comment-17044770
]
Vinod Kone commented on MESOS-10092:
Upto 1.7 should be fine I think.
> Cannot pull image from docker registry which does not reply with
> 'scope'/'service' in WWW-Authenticate header
> -
>
> Key: MESOS-10092
> URL: https://issues.apache.org/jira/browse/MESOS-10092
> Project: Mesos
> Issue Type: Bug
>Reporter: Andrei Sekretenko
>Assignee: Andrei Sekretenko
>Priority: Critical
> Fix For: 1.8.2, 1.9.1, 1.10.0
>
>
> This problem was encountered when trying to specify container image
> nvcr.io/nvidia/tensorflow:19.12-tf1-py3
> When initiating Docker Registry authentication
> (https://docs.docker.com/registry/spec/auth/token/) with nvcr.io, Mesos URI
> fetcher receives 'WWW-Authenticate' header without 'service' and 'scope'
> params, and fails here:
> https://github.com/apache/mesos/blob/1e9b121273a6d9248a78ab44798bd4c1138c31ee/src/uri/fetchers/docker.cpp#L1083
> This is an example of an unsuccessful request made by Mesos:
> {code}
> curl -s -S -L -i --raw --http1.1 -H "Accept:
> application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws"
> -y 60 https://nvcr.io/v2/nvidia/tensorflow/manifests/19.08-py3
> HTTP/1.1 401 Unauthorized
> Content-Type: text/html
> Date: Wed, 22 Jan 2020 19:01:57 GMT
> Server: nginx/1.14.2
> Www-Authenticate: Bearer
> realm="https://nvcr.io/proxy_auth?scope=repository:nvidia/tensorflow:pull,push";
> Content-Length: 195
> Connection: keep-alive
>
> 401 Authorization Required
>
> 401 Authorization Required
> nginx/1.14.2
>
>
> {code}
> At the same time, docker is perfectly capable of pulling this image.
> Note that the document "Token Authentication Specification"
> (https://docs.docker.com/registry/spec/auth/token/), on which the Mesos
> implementation is based, is vague on the issue of registries that do not
> provide 'scope'/'service' in WWW-Authenticate header.
> What Docker does differently (at the very least, in the case of nvcr.io):
> It sends the initial request not to the maniferst/blob URI, but to the
> repository root URI (http:://nvcr.io/v2 in this case):
> {code}
> GET /v2/ HTTP/1.1
> Host: nvcr.io
> User-Agent: docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f402cd
> kernel/4.15.0-60-generic os/linux arch/amd64
> UpstreamClient(Docker-Client/18.09.7 \(linux\))
> {code}
> To this, it receives response with a "realm" that contains no query arguments:
> {code}
> HTTP/1.1 401 Unauthorized
> Connection: close
> Content-Length: 195
> Content-Type: text/html
> Date: Wed, 29 Jan 2020 12:22:43 GMT
> Server: nginx/1.14.2
> Www-Authenticate: Bearer realm="https://nvcr.io/proxy_auth
> {code}
> Then, it composes the scope using the image ref and a hardcoded "pull"
> action:
> https://github.com/docker/distribution/blob/a8371794149d1d95f1e846744b05c87f2f825e5a/registry/client/auth/session.go#L174
> (in a full accordance with this spec:
> https://docs.docker.com/registry/spec/auth/scope/)
> and sends the following request to https://nvcr.io/proxy_auth :
> {code}
> GET /proxy_auth?scope=repository%3Anvidia%2Ftensorflow%3Apull HTTP/1.1
> Host: nvcr.io
> User-Agent: Go-http-client/1.1
> {code}
> (Note that 'push' is absent from the scope)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (MESOS-10092) Cannot pull image from docker registry which does not reply with 'scope'/'service' in WWW-Authenticate header
[
https://issues.apache.org/jira/browse/MESOS-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044766#comment-17044766
]
Andrei Sekretenko commented on MESOS-10092:
---
[~vinodkone] I've just backported this into 1.9 and 1.8. I guess we might want
this in 1.7 (tests in progress), but do we need this in older versions?
> Cannot pull image from docker registry which does not reply with
> 'scope'/'service' in WWW-Authenticate header
> -
>
> Key: MESOS-10092
> URL: https://issues.apache.org/jira/browse/MESOS-10092
> Project: Mesos
> Issue Type: Bug
>Reporter: Andrei Sekretenko
>Assignee: Andrei Sekretenko
>Priority: Critical
> Fix For: 1.8.2, 1.9.1, 1.10.0
>
>
> This problem was encountered when trying to specify container image
> nvcr.io/nvidia/tensorflow:19.12-tf1-py3
> When initiating Docker Registry authentication
> (https://docs.docker.com/registry/spec/auth/token/) with nvcr.io, Mesos URI
> fetcher receives 'WWW-Authenticate' header without 'service' and 'scope'
> params, and fails here:
> https://github.com/apache/mesos/blob/1e9b121273a6d9248a78ab44798bd4c1138c31ee/src/uri/fetchers/docker.cpp#L1083
> This is an example of an unsuccessful request made by Mesos:
> {code}
> curl -s -S -L -i --raw --http1.1 -H "Accept:
> application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws"
> -y 60 https://nvcr.io/v2/nvidia/tensorflow/manifests/19.08-py3
> HTTP/1.1 401 Unauthorized
> Content-Type: text/html
> Date: Wed, 22 Jan 2020 19:01:57 GMT
> Server: nginx/1.14.2
> Www-Authenticate: Bearer
> realm="https://nvcr.io/proxy_auth?scope=repository:nvidia/tensorflow:pull,push";
> Content-Length: 195
> Connection: keep-alive
>
> 401 Authorization Required
>
> 401 Authorization Required
> nginx/1.14.2
>
>
> {code}
> At the same time, docker is perfectly capable of pulling this image.
> Note that the document "Token Authentication Specification"
> (https://docs.docker.com/registry/spec/auth/token/), on which the Mesos
> implementation is based, is vague on the issue of registries that do not
> provide 'scope'/'service' in WWW-Authenticate header.
> What Docker does differently (at the very least, in the case of nvcr.io):
> It sends the initial request not to the maniferst/blob URI, but to the
> repository root URI (http:://nvcr.io/v2 in this case):
> {code}
> GET /v2/ HTTP/1.1
> Host: nvcr.io
> User-Agent: docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f402cd
> kernel/4.15.0-60-generic os/linux arch/amd64
> UpstreamClient(Docker-Client/18.09.7 \(linux\))
> {code}
> To this, it receives response with a "realm" that contains no query arguments:
> {code}
> HTTP/1.1 401 Unauthorized
> Connection: close
> Content-Length: 195
> Content-Type: text/html
> Date: Wed, 29 Jan 2020 12:22:43 GMT
> Server: nginx/1.14.2
> Www-Authenticate: Bearer realm="https://nvcr.io/proxy_auth
> {code}
> Then, it composes the scope using the image ref and a hardcoded "pull"
> action:
> https://github.com/docker/distribution/blob/a8371794149d1d95f1e846744b05c87f2f825e5a/registry/client/auth/session.go#L174
> (in a full accordance with this spec:
> https://docs.docker.com/registry/spec/auth/scope/)
> and sends the following request to https://nvcr.io/proxy_auth :
> {code}
> GET /proxy_auth?scope=repository%3Anvidia%2Ftensorflow%3Apull HTTP/1.1
> Host: nvcr.io
> User-Agent: Go-http-client/1.1
> {code}
> (Note that 'push' is absent from the scope)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (MESOS-10092) Cannot pull image from docker registry which does not reply with 'scope'/'service' in WWW-Authenticate header
[
https://issues.apache.org/jira/browse/MESOS-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17044728#comment-17044728
]
Vinod Kone commented on MESOS-10092:
[~asekretenko] Should this be resolved? Also, is this being backported?
> Cannot pull image from docker registry which does not reply with
> 'scope'/'service' in WWW-Authenticate header
> -
>
> Key: MESOS-10092
> URL: https://issues.apache.org/jira/browse/MESOS-10092
> Project: Mesos
> Issue Type: Bug
>Reporter: Andrei Sekretenko
>Assignee: Andrei Sekretenko
>Priority: Critical
>
> This problem was encountered when trying to specify container image
> nvcr.io/nvidia/tensorflow:19.12-tf1-py3
> When initiating Docker Registry authentication
> (https://docs.docker.com/registry/spec/auth/token/) with nvcr.io, Mesos URI
> fetcher receives 'WWW-Authenticate' header without 'service' and 'scope'
> params, and fails here:
> https://github.com/apache/mesos/blob/1e9b121273a6d9248a78ab44798bd4c1138c31ee/src/uri/fetchers/docker.cpp#L1083
> This is an example of an unsuccessful request made by Mesos:
> {code}
> curl -s -S -L -i --raw --http1.1 -H "Accept:
> application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws"
> -y 60 https://nvcr.io/v2/nvidia/tensorflow/manifests/19.08-py3
> HTTP/1.1 401 Unauthorized
> Content-Type: text/html
> Date: Wed, 22 Jan 2020 19:01:57 GMT
> Server: nginx/1.14.2
> Www-Authenticate: Bearer
> realm="https://nvcr.io/proxy_auth?scope=repository:nvidia/tensorflow:pull,push";
> Content-Length: 195
> Connection: keep-alive
>
> 401 Authorization Required
>
> 401 Authorization Required
> nginx/1.14.2
>
>
> {code}
> At the same time, docker is perfectly capable of pulling this image.
> Note that the document "Token Authentication Specification"
> (https://docs.docker.com/registry/spec/auth/token/), on which the Mesos
> implementation is based, is vague on the issue of registries that do not
> provide 'scope'/'service' in WWW-Authenticate header.
> What Docker does differently (at the very least, in the case of nvcr.io):
> It sends the initial request not to the maniferst/blob URI, but to the
> repository root URI (http:://nvcr.io/v2 in this case):
> {code}
> GET /v2/ HTTP/1.1
> Host: nvcr.io
> User-Agent: docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f402cd
> kernel/4.15.0-60-generic os/linux arch/amd64
> UpstreamClient(Docker-Client/18.09.7 \(linux\))
> {code}
> To this, it receives response with a "realm" that contains no query arguments:
> {code}
> HTTP/1.1 401 Unauthorized
> Connection: close
> Content-Length: 195
> Content-Type: text/html
> Date: Wed, 29 Jan 2020 12:22:43 GMT
> Server: nginx/1.14.2
> Www-Authenticate: Bearer realm="https://nvcr.io/proxy_auth
> {code}
> Then, it composes the scope using the image ref and a hardcoded "pull"
> action:
> https://github.com/docker/distribution/blob/a8371794149d1d95f1e846744b05c87f2f825e5a/registry/client/auth/session.go#L174
> (in a full accordance with this spec:
> https://docs.docker.com/registry/spec/auth/scope/)
> and sends the following request to https://nvcr.io/proxy_auth :
> {code}
> GET /proxy_auth?scope=repository%3Anvidia%2Ftensorflow%3Apull HTTP/1.1
> Host: nvcr.io
> User-Agent: Go-http-client/1.1
> {code}
> (Note that 'push' is absent from the scope)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (MESOS-10092) Cannot pull image from docker registry which does not reply with 'scope'/'service' in WWW-Authenticate header
[
https://issues.apache.org/jira/browse/MESOS-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17031894#comment-17031894
]
Andrei Sekretenko commented on MESOS-10092:
---
{noformat}
{noformat}
commit 4ed912c6b6eee95ad5a1267fad20b7c279ccc153
Author: Andrei Sekretenko
Date: Thu Jan 30 12:39:31 2020 +0100
Factored out common code for building URIs on a registry host.
This is a prerequisite for adding fallback authorization server URI
generation (see MESOS-10092) in the next patches, which will need one
more URI extractor similar to `getManifestUri()`/`getBlobUri()`.
Review: https://reviews.apache.org/r/72077
{noformat}
{noformat}
commit d97a01b5f7e5351564b530c08ee6fd937ef62732
Author: Andrei Sekretenko
Date: Thu Jan 30 13:02:45 2020 +0100
Factored out getting authorization service URI.
Review: https://reviews.apache.org/r/72078
{noformat}
{noformat}
commit c37e6df142322917288aa01e7226f9a5fb59125d
Author: Andrei Sekretenko
Date: Thu Jan 30 14:26:38 2020 +0100
Added workaround for Docker repositories not providing scope/service.
This patch adds a fallback Docker authorization server URI generation
mechanism (see MESOS-10092) for repository servers that provide no
"scope"/"service" params in the "WWW-Authenticate" header of the initial
"401 Unathorized" response.
Review: https://reviews.apache.org/r/72079
{noformat}
{noformat}
commit b7076c9fe5ece573361dcfb899962f4f30df2fd2
Author: Andrei Sekretenko
Date: Thu Jan 23 12:42:33 2020 +0100
Added test for fetch from repositories not providing scope/service.
Review: https://reviews.apache.org/r/72080
{noformat}
> Cannot pull image from docker registry which does not reply with
> 'scope'/'service' in WWW-Authenticate header
> -
>
> Key: MESOS-10092
> URL: https://issues.apache.org/jira/browse/MESOS-10092
> Project: Mesos
> Issue Type: Bug
>Reporter: Andrei Sekretenko
>Assignee: Andrei Sekretenko
>Priority: Critical
>
> This problem was encountered when trying to specify container image
> nvcr.io/nvidia/tensorflow:19.12-tf1-py3
> When initiating Docker Registry authentication
> (https://docs.docker.com/registry/spec/auth/token/) with nvcr.io, Mesos URI
> fetcher receives 'WWW-Authenticate' header without 'service' and 'scope'
> params, and fails here:
> https://github.com/apache/mesos/blob/1e9b121273a6d9248a78ab44798bd4c1138c31ee/src/uri/fetchers/docker.cpp#L1083
> This is an example of an unsuccessful request made by Mesos:
> {code}
> curl -s -S -L -i --raw --http1.1 -H "Accept:
> application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws"
> -y 60 https://nvcr.io/v2/nvidia/tensorflow/manifests/19.08-py3
> HTTP/1.1 401 Unauthorized
> Content-Type: text/html
> Date: Wed, 22 Jan 2020 19:01:57 GMT
> Server: nginx/1.14.2
> Www-Authenticate: Bearer
> realm="https://nvcr.io/proxy_auth?scope=repository:nvidia/tensorflow:pull,push";
> Content-Length: 195
> Connection: keep-alive
>
> 401 Authorization Required
>
> 401 Authorization Required
> nginx/1.14.2
>
>
> {code}
> At the same time, docker is perfectly capable of pulling this image.
> Note that the document "Token Authentication Specification"
> (https://docs.docker.com/registry/spec/auth/token/), on which the Mesos
> implementation is based, is vague on the issue of registries that do not
> provide 'scope'/'service' in WWW-Authenticate header.
> What Docker does differently (at the very least, in the case of nvcr.io):
> It sends the initial request not to the maniferst/blob URI, but to the
> repository root URI (http:://nvcr.io/v2 in this case):
> {code}
> GET /v2/ HTTP/1.1
> Host: nvcr.io
> User-Agent: docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f402cd
> kernel/4.15.0-60-generic os/linux arch/amd64
> UpstreamClient(Docker-Client/18.09.7 \(linux\))
> {code}
> To this, it receives response with a "realm" that contains no query arguments:
> {code}
> HTTP/1.1 401 Unauthorized
> Connection: close
> Content-Length: 195
> Content-Type: text/html
> Date: Wed, 29 Jan 2020 12:22:43 GMT
> Server: nginx/1.14.2
> Www-Authenticate: Bearer realm="https://nvcr.io/proxy_auth
> {code}
> Then, it composes the scope using the image ref and a hardcoded "pull"
> action:
> https://github.com/docker/distribution/blob/a8371794149d1d95f1e846744b05c87f2f825e5a/registry/client/auth/session.go#L174
> (in a full accordance with this spec:
> https://docs.docker.com/registry/spec/auth/scope/)
> and sends the following request to https://nvcr.io/proxy_auth :
> {code}
> GET /proxy_auth?scope=repository%3Anvidia%2Ftensorflow%3Apull HTT
[jira] [Commented] (MESOS-10092) Cannot pull image from docker registry which does not reply with 'scope'/'service' in WWW-Authenticate header
[
https://issues.apache.org/jira/browse/MESOS-10092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17029146#comment-17029146
]
Andrei Sekretenko commented on MESOS-10092:
---
Briefly discussed with [~vinodkone] and [~abudnik]; we came to conclusion that
the safest approach to fix this is the following:
First, we query the repository the old way, with the manifest/blob URI.
If it replies with both three of service, scope and realm, we assume that they
know what they are doing and proceed with building auth server URI as before.
If the initial `WWW-Authenticate` is missing any of the three, then, instead of
failing, we re-send the initial query to the repository root URI (nvcr.io/v2 in
this example), get the realm from the response and compose the scope ourselves
(like Docker does).
> Cannot pull image from docker registry which does not reply with
> 'scope'/'service' in WWW-Authenticate header
> -
>
> Key: MESOS-10092
> URL: https://issues.apache.org/jira/browse/MESOS-10092
> Project: Mesos
> Issue Type: Bug
>Reporter: Andrei Sekretenko
>Assignee: Andrei Sekretenko
>Priority: Critical
>
> This problem was encountered when trying to specify container image
> nvcr.io/nvidia/tensorflow:19.12-tf1-py3
> When initiating Docker Registry authentication
> (https://docs.docker.com/registry/spec/auth/token/) with nvcr.io, Mesos URI
> fetcher receives 'WWW-Authenticate' header without 'service' and 'scope'
> params, and fails here:
> https://github.com/apache/mesos/blob/1e9b121273a6d9248a78ab44798bd4c1138c31ee/src/uri/fetchers/docker.cpp#L1083
> This is an example of an unsuccessful request made by Mesos:
> {code}
> curl -s -S -L -i --raw --http1.1 -H "Accept:
> application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws"
> -y 60 https://nvcr.io/v2/nvidia/tensorflow/manifests/19.08-py3
> HTTP/1.1 401 Unauthorized
> Content-Type: text/html
> Date: Wed, 22 Jan 2020 19:01:57 GMT
> Server: nginx/1.14.2
> Www-Authenticate: Bearer
> realm="https://nvcr.io/proxy_auth?scope=repository:nvidia/tensorflow:pull,push";
> Content-Length: 195
> Connection: keep-alive
>
> 401 Authorization Required
>
> 401 Authorization Required
> nginx/1.14.2
>
>
> {code}
> At the same time, docker is perfectly capable of pulling this image.
> Note that the document "Token Authentication Specification"
> (https://docs.docker.com/registry/spec/auth/token/), on which the Mesos
> implementation is based, is vague on the issue of registries that do not
> provide 'scope'/'service' in WWW-Authenticate header.
> What Docker does differently (at the very least, in the case of nvcr.io):
> It sends the initial request not to the maniferst/blob URI, but to the
> repository root URI (http:://nvcr.io/v2 in this case):
> {code}
> GET /v2/ HTTP/1.1
> Host: nvcr.io
> User-Agent: docker/18.03.1-ce go/go1.9.5 git-commit/9ee9f402cd
> kernel/4.15.0-60-generic os/linux arch/amd64
> UpstreamClient(Docker-Client/18.09.7 \(linux\))
> {code}
> To this, it receives response with a "realm" that contains no query arguments:
> {code}
> HTTP/1.1 401 Unauthorized
> Connection: close
> Content-Length: 195
> Content-Type: text/html
> Date: Wed, 29 Jan 2020 12:22:43 GMT
> Server: nginx/1.14.2
> Www-Authenticate: Bearer realm="https://nvcr.io/proxy_auth
> {code}
> Then, it composes the scope using the image ref and a hardcoded "pull"
> action:
> https://github.com/docker/distribution/blob/a8371794149d1d95f1e846744b05c87f2f825e5a/registry/client/auth/session.go#L174
> (in a full accordance with this spec:
> https://docs.docker.com/registry/spec/auth/scope/)
> and sends the following request to https://nvcr.io/proxy_auth :
> {code}
> GET /proxy_auth?scope=repository%3Anvidia%2Ftensorflow%3Apull HTTP/1.1
> Host: nvcr.io
> User-Agent: Go-http-client/1.1
> {code}
> (Note that 'push' is absent from the scope)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
