Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On 11/05/15 11:15 -0400, Vincent Batts wrote:
On 11/05/15 17:07 +0200, Pavel Odvody wrote:
On Mon, 2015-05-11 at 10:32 -0400, Vincent Batts wrote:
On 09/05/15 00:31 +0200, Pavel Odvody wrote:
On Fri, 2015-05-08 at 14:33 -0400, Vincent Batts wrote:
On 08/05/15 11:31 +1000, Daurnimator wrote:
On 8 May 2015 at 01:46, Pavel Odvody podv...@redhat.com wrote:
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
Is this really required?
Can we request they change something server side?
I would have to double check the behavior on their docker hub, but for
local registries this user agent header is not required. It is the
expectation that a docker registry can always be served as a static file
tree (pull only).
vb
Hey,
$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
!DOCTYPE HTML PUBLIC -//W3C//DTD HTML 3.2 Final//EN
title404 Not Found/title
h1Not Found/h1
pThe requested URL was not found on the server. If you entered the URL manually
please check your spelling and try again./p
$ curl -XGET -HUser-Agent: docker/1.6.0
https://registry-1.docker.io/v2/library/node/manifests/latest
{errors:[{code:UNAUTHORIZED,message:access to the requested resource is not
authorized,detail:[{Type:repository,Name:library/node,Action:pull}]}]}
I actually added a little clarification in my 5th patch:
User-Agent: do /* otherwise we get load-balanced(!) to a V1 registyry */
(I got this information from Andy G.)
The second request obviously fails due to the bearer token not being provided,
but at least we can see that we're hitting the correct endpoint here.
I think that this is the correct behavior, since the original systemd-pull
workflow was to check the Hub first and obtain the token, which I'm simply
following here, however the token is now obtained from a separate endpoint.
The thing is that the argument is --dkr-index-url, so we're actually specifying
the Hub URL here and there's no way to specify a registry alone.
(the mirror registry is received in HTTP headers from the Hub)
Sounds like a topic for another patch?
I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips ...
I understand they've done this on their hub, to route client versions
1.6.0 which can not do the v2 api. There ought to be a way not no
require UA headers. Will see what I can do.
vb
I find that solution rather unfortunate, since the endpoints are already
versioned /v1/ and /v2/ respectively.
The clients before 1.6.0 have hardcoded the /v1/ endpoints so really no
additional reason for this. (oh, didn't 1.5.0 ship half-assed
implementation of v2 with old header names?)
My same argument to them. The issue has been raised with them.
Cool. My raised issue was heard and acted on. Now the UA header is not
required on the hub either.
$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
{errors:[{code:UNAUTHORIZED,message:access to the requested
resource is not
authorized,detail:[{Type:repository,Name:library/node,Action:pull}]}]}
vb
pgp976fDwuTnv.pgp
Description: PGP signature
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On Mon, 2015-05-11 at 10:32 -0400, Vincent Batts wrote:
On 09/05/15 00:31 +0200, Pavel Odvody wrote:
On Fri, 2015-05-08 at 14:33 -0400, Vincent Batts wrote:
On 08/05/15 11:31 +1000, Daurnimator wrote:
On 8 May 2015 at 01:46, Pavel Odvody podv...@redhat.com wrote:
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
Is this really required?
Can we request they change something server side?
I would have to double check the behavior on their docker hub, but for
local registries this user agent header is not required. It is the
expectation that a docker registry can always be served as a static file
tree (pull only).
vb
Hey,
$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
!DOCTYPE HTML PUBLIC -//W3C//DTD HTML 3.2 Final//EN
title404 Not Found/title
h1Not Found/h1
pThe requested URL was not found on the server. If you entered the URL
manually please check your spelling and try again./p
$ curl -XGET -HUser-Agent: docker/1.6.0
https://registry-1.docker.io/v2/library/node/manifests/latest
{errors:[{code:UNAUTHORIZED,message:access to the requested
resource is not
authorized,detail:[{Type:repository,Name:library/node,Action:pull}]}]}
I actually added a little clarification in my 5th patch:
User-Agent: do /* otherwise we get load-balanced(!) to a V1 registyry */
(I got this information from Andy G.)
The second request obviously fails due to the bearer token not being
provided,
but at least we can see that we're hitting the correct endpoint here.
I think that this is the correct behavior, since the original systemd-pull
workflow was to check the Hub first and obtain the token, which I'm simply
following here, however the token is now obtained from a separate endpoint.
The thing is that the argument is --dkr-index-url, so we're actually
specifying
the Hub URL here and there's no way to specify a registry alone.
(the mirror registry is received in HTTP headers from the Hub)
Sounds like a topic for another patch?
I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips
...
I understand they've done this on their hub, to route client versions
1.6.0 which can not do the v2 api. There ought to be a way not no
require UA headers. Will see what I can do.
vb
I find that solution rather unfortunate, since the endpoints are already
versioned /v1/ and /v2/ respectively.
The clients before 1.6.0 have hardcoded the /v1/ endpoints so really no
additional reason for this. (oh, didn't 1.5.0 ship half-assed
implementation of v2 with old header names?)
--
Pavel Odvody podv...@redhat.com
Software Engineer - EMEA ENG Developer Experience
5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D
Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno
signature.asc
Description: This is a digitally signed message part
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On 11/05/15 17:07 +0200, Pavel Odvody wrote:
On Mon, 2015-05-11 at 10:32 -0400, Vincent Batts wrote:
On 09/05/15 00:31 +0200, Pavel Odvody wrote:
On Fri, 2015-05-08 at 14:33 -0400, Vincent Batts wrote:
On 08/05/15 11:31 +1000, Daurnimator wrote:
On 8 May 2015 at 01:46, Pavel Odvody podv...@redhat.com wrote:
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
Is this really required?
Can we request they change something server side?
I would have to double check the behavior on their docker hub, but for
local registries this user agent header is not required. It is the
expectation that a docker registry can always be served as a static file
tree (pull only).
vb
Hey,
$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
!DOCTYPE HTML PUBLIC -//W3C//DTD HTML 3.2 Final//EN
title404 Not Found/title
h1Not Found/h1
pThe requested URL was not found on the server. If you entered the URL manually
please check your spelling and try again./p
$ curl -XGET -HUser-Agent: docker/1.6.0
https://registry-1.docker.io/v2/library/node/manifests/latest
{errors:[{code:UNAUTHORIZED,message:access to the requested resource is not
authorized,detail:[{Type:repository,Name:library/node,Action:pull}]}]}
I actually added a little clarification in my 5th patch:
User-Agent: do /* otherwise we get load-balanced(!) to a V1 registyry */
(I got this information from Andy G.)
The second request obviously fails due to the bearer token not being provided,
but at least we can see that we're hitting the correct endpoint here.
I think that this is the correct behavior, since the original systemd-pull
workflow was to check the Hub first and obtain the token, which I'm simply
following here, however the token is now obtained from a separate endpoint.
The thing is that the argument is --dkr-index-url, so we're actually specifying
the Hub URL here and there's no way to specify a registry alone.
(the mirror registry is received in HTTP headers from the Hub)
Sounds like a topic for another patch?
I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips
...
I understand they've done this on their hub, to route client versions
1.6.0 which can not do the v2 api. There ought to be a way not no
require UA headers. Will see what I can do.
vb
I find that solution rather unfortunate, since the endpoints are already
versioned /v1/ and /v2/ respectively.
The clients before 1.6.0 have hardcoded the /v1/ endpoints so really no
additional reason for this. (oh, didn't 1.5.0 ship half-assed
implementation of v2 with old header names?)
My same argument to them. The issue has been raised with them.
vb
pgp3Q8INa7sR9.pgp
Description: PGP signature
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On 09/05/15 00:31 +0200, Pavel Odvody wrote:
On Fri, 2015-05-08 at 14:33 -0400, Vincent Batts wrote:
On 08/05/15 11:31 +1000, Daurnimator wrote:
On 8 May 2015 at 01:46, Pavel Odvody podv...@redhat.com wrote:
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
Is this really required?
Can we request they change something server side?
I would have to double check the behavior on their docker hub, but for
local registries this user agent header is not required. It is the
expectation that a docker registry can always be served as a static file
tree (pull only).
vb
Hey,
$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
!DOCTYPE HTML PUBLIC -//W3C//DTD HTML 3.2 Final//EN
title404 Not Found/title
h1Not Found/h1
pThe requested URL was not found on the server. If you entered the URL manually
please check your spelling and try again./p
$ curl -XGET -HUser-Agent: docker/1.6.0
https://registry-1.docker.io/v2/library/node/manifests/latest
{errors:[{code:UNAUTHORIZED,message:access to the requested resource is not
authorized,detail:[{Type:repository,Name:library/node,Action:pull}]}]}
I actually added a little clarification in my 5th patch:
User-Agent: do /* otherwise we get load-balanced(!) to a V1 registyry */
(I got this information from Andy G.)
The second request obviously fails due to the bearer token not being provided,
but at least we can see that we're hitting the correct endpoint here.
I think that this is the correct behavior, since the original systemd-pull
workflow was to check the Hub first and obtain the token, which I'm simply
following here, however the token is now obtained from a separate endpoint.
The thing is that the argument is --dkr-index-url, so we're actually specifying
the Hub URL here and there's no way to specify a registry alone.
(the mirror registry is received in HTTP headers from the Hub)
Sounds like a topic for another patch?
I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips ...
I understand they've done this on their hub, to route client versions
1.6.0 which can not do the v2 api. There ought to be a way not no
require UA headers. Will see what I can do.
vb
pgpv1TcYooU4Y.pgp
Description: PGP signature
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On 08/05/15 11:31 +1000, Daurnimator wrote: On 8 May 2015 at 01:46, Pavel Odvody podv...@redhat.com wrote: - To access the V2 registry we need to send a special User-Agent docker/1.6.0 Is this really required? Can we request they change something server side? I would have to double check the behavior on their docker hub, but for local registries this user agent header is not required. It is the expectation that a docker registry can always be served as a static file tree (pull only). vb pgp9IcTFQpyb0.pgp Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On Fri, 2015-05-08 at 14:33 -0400, Vincent Batts wrote:
On 08/05/15 11:31 +1000, Daurnimator wrote:
On 8 May 2015 at 01:46, Pavel Odvody podv...@redhat.com wrote:
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
Is this really required?
Can we request they change something server side?
I would have to double check the behavior on their docker hub, but for
local registries this user agent header is not required. It is the
expectation that a docker registry can always be served as a static file
tree (pull only).
vb
Hey,
$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
!DOCTYPE HTML PUBLIC -//W3C//DTD HTML 3.2 Final//EN
title404 Not Found/title
h1Not Found/h1
pThe requested URL was not found on the server. If you entered the URL
manually please check your spelling and try again./p
$ curl -XGET -HUser-Agent: docker/1.6.0
https://registry-1.docker.io/v2/library/node/manifests/latest
{errors:[{code:UNAUTHORIZED,message:access to the requested resource
is not
authorized,detail:[{Type:repository,Name:library/node,Action:pull}]}]}
I actually added a little clarification in my 5th patch:
User-Agent: do /* otherwise we get load-balanced(!) to a V1 registyry */
(I got this information from Andy G.)
The second request obviously fails due to the bearer token not being provided,
but at least we can see that we're hitting the correct endpoint here.
I think that this is the correct behavior, since the original systemd-pull
workflow was to check the Hub first and obtain the token, which I'm simply
following here, however the token is now obtained from a separate endpoint.
The thing is that the argument is --dkr-index-url, so we're actually specifying
the Hub URL here and there's no way to specify a registry alone.
(the mirror registry is received in HTTP headers from the Hub)
Sounds like a topic for another patch?
I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips ...
--
Pavel Odvody podv...@redhat.com
Software Engineer - EMEA ENG Developer Experience
5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D
Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno
signature.asc
Description: This is a digitally signed message part
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On 8 May 2015 at 01:46, Pavel Odvody podv...@redhat.com wrote: - To access the V2 registry we need to send a special User-Agent docker/1.6.0 Is this really required? Can we request they change something server side? ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
Hi, the attached series of patches add support for pulling from V2 docker registries, so let me break down first what happened to the format since V1 - Image is now defined by a JSON manifest - contains fields like name, tag, schemaVersion ... - and fsLayers - which is an array of sha256 references to a *content-addressable FS layers* - the manifest is now also signed using JWS/JWT (ECDSA p-256 mostly) - Authentication/Authorization now bearer token only - To access the V2 registry we need to send a special User-Agent docker/1.6.0 - The whole manifest can be hashed using sha256 to obtain a digest, which provides an immutable global identifier of the image, and can be used instead of a tag when pulling the image (the REST API endpoints are the same). So far so good, now what's in the patches, besides the V2 workflow - lightweight JSON parser, written around json_tokenize - I've renamed 'tag' to 'reference' to accommodate for the digest semantics - all layers are saved in a directory .dkr-$imageid - image id is resolved from the v1 compatibility section of the manifest - since the layers are now CAS, we can't assume that the order, or mere presence of certain layers will be preserved throughout multitude of images/manifests, and therefore due to the incremental nature of BTRFS snapshots we need to throw any intermediary snapshots away. - small bugfix for the JSON tokenizer (it'd choke after reading any digit) This is the bare minimum to pullrun V2 images, since the signature is now embedded in the manifest, it could now support --verify=signature. However, I've got one open question - how do we support V1/V2 concurrently (this patch makes V2 the default and only)? Docker first pings the V2 endpoint and then falls back to V1, but I think that this is sub optimal, since --verify=signature makes sense only with V2, so I think something like --dkr-pull-strategy=v1|v2 as an argument would be the best? Thanks, Pavel Pavel Odvody (5): shared/import-util: tag renamed to reference to support v2 pull by digest shared/json: JSON parser + number tokenizer bugfix test/test-json: Tests for the JSON parser and the tokenizer bugfix import/pull: Tag replaced with reference import/pull-dkr: V2 Image specification + manifest support src/import/pull-dkr.c| 531 +-- src/import/pull-dkr.h| 48 - src/import/pull.c| 28 ++- src/shared/import-util.c | 19 ++ src/shared/import-util.h | 1 + src/shared/json.c| 437 +- src/shared/json.h| 36 src/test/test-json.c | 16 ++ 8 files changed, 1034 insertions(+), 82 deletions(-) -- 2.1.0 signature.asc Description: This is a digitally signed message part ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
