Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On 11/05/15 11:15 -0400, Vincent Batts wrote: On 11/05/15 17:07 +0200, Pavel Odvody wrote: On Mon, 2015-05-11 at 10:32 -0400, Vincent Batts wrote: On 09/05/15 00:31 +0200, Pavel Odvody wrote: On Fri, 2015-05-08 at 14:33 -0400, Vincent Batts wrote: On 08/05/15 11:31 +1000, Daurnimator wrote: >On 8 May 2015 at 01:46, Pavel Odvody wrote: >> - To access the V2 registry we need to send a special User-Agent >>docker/1.6.0 > >Is this really required? >Can we request they change something server side? I would have to double check the behavior on their docker hub, but for local registries this user agent header is not required. It is the expectation that a docker registry can always be served as a static file tree (pull only). vb Hey, $ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest 404 Not Found Not Found The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again. $ curl -XGET -H"User-Agent: docker/1.6.0" https://registry-1.docker.io/v2/library/node/manifests/latest {"errors":[{"code":"UNAUTHORIZED","message":"access to the requested resource is not authorized","detail":[{"Type":"repository","Name":"library/node","Action":"pull"}]}]} I actually added a little clarification in my 5th patch: "User-Agent: do" /* otherwise we get load-balanced(!) to a V1 registyry */ (I got this information from Andy G.) The second request obviously fails due to the bearer token not being provided, but at least we can see that we're hitting the correct endpoint here. I think that this is the correct behavior, since the original systemd-pull workflow was to check the Hub first and obtain the token, which I'm simply following here, however the token is now obtained from a separate endpoint. The thing is that the argument is --dkr-index-url, so we're actually specifying the Hub URL here and there's no way to specify a registry alone. (the "mirror" registry is received in HTTP headers from the Hub) Sounds like a topic for another patch? I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips ... I understand they've done this on their hub, to route client versions < 1.6.0 which can not do the v2 api. There ought to be a way not no require UA headers. Will see what I can do. vb I find that solution rather unfortunate, since the endpoints are already versioned /v1/ and /v2/ respectively. The clients before 1.6.0 have hardcoded the /v1/ endpoints so really no additional reason for this. (oh, didn't 1.5.0 ship half-assed implementation of v2 with old header names?) My same argument to them. The issue has been raised with them. Cool. My raised issue was heard and acted on. Now the UA header is not required on the hub either. $ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest {"errors":[{"code":"UNAUTHORIZED","message":"access to the requested resource is not authorized","detail":[{"Type":"repository","Name":"library/node","Action":"pull"}]}]} vb pgp976fDwuTnv.pgp Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On 11/05/15 17:07 +0200, Pavel Odvody wrote: On Mon, 2015-05-11 at 10:32 -0400, Vincent Batts wrote: On 09/05/15 00:31 +0200, Pavel Odvody wrote: >On Fri, 2015-05-08 at 14:33 -0400, Vincent Batts wrote: >> On 08/05/15 11:31 +1000, Daurnimator wrote: >> >On 8 May 2015 at 01:46, Pavel Odvody wrote: >> >> - To access the V2 registry we need to send a special User-Agent >> >>docker/1.6.0 >> > >> >Is this really required? >> >Can we request they change something server side? >> >> I would have to double check the behavior on their docker hub, but for >> local registries this user agent header is not required. It is the >> expectation that a docker registry can always be served as a static file >> tree (pull only). >> >> vb > >Hey, > >$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest > >404 Not Found >Not Found >The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again. > >$ curl -XGET -H"User-Agent: docker/1.6.0" https://registry-1.docker.io/v2/library/node/manifests/latest >{"errors":[{"code":"UNAUTHORIZED","message":"access to the requested resource is not authorized","detail":[{"Type":"repository","Name":"library/node","Action":"pull"}]}]} > >I actually added a little clarification in my 5th patch: > >"User-Agent: do" /* otherwise we get load-balanced(!) to a V1 registyry */ >(I got this information from Andy G.) > >The second request obviously fails due to the bearer token not being provided, >but at least we can see that we're hitting the correct endpoint here. > >I think that this is the correct behavior, since the original systemd-pull >workflow was to check the Hub first and obtain the token, which I'm simply >following here, however the token is now obtained from a separate endpoint. > >The thing is that the argument is --dkr-index-url, so we're actually specifying >the Hub URL here and there's no way to specify a registry alone. >(the "mirror" registry is received in HTTP headers from the Hub) > >Sounds like a topic for another patch? > >I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips ... I understand they've done this on their hub, to route client versions < 1.6.0 which can not do the v2 api. There ought to be a way not no require UA headers. Will see what I can do. vb I find that solution rather unfortunate, since the endpoints are already versioned /v1/ and /v2/ respectively. The clients before 1.6.0 have hardcoded the /v1/ endpoints so really no additional reason for this. (oh, didn't 1.5.0 ship half-assed implementation of v2 with old header names?) My same argument to them. The issue has been raised with them. vb pgp3Q8INa7sR9.pgp Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On Mon, 2015-05-11 at 10:32 -0400, Vincent Batts wrote: > On 09/05/15 00:31 +0200, Pavel Odvody wrote: > >On Fri, 2015-05-08 at 14:33 -0400, Vincent Batts wrote: > >> On 08/05/15 11:31 +1000, Daurnimator wrote: > >> >On 8 May 2015 at 01:46, Pavel Odvody wrote: > >> >> - To access the V2 registry we need to send a special User-Agent > >> >>docker/1.6.0 > >> > > >> >Is this really required? > >> >Can we request they change something server side? > >> > >> I would have to double check the behavior on their docker hub, but for > >> local registries this user agent header is not required. It is the > >> expectation that a docker registry can always be served as a static file > >> tree (pull only). > >> > >> vb > > > >Hey, > > > >$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest > > > >404 Not Found > >Not Found > >The requested URL was not found on the server. If you entered the URL > >manually please check your spelling and try again. > > > >$ curl -XGET -H"User-Agent: docker/1.6.0" > >https://registry-1.docker.io/v2/library/node/manifests/latest > >{"errors":[{"code":"UNAUTHORIZED","message":"access to the requested > >resource is not > >authorized","detail":[{"Type":"repository","Name":"library/node","Action":"pull"}]}]} > > > >I actually added a little clarification in my 5th patch: > > > >"User-Agent: do" /* otherwise we get load-balanced(!) to a V1 registyry */ > >(I got this information from Andy G.) > > > >The second request obviously fails due to the bearer token not being > >provided, > >but at least we can see that we're hitting the correct endpoint here. > > > >I think that this is the correct behavior, since the original systemd-pull > >workflow was to check the Hub first and obtain the token, which I'm simply > >following here, however the token is now obtained from a separate endpoint. > > > >The thing is that the argument is --dkr-index-url, so we're actually > >specifying > >the Hub URL here and there's no way to specify a registry alone. > >(the "mirror" registry is received in HTTP headers from the Hub) > > > >Sounds like a topic for another patch? > > > >I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips > >... > > I understand they've done this on their hub, to route client versions > < 1.6.0 which can not do the v2 api. There ought to be a way not no > require UA headers. Will see what I can do. > > vb I find that solution rather unfortunate, since the endpoints are already versioned /v1/ and /v2/ respectively. The clients before 1.6.0 have hardcoded the /v1/ endpoints so really no additional reason for this. (oh, didn't 1.5.0 ship half-assed implementation of v2 with old header names?) -- Pavel Odvody Software Engineer - EMEA ENG Developer Experience 5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno signature.asc Description: This is a digitally signed message part ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On 09/05/15 00:31 +0200, Pavel Odvody wrote: On Fri, 2015-05-08 at 14:33 -0400, Vincent Batts wrote: On 08/05/15 11:31 +1000, Daurnimator wrote: >On 8 May 2015 at 01:46, Pavel Odvody wrote: >> - To access the V2 registry we need to send a special User-Agent >>docker/1.6.0 > >Is this really required? >Can we request they change something server side? I would have to double check the behavior on their docker hub, but for local registries this user agent header is not required. It is the expectation that a docker registry can always be served as a static file tree (pull only). vb Hey, $ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest 404 Not Found Not Found The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again. $ curl -XGET -H"User-Agent: docker/1.6.0" https://registry-1.docker.io/v2/library/node/manifests/latest {"errors":[{"code":"UNAUTHORIZED","message":"access to the requested resource is not authorized","detail":[{"Type":"repository","Name":"library/node","Action":"pull"}]}]} I actually added a little clarification in my 5th patch: "User-Agent: do" /* otherwise we get load-balanced(!) to a V1 registyry */ (I got this information from Andy G.) The second request obviously fails due to the bearer token not being provided, but at least we can see that we're hitting the correct endpoint here. I think that this is the correct behavior, since the original systemd-pull workflow was to check the Hub first and obtain the token, which I'm simply following here, however the token is now obtained from a separate endpoint. The thing is that the argument is --dkr-index-url, so we're actually specifying the Hub URL here and there's no way to specify a registry alone. (the "mirror" registry is received in HTTP headers from the Hub) Sounds like a topic for another patch? I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips ... I understand they've done this on their hub, to route client versions < 1.6.0 which can not do the v2 api. There ought to be a way not no require UA headers. Will see what I can do. vb pgpv1TcYooU4Y.pgp Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On Fri, 2015-05-08 at 14:33 -0400, Vincent Batts wrote: > On 08/05/15 11:31 +1000, Daurnimator wrote: > >On 8 May 2015 at 01:46, Pavel Odvody wrote: > >> - To access the V2 registry we need to send a special User-Agent > >>docker/1.6.0 > > > >Is this really required? > >Can we request they change something server side? > > I would have to double check the behavior on their docker hub, but for > local registries this user agent header is not required. It is the > expectation that a docker registry can always be served as a static file > tree (pull only). > > vb Hey, $ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest 404 Not Found Not Found The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again. $ curl -XGET -H"User-Agent: docker/1.6.0" https://registry-1.docker.io/v2/library/node/manifests/latest {"errors":[{"code":"UNAUTHORIZED","message":"access to the requested resource is not authorized","detail":[{"Type":"repository","Name":"library/node","Action":"pull"}]}]} I actually added a little clarification in my 5th patch: "User-Agent: do" /* otherwise we get load-balanced(!) to a V1 registyry */ (I got this information from Andy G.) The second request obviously fails due to the bearer token not being provided, but at least we can see that we're hitting the correct endpoint here. I think that this is the correct behavior, since the original systemd-pull workflow was to check the Hub first and obtain the token, which I'm simply following here, however the token is now obtained from a separate endpoint. The thing is that the argument is --dkr-index-url, so we're actually specifying the Hub URL here and there's no way to specify a registry alone. (the "mirror" registry is received in HTTP headers from the Hub) Sounds like a topic for another patch? I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips ... -- Pavel Odvody Software Engineer - EMEA ENG Developer Experience 5EC1 95C1 8E08 5BD9 9BBF 9241 3AFA 3A66 024F F68D Red Hat Czech s.r.o., Purkyňova 99/71, 612 45, Brno signature.asc Description: This is a digitally signed message part ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On 08/05/15 11:31 +1000, Daurnimator wrote: On 8 May 2015 at 01:46, Pavel Odvody wrote: - To access the V2 registry we need to send a special User-Agent docker/1.6.0 Is this really required? Can we request they change something server side? I would have to double check the behavior on their docker hub, but for local registries this user agent header is not required. It is the expectation that a docker registry can always be served as a static file tree (pull only). vb pgp9IcTFQpyb0.pgp Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On 8 May 2015 at 01:46, Pavel Odvody wrote: > - To access the V2 registry we need to send a special User-Agent >docker/1.6.0 Is this really required? Can we request they change something server side? ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
Hi, the attached series of patches add support for pulling from V2 docker registries, so let me break down first what happened to the format since V1 - Image is now defined by a JSON manifest - contains fields like name, tag, schemaVersion ... - and fsLayers - which is an array of sha256 references to a *content-addressable FS layers* - the manifest is now also signed using JWS/JWT (ECDSA p-256 mostly) - Authentication/Authorization now bearer token only - To access the V2 registry we need to send a special User-Agent docker/1.6.0 - The whole manifest can be hashed using sha256 to obtain a "digest", which provides an immutable global identifier of the image, and can be used instead of a tag when pulling the image (the REST API endpoints are the same). So far so good, now what's in the patches, besides the V2 workflow - lightweight JSON parser, written around json_tokenize - I've renamed 'tag' to 'reference' to accommodate for the digest semantics - all layers are saved in a directory .dkr-$imageid - image id is resolved from the v1 compatibility section of the manifest - since the layers are now CAS, we can't assume that the order, or mere presence of certain layers will be preserved throughout multitude of images/manifests, and therefore due to the incremental nature of BTRFS snapshots we need to throw any intermediary snapshots away. - small bugfix for the JSON tokenizer (it'd choke after reading any digit) This is the bare minimum to pull&run V2 images, since the signature is now embedded in the manifest, it could now support --verify=signature. However, I've got one open question - how do we support V1/V2 concurrently (this patch makes V2 the default and only)? Docker first pings the V2 endpoint and then falls back to V1, but I think that this is sub optimal, since --verify=signature makes sense only with V2, so I think something like --dkr-pull-strategy=v1|v2 as an argument would be the best? Thanks, Pavel Pavel Odvody (5): shared/import-util: tag renamed to reference to support v2 pull by digest shared/json: JSON parser + number tokenizer bugfix test/test-json: Tests for the JSON parser and the tokenizer bugfix import/pull: Tag replaced with reference import/pull-dkr: V2 Image specification + manifest support src/import/pull-dkr.c| 531 +-- src/import/pull-dkr.h| 48 - src/import/pull.c| 28 ++- src/shared/import-util.c | 19 ++ src/shared/import-util.h | 1 + src/shared/json.c| 437 +- src/shared/json.h| 36 src/test/test-json.c | 16 ++ 8 files changed, 1034 insertions(+), 82 deletions(-) -- 2.1.0 signature.asc Description: This is a digitally signed message part ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel