Re: [systemd-devel] use-case for sparse tar archives
https://github.com/systemd/systemd/pull/998 unless you want to bubble this all the way up to the commandline. On Tue, Aug 18, 2015 at 4:31 PM, Vincent Batts vba...@redhat.com wrote: On Tue, Aug 18, 2015 at 12:29 PM, Lennart Poettering lenn...@poettering.net wrote: On Tue, 18.08.15 12:07, Vincent Batts (vba...@redhat.com) wrote: Before submitting a change for this, why is the creation of tar's using `--sparse`? https://github.com/systemd/systemd/blob/master/src/import/import-common.c#L213 If you save/restore/migrate disk trees that include sparse files like utmp/wtmp or journal files it sounds like a good idea to keep them sparse. Why would you not do this? Lennart It makes reassembling the pristine tar archive almost impossible. https://github.com/vbatts/tar-split is a project of mine to reassemble cryptographically verifiable archives. It's merged into docker now, but i'm looking to investigate portability of containers as it relates to nspawn and machinectl import. vb ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] use-case for sparse tar archives
On Tue, Aug 18, 2015 at 12:29 PM, Lennart Poettering lenn...@poettering.net wrote: On Tue, 18.08.15 12:07, Vincent Batts (vba...@redhat.com) wrote: Before submitting a change for this, why is the creation of tar's using `--sparse`? https://github.com/systemd/systemd/blob/master/src/import/import-common.c#L213 If you save/restore/migrate disk trees that include sparse files like utmp/wtmp or journal files it sounds like a good idea to keep them sparse. Why would you not do this? Lennart It makes reassembling the pristine tar archive almost impossible. https://github.com/vbatts/tar-split is a project of mine to reassemble cryptographically verifiable archives. It's merged into docker now, but i'm looking to investigate portability of containers as it relates to nspawn and machinectl import. vb ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] use-case for sparse tar archives
Before submitting a change for this, why is the creation of tar's using `--sparse`? https://github.com/systemd/systemd/blob/master/src/import/import-common.c#L213 vb ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 5/5] import/pull-dkr: V2 Image specification + manifest support
On 15/05/15 16:24 +0200, Pavel Odvody wrote: On Fri, 2015-05-15 at 15:54 +0200, Lennart Poettering wrote: On Thu, 07.05.15 17:47, Pavel Odvody (podv...@redhat.com) wrote: #define HEADER_TOKEN X-Do /* the HTTP header for the auth token */ cker-Token: -#define HEADER_REGISTRY X-Do /*the HTTP header for the registry */ cker-Endpoints: +#define HEADER_REGISTRY X-Do /* the HTTP header for the registry */ cker-Endpoints: +#define HEADER_DIGEST Do /* the HTTP header for the manifest digest */ cker-Content-Digest: +#define HEADER_USER_AGENT_V2 User-Agent: do /* otherwise we get load-balanced(!) to a V1 registyry */ cker/1.6.0 I hope we can get rid of this. I'd prefer if we wouldn't have to pretend to be other software. As of today we can, kudos to Vincent for pushing this. \ö/ pgp6d98q3knhA.pgp Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On 11/05/15 11:15 -0400, Vincent Batts wrote:
On 11/05/15 17:07 +0200, Pavel Odvody wrote:
On Mon, 2015-05-11 at 10:32 -0400, Vincent Batts wrote:
On 09/05/15 00:31 +0200, Pavel Odvody wrote:
On Fri, 2015-05-08 at 14:33 -0400, Vincent Batts wrote:
On 08/05/15 11:31 +1000, Daurnimator wrote:
On 8 May 2015 at 01:46, Pavel Odvody podv...@redhat.com wrote:
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
Is this really required?
Can we request they change something server side?
I would have to double check the behavior on their docker hub, but for
local registries this user agent header is not required. It is the
expectation that a docker registry can always be served as a static file
tree (pull only).
vb
Hey,
$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
!DOCTYPE HTML PUBLIC -//W3C//DTD HTML 3.2 Final//EN
title404 Not Found/title
h1Not Found/h1
pThe requested URL was not found on the server. If you entered the URL manually
please check your spelling and try again./p
$ curl -XGET -HUser-Agent: docker/1.6.0
https://registry-1.docker.io/v2/library/node/manifests/latest
{errors:[{code:UNAUTHORIZED,message:access to the requested resource is not
authorized,detail:[{Type:repository,Name:library/node,Action:pull}]}]}
I actually added a little clarification in my 5th patch:
User-Agent: do /* otherwise we get load-balanced(!) to a V1 registyry */
(I got this information from Andy G.)
The second request obviously fails due to the bearer token not being provided,
but at least we can see that we're hitting the correct endpoint here.
I think that this is the correct behavior, since the original systemd-pull
workflow was to check the Hub first and obtain the token, which I'm simply
following here, however the token is now obtained from a separate endpoint.
The thing is that the argument is --dkr-index-url, so we're actually specifying
the Hub URL here and there's no way to specify a registry alone.
(the mirror registry is received in HTTP headers from the Hub)
Sounds like a topic for another patch?
I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips ...
I understand they've done this on their hub, to route client versions
1.6.0 which can not do the v2 api. There ought to be a way not no
require UA headers. Will see what I can do.
vb
I find that solution rather unfortunate, since the endpoints are already
versioned /v1/ and /v2/ respectively.
The clients before 1.6.0 have hardcoded the /v1/ endpoints so really no
additional reason for this. (oh, didn't 1.5.0 ship half-assed
implementation of v2 with old header names?)
My same argument to them. The issue has been raised with them.
Cool. My raised issue was heard and acted on. Now the UA header is not
required on the hub either.
$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
{errors:[{code:UNAUTHORIZED,message:access to the requested
resource is not
authorized,detail:[{Type:repository,Name:library/node,Action:pull}]}]}
vb
pgp976fDwuTnv.pgp
Description: PGP signature
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On 11/05/15 17:07 +0200, Pavel Odvody wrote:
On Mon, 2015-05-11 at 10:32 -0400, Vincent Batts wrote:
On 09/05/15 00:31 +0200, Pavel Odvody wrote:
On Fri, 2015-05-08 at 14:33 -0400, Vincent Batts wrote:
On 08/05/15 11:31 +1000, Daurnimator wrote:
On 8 May 2015 at 01:46, Pavel Odvody podv...@redhat.com wrote:
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
Is this really required?
Can we request they change something server side?
I would have to double check the behavior on their docker hub, but for
local registries this user agent header is not required. It is the
expectation that a docker registry can always be served as a static file
tree (pull only).
vb
Hey,
$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
!DOCTYPE HTML PUBLIC -//W3C//DTD HTML 3.2 Final//EN
title404 Not Found/title
h1Not Found/h1
pThe requested URL was not found on the server. If you entered the URL manually
please check your spelling and try again./p
$ curl -XGET -HUser-Agent: docker/1.6.0
https://registry-1.docker.io/v2/library/node/manifests/latest
{errors:[{code:UNAUTHORIZED,message:access to the requested resource is not
authorized,detail:[{Type:repository,Name:library/node,Action:pull}]}]}
I actually added a little clarification in my 5th patch:
User-Agent: do /* otherwise we get load-balanced(!) to a V1 registyry */
(I got this information from Andy G.)
The second request obviously fails due to the bearer token not being provided,
but at least we can see that we're hitting the correct endpoint here.
I think that this is the correct behavior, since the original systemd-pull
workflow was to check the Hub first and obtain the token, which I'm simply
following here, however the token is now obtained from a separate endpoint.
The thing is that the argument is --dkr-index-url, so we're actually specifying
the Hub URL here and there's no way to specify a registry alone.
(the mirror registry is received in HTTP headers from the Hub)
Sounds like a topic for another patch?
I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips
...
I understand they've done this on their hub, to route client versions
1.6.0 which can not do the v2 api. There ought to be a way not no
require UA headers. Will see what I can do.
vb
I find that solution rather unfortunate, since the endpoints are already
versioned /v1/ and /v2/ respectively.
The clients before 1.6.0 have hardcoded the /v1/ endpoints so really no
additional reason for this. (oh, didn't 1.5.0 ship half-assed
implementation of v2 with old header names?)
My same argument to them. The issue has been raised with them.
vb
pgp3Q8INa7sR9.pgp
Description: PGP signature
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On 09/05/15 00:31 +0200, Pavel Odvody wrote:
On Fri, 2015-05-08 at 14:33 -0400, Vincent Batts wrote:
On 08/05/15 11:31 +1000, Daurnimator wrote:
On 8 May 2015 at 01:46, Pavel Odvody podv...@redhat.com wrote:
- To access the V2 registry we need to send a special User-Agent
docker/1.6.0
Is this really required?
Can we request they change something server side?
I would have to double check the behavior on their docker hub, but for
local registries this user agent header is not required. It is the
expectation that a docker registry can always be served as a static file
tree (pull only).
vb
Hey,
$ curl -XGET https://registry-1.docker.io/v2/library/node/manifests/latest
!DOCTYPE HTML PUBLIC -//W3C//DTD HTML 3.2 Final//EN
title404 Not Found/title
h1Not Found/h1
pThe requested URL was not found on the server. If you entered the URL manually
please check your spelling and try again./p
$ curl -XGET -HUser-Agent: docker/1.6.0
https://registry-1.docker.io/v2/library/node/manifests/latest
{errors:[{code:UNAUTHORIZED,message:access to the requested resource is not
authorized,detail:[{Type:repository,Name:library/node,Action:pull}]}]}
I actually added a little clarification in my 5th patch:
User-Agent: do /* otherwise we get load-balanced(!) to a V1 registyry */
(I got this information from Andy G.)
The second request obviously fails due to the bearer token not being provided,
but at least we can see that we're hitting the correct endpoint here.
I think that this is the correct behavior, since the original systemd-pull
workflow was to check the Hub first and obtain the token, which I'm simply
following here, however the token is now obtained from a separate endpoint.
The thing is that the argument is --dkr-index-url, so we're actually specifying
the Hub URL here and there's no way to specify a registry alone.
(the mirror registry is received in HTTP headers from the Hub)
Sounds like a topic for another patch?
I hope that the pull-only policy will be relaxed soon :) A lot of roundtrips ...
I understand they've done this on their hub, to route client versions
1.6.0 which can not do the v2 api. There ought to be a way not no
require UA headers. Will see what I can do.
vb
pgpv1TcYooU4Y.pgp
Description: PGP signature
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] [PATCH 0/5] systemd-importd - support for pulling from V2 Dkr registries
On 08/05/15 11:31 +1000, Daurnimator wrote: On 8 May 2015 at 01:46, Pavel Odvody podv...@redhat.com wrote: - To access the V2 registry we need to send a special User-Agent docker/1.6.0 Is this really required? Can we request they change something server side? I would have to double check the behavior on their docker hub, but for local registries this user agent header is not required. It is the expectation that a docker registry can always be served as a static file tree (pull only). vb pgp9IcTFQpyb0.pgp Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Appc support in systemd-importd
On 11/03/15 16:56 +0100, Lennart Poettering wrote: On Wed, 11.03.15 16:33, Lennart Poettering (lenn...@poettering.net) wrote: * Alban liked the idea of saving the manifest so we can extract the whole ACI and modify nspawn to detect a container is an ACI and set /var/lib/machines/appc-image.aci/rootfs as the container's root. The benefit of keeping the manifest would be knowing which binary to start. Is that acceptable? This would be quite different from dkr handling though. Currently my thinking for dkr is after all that everything gets converted at import time and from that point on is just a raw directory with an associated native config file for nspawn. Or in other words: nspawn + machined have no idea what dkr is, only importd has... I wonder what it would take to make ACI imports work the same way... If I understand things right ACI tarballs come with only two directories: rootfs plus manifest. Maybe it would work to place rootfs for a container foobar directly as directory in /var/lib/machines/foobar, and then placing the manifest as /var/lib/machines/foobar.aci-manifest, with a converted version as /var/lib/machines/foobar.nspawn or so? Thinking about this more: Given the ACI requires some HTTP service to be exposed towards the container I figure we have to teach ACI support anyway to nspawn. And if that's the case we can as well support it all the way in machined and nspawn, and support its untarred stuff natively without rearranging the directories. I figure Alban's idea is good then! This intermediary state information for containers, is there a format you had in mind? I'm thinking it would largely be the pieces that are needed to plug into a unit file. vb pgpJaK8ZPHY5k.pgp Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Appc support in systemd-importd
On 11/03/15 15:24 +0100, Iago L?pez Galeiras wrote: Hi, We're looking into adding appc[1] support in systemd-importd. An appc image (ACI) is just a tar with a rootfs directory and a json manifest. We would have to implement image discovery and simply extract the rootfs. Some questions: * I see that the dkr implementation ignores the information in the docker manifest (resource restrictions, binary to exec...). Is that by design or just not implemented yet? Should we do the same with appc (*only* extract the rootfs)? * Alban liked the idea of saving the manifest so we can extract the whole ACI and modify nspawn to detect a container is an ACI and set /var/lib/machines/appc-image.aci/rootfs as the container's root. The benefit of keeping the manifest would be knowing which binary to start. Is that acceptable? [1]: https://github.com/appc/spec/blob/master/SPEC.md Is there a place that you are working on this logic? I'm will participate. My simple stub is https://github.com/vbatts/systemd/commit/6067a9f484af4cfeb9c4b01965edcd50ff0cffc3 rebased on master. vb pgpkSlGy3g8ke.pgp Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Appc support in systemd-importd
On 11/03/15 16:33 +0100, Lennart Poettering wrote: On Wed, 11.03.15 15:24, Iago López Galeiras (i...@endocode.com) wrote: (added Vincent Batts to CC) Hi, We're looking into adding appc[1] support in systemd-importd. An appc image (ACI) is just a tar with a rootfs directory and a json manifest. We would have to implement image discovery and simply extract the rootfs. Some questions: A great! I remember Vincent Batts was interested in that too. Vincent, do you have any code already? What began as a noble effort has so far been sidelined by continuing work on the trust story of distributing these container images. Looking at my branch, it's not more than stubs. I was last reading through the other import/pull logic in use. vb pgpV9GkrRxxaq.pgp Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] [PATCH] .gitignore: add systemd-pull
--- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index ab6d9d1..e8a4085 100644 --- a/.gitignore +++ b/.gitignore @@ -103,6 +103,7 @@ /systemd-notify /systemd-nspawn /systemd-path +/systemd-pull /systemd-quotacheck /systemd-random-seed /systemd-rc-local-generator -- 1.9.3 pgpfSyc5R3OK3.pgp Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] Docker vs PrivateTmp
On 19/01/15 08:39 -0500, Daniel J Walsh wrote: On 01/19/2015 12:27 AM, Lars Kellogg-Stedman wrote: On Sun, Jan 18, 2015 at 11:38:12PM -0500, Lars Kellogg-Stedman wrote: I think we actually want MountFlags=slave, which will permit mounts from the global namespace to propagate into the service namespace without permitting propagation in the other direction. It seems like this would the Least Surprising behavior. ...which would be the default if docker.service were itself using PrivateTmp=true, because from systemd.exec: Note that the file system namespace related options (PrivateTmp=, PrivateDevices=, ProtectSystem=, ProtectHome=, ReadOnlyDirectories=, InaccessibleDirectories= and ReadWriteDirectories=) require that mount and unmount propagation from the unit's file system namespace is disabled, and hence downgrade shared to slave. So either explicitly setting MountFlags=slave, or setting PrivateTmp=true if that doesn't cause any issues of which I am not aware. Vincent what do you think about MountFlags=slave? 'slave' sounds like the correct subtree mount. We were targeting 'MountFlags' to make use of unsharing the mount namespace. vb pgpbxSgwQKy9E.pgp Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] statically linking systemd-socket-proxyd
On 28/02/14 00:24 +0100, Lennart Poettering wrote: On Thu, 27.02.14 15:35, Vincent Batts (vba...@redhat.com) wrote: I see that despite --enable-static listed as a ./configure flag, it is not supported. Can we get a Makefile target for compiling the systemd-socket-proxyd as statically linked? For it to be portable for any guest container. As Cristian said, static linking is really a false holy grail... That said, the tool has exactly zero deps besides glibc. (where glibc in this definition consists of libpthread, librt, libc). Hence, when it comes to portability it gets pretty much as close to portability to various systems as it can get. Linux systems might vary greatly in the libraries they offer or have installed, but a glibc they all should have I'm completely familiar with the reasons to not statically link, like drepper's page lists. Though it does not address portability, except for dependency on _other_ tools which are dynamic. That is not the case here. Bind mount in the proxyd to the root of a container, then non-socket-activation code is transparently usable. No other tools called. Simple enough. glibc-static is all that is currently needed, and even if a libresolv link is added. Take care, vb pgprxqKkWCNzw.pgp Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] statically linking systemd-socket-proxyd
I see that despite --enable-static listed as a ./configure flag, it is not supported. Can we get a Makefile target for compiling the systemd-socket-proxyd as statically linked? For it to be portable for any guest container. Take care, vb pgpdy2yt5pm0C.pgp Description: PGP signature ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
[systemd-devel] [PATCH] nspawn: require /etc/os-release only for init
/etc/os-release is expected for the case for booting a full system, and
need not be required for thin container execution.
---
src/nspawn/nspawn.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/src/nspawn/nspawn.c b/src/nspawn/nspawn.c
index f904ebe..cf03a40 100644
--- a/src/nspawn/nspawn.c
+++ b/src/nspawn/nspawn.c
@@ -1200,7 +1200,7 @@ int main(int argc, char *argv[]) {
goto finish;
}
-if (path_is_os_tree(arg_directory) = 0) {
+if (arg_boot path_is_os_tree(arg_directory) = 0) {
log_error(Directory %s doesn't look like an OS root directory
(/etc/os-release is missing). Refusing., arg_directory);
goto finish;
}
--
1.8.4
pgpHaLyAHfSFC.pgp
Description: PGP signature
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/systemd-devel
