[gentoo-dev] Last rites: dev-python/django-cms and related
# Virgil Dupras (14 Nov 2018) # Unmaintained and depending on an purged version (1.8) of Django. # Removal in 30 days. bug #664012 dev-python/django-cms dev-python/djangocms-admin-style dev-python/djangocms-file dev-python/djangocms-flash dev-python/djangocms-inherit dev-python/djangocms-link dev-python/djangocms-picture dev-python/djangocms-snippet dev-python/djangocms-teaser dev-python/djangocms-video dev-python/django-classy-tags dev-python/django-formtools dev-python/djangocms-text-ckeditor dev-python/django-sekizai dev-python/django-treebeard dev-python/aldryn-search dev-python/aldryn-bootstrap3 dev-python/django-filer dev-python/django-mptt pgpvX1IwCYI3a.pgp Description: PGP signature
Re: [RFC] gpkg format proposal v2 (was: Re: [gentoo-portage-dev] [RFC] Improving Gentoo package format)
On Sun, 2018-11-11 at 21:53 +0100, Michał Górny wrote:
> Hi,
>
> Ok, here's the second version integrating the feedback received.
> The format is much simpler, based on nested tarballs inspired by Debian.
>
> The outer tarball is uncompressed and uses '.gpkg.tar' suffix. It
> contains (preferably in order but PM should also handle packages with
> mismatched order):
>
> 1. Optional (but recommended) "gpkg: ${PF}" package label that can be
> used to quickly distinguish Gentoo binpkgs from regular tarballs
> (for file(1)).
>
> 2. "metadata.tar${comp}" tarball containing binary package metadata
> as files.
>
> 3. Optional "metadata.tar${comp}.sig" containing detached signature
> for the metadata archive.
>
> 4. "contents.tar${comp}" tarball containing files to be installed.
>
> 5. Optional "contents.tar${comp}.sig" containing detached signature for
> the contents archive.
>
> Notes:
>
> a. ${comp} can be any compression format supported by binary packages.
> Technically, metadata and content archives may use different
> compression. Either or both may be uncompressed as well.
>
> b. While signatures are optional, the PM should have a switch
> controlling whether to expect them, and fail hard if they're not present
> when expected.
>
>
> Advantages
> --
> Guaranteed:
>
> + The binary package is still one file, so can be fetched easily.
>
> + File format is trivial and can be extracted using tar(1) + compressor.
>
> + The metadata and contents are compressed independently, and so can be
> easily extracted or modified independently.
>
> + The package format provides for separate metadata and content
> signatures, so they can be verified independently.
>
> + Metadata can be compressed now.
>
> Achieved by regular archives (but might be broken if modified by user):
>
> + Easy recognition by magic(1).
>
> + The metadata archive (and its signature) is packed first, so it may be
> read without fetching the whole binpkg.
>
>
> Why not .ar format?
> ---
> The use of .ar format has been proposed, akin to Debian. While
> the option is mostly feasible, and the simplicity of .ar format would
> reduce the outer size of binary packages, I think the format is simply
> too obscure. It lives mostly as static library format, and the tooling
> for it is part of binutils. LSB considers it deprecated. While I don't
> see it going away anytime soon, I'd rather not rely on it in order to
> save a few KiB.
>
>
> Is there anything left to address?
Here's a quick & dirty xpak2gpkg converter:
https://gist.github.com/mgorny/cca78fb93f14aad11f43abe352caad06
It can be used to try out the format practically and flesh out
the details before we go for a formal spec.
--
Best regards,
Michał Górny
signature.asc
Description: This is a digitally signed message part
[gentoo-portage-dev] [PATCH 2/3] Support FEATURES=pid-sandbox
Supporting using PID namespace in order to isolate the ebuild processes
from host system, and make it possible to kill them all easily
(similarly to cgroups but easier to use).
Bug: https://bugs.gentoo.org/659582
Signed-off-by: Michał Górny
---
lib/portage/const.py | 1 +
lib/portage/package/ebuild/doebuild.py | 8 +++--
lib/portage/process.py | 43 +++---
man/make.conf.5| 7 +
4 files changed, 52 insertions(+), 7 deletions(-)
diff --git a/lib/portage/const.py b/lib/portage/const.py
index e0f93f7cc..ca66bc46e 100644
--- a/lib/portage/const.py
+++ b/lib/portage/const.py
@@ -174,6 +174,7 @@ SUPPORTED_FEATURES = frozenset([
"notitles",
"parallel-fetch",
"parallel-install",
+ "pid-sandbox",
"prelink-checksums",
"preserve-libs",
"protect-owned",
diff --git a/lib/portage/package/ebuild/doebuild.py
b/lib/portage/package/ebuild/doebuild.py
index e84a618d2..9917ac82c 100644
--- a/lib/portage/package/ebuild/doebuild.py
+++ b/lib/portage/package/ebuild/doebuild.py
@@ -1,4 +1,4 @@
-# Copyright 2010-2018 Gentoo Foundation
+# Copyright 2010-2018 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
from __future__ import unicode_literals
@@ -152,6 +152,7 @@ def _doebuild_spawn(phase, settings, actionmap=None,
**kwargs):
kwargs['networked'] = 'network-sandbox' not in settings.features or \
phase in _networked_phases or \
'network-sandbox' in settings['PORTAGE_RESTRICT'].split()
+ kwargs['pidns'] = 'pid-sandbox' in settings.features
if phase == 'depend':
kwargs['droppriv'] = 'userpriv' in settings.features
@@ -1482,7 +1483,7 @@ def _validate_deps(mysettings, myroot, mydo, mydbapi):
# XXX Issue: cannot block execution. Deadlock condition.
def spawn(mystring, mysettings, debug=False, free=False, droppriv=False,
sesandbox=False, fakeroot=False, networked=True, ipc=True,
- mountns=False, **keywords):
+ mountns=False, pidns=False, **keywords):
"""
Spawn a subprocess with extra portage-specific options.
Optiosn include:
@@ -1518,6 +1519,8 @@ def spawn(mystring, mysettings, debug=False, free=False,
droppriv=False,
@type ipc: Boolean
@param mountns: Run this command inside mount namespace
@type mountns: Boolean
+ @param pidns: Run this command in isolated PID namespace
+ @type pidns: Boolean
@param keywords: Extra options encoded as a dict, to be passed to spawn
@type keywords: Dictionary
@rtype: Integer
@@ -1551,6 +1554,7 @@ def spawn(mystring, mysettings, debug=False, free=False,
droppriv=False,
keywords['unshare_net'] = not networked
keywords['unshare_ipc'] = not ipc
keywords['unshare_mount'] = mountns
+ keywords['unshare_pid'] = pidns
if not networked and mysettings.get("EBUILD_PHASE") !=
"nofetch" and \
("network-sandbox-proxy" in features or "distcc" in
features):
diff --git a/lib/portage/process.py b/lib/portage/process.py
index e2ad89b43..beeb09529 100644
--- a/lib/portage/process.py
+++ b/lib/portage/process.py
@@ -223,7 +223,8 @@ def spawn(mycommand, env={}, opt_name=None, fd_pipes=None,
returnpid=False,
uid=None, gid=None, groups=None, umask=None, logfile=None,
path_lookup=True, pre_exec=None,
close_fds=(sys.version_info < (3, 4)), unshare_net=False,
- unshare_ipc=False, unshare_mount=False, cgroup=None):
+ unshare_ipc=False, unshare_mount=False, unshare_pid=False,
+ cgroup=None):
"""
Spawns a given command.
@@ -264,6 +265,8 @@ def spawn(mycommand, env={}, opt_name=None, fd_pipes=None,
returnpid=False,
@param unshare_mount: If True, mount namespace will be unshared and
mounts will
be private to the namespace
@type unshare_mount: Boolean
+ @param unshare_pid: If True, PID ns will be unshared from the spawned
process
+ @type unshare_pid: Boolean
@param cgroup: CGroup path to bind the process to
@type cgroup: String
@@ -332,7 +335,7 @@ def spawn(mycommand, env={}, opt_name=None, fd_pipes=None,
returnpid=False,
# This caches the libc library lookup in the current
# process, so that it's only done once rather than
# for each child process.
- if unshare_net or unshare_ipc or unshare_mount:
+ if unshare_net or unshare_ipc or unshare_mount or unshare_pid:
find_library("c")
# Force instantiation of portage.data.userpriv_groups before the
@@ -348,7 +351,8 @@ def spawn(mycommand, env={}, opt_name=None, fd_pipes=None,
returnpid=False,
try:
_exec(binary, mycommand, opt_name, fd_pipes,
[gentoo-portage-dev] [PATCH 3/3] Introduce a tiny init replacement for inside pid namespace
Signed-off-by: Michał Górny
---
bin/pid-ns-init| 25 +
lib/portage/process.py | 11 ++-
2 files changed, 31 insertions(+), 5 deletions(-)
create mode 100644 bin/pid-ns-init
diff --git a/bin/pid-ns-init b/bin/pid-ns-init
new file mode 100644
index 0..90660571a
--- /dev/null
+++ b/bin/pid-ns-init
@@ -0,0 +1,25 @@
+#!/usr/bin/env python
+# Copyright 2018 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+import os
+import sys
+
+
+def main(argv):
+ if len(argv) < 2:
+ return 'Usage: {} '.format(argv[0])
+ main_child_pid = int(argv[1])
+
+ # wait for child processes
+ while True:
+ pid, status = os.wait()
+ if pid == main_child_pid:
+ return os.WEXITSTATUS(status)
+
+ # this should never be reached
+ return 127
+
+
+if __name__ == '__main__':
+ sys.exit(main(sys.argv))
diff --git a/lib/portage/process.py b/lib/portage/process.py
index beeb09529..4b0d8f7bc 100644
--- a/lib/portage/process.py
+++ b/lib/portage/process.py
@@ -545,13 +545,14 @@ def _exec(binary, mycommand, opt_name, fd_pipes, env,
gid, groups, uid, umask,
else:
if unshare_pid:
# pid namespace
requires us to become init
- # TODO: do init-ty stuff
- # therefore, fork() ASAP
fork_ret = os.fork()
if fork_ret != 0:
- pid, status =
os.waitpid(fork_ret, 0)
- assert pid ==
fork_ret
- os._exit(status)
+
os.execv(portage._python_interpreter, [
+
portage._python_interpreter,
+
os.path.join(portage._bin_path,
+
'pid-ns-init'),
+ '%s' %
fork_ret,
+ ])
if unshare_mount:
# mark the whole
filesystem as private to avoid
# mounts escaping the
namespace
--
2.19.1
[gentoo-portage-dev] [PATCH 1/3] Add FEATURES=mount-sandbox to take advantage of mount ns
Support FEATURES=mount-sandbox that unshares the ebuild processes
into a new mount namespace and makes all the mounts private by default.
Signed-off-by: Michał Górny
---
lib/portage/const.py | 1 +
lib/portage/package/ebuild/doebuild.py | 7 +-
lib/portage/process.py | 34 +-
man/make.conf.5| 5
4 files changed, 40 insertions(+), 7 deletions(-)
diff --git a/lib/portage/const.py b/lib/portage/const.py
index 602caeb34..e0f93f7cc 100644
--- a/lib/portage/const.py
+++ b/lib/portage/const.py
@@ -160,6 +160,7 @@ SUPPORTED_FEATURES = frozenset([
"merge-sync",
"metadata-transfer",
"mirror",
+ "mount-sandbox",
"multilib-strict",
"network-sandbox",
"network-sandbox-proxy",
diff --git a/lib/portage/package/ebuild/doebuild.py
b/lib/portage/package/ebuild/doebuild.py
index d0e96f34c..e84a618d2 100644
--- a/lib/portage/package/ebuild/doebuild.py
+++ b/lib/portage/package/ebuild/doebuild.py
@@ -148,6 +148,7 @@ def _doebuild_spawn(phase, settings, actionmap=None,
**kwargs):
kwargs['ipc'] = 'ipc-sandbox' not in settings.features or \
phase in _ipc_phases
+ kwargs['mountns'] = 'mount-sandbox' in settings.features
kwargs['networked'] = 'network-sandbox' not in settings.features or \
phase in _networked_phases or \
'network-sandbox' in settings['PORTAGE_RESTRICT'].split()
@@ -1480,7 +1481,8 @@ def _validate_deps(mysettings, myroot, mydo, mydbapi):
# XXX This would be to replace getstatusoutput completely.
# XXX Issue: cannot block execution. Deadlock condition.
def spawn(mystring, mysettings, debug=False, free=False, droppriv=False,
- sesandbox=False, fakeroot=False, networked=True, ipc=True, **keywords):
+ sesandbox=False, fakeroot=False, networked=True, ipc=True,
+ mountns=False, **keywords):
"""
Spawn a subprocess with extra portage-specific options.
Optiosn include:
@@ -1514,6 +1516,8 @@ def spawn(mystring, mysettings, debug=False, free=False,
droppriv=False,
@type networked: Boolean
@param ipc: Run this command with host IPC access enabled
@type ipc: Boolean
+ @param mountns: Run this command inside mount namespace
+ @type mountns: Boolean
@param keywords: Extra options encoded as a dict, to be passed to spawn
@type keywords: Dictionary
@rtype: Integer
@@ -1546,6 +1550,7 @@ def spawn(mystring, mysettings, debug=False, free=False,
droppriv=False,
if uid == 0 and platform.system() == 'Linux':
keywords['unshare_net'] = not networked
keywords['unshare_ipc'] = not ipc
+ keywords['unshare_mount'] = mountns
if not networked and mysettings.get("EBUILD_PHASE") !=
"nofetch" and \
("network-sandbox-proxy" in features or "distcc" in
features):
diff --git a/lib/portage/process.py b/lib/portage/process.py
index fd326731a..e2ad89b43 100644
--- a/lib/portage/process.py
+++ b/lib/portage/process.py
@@ -1,5 +1,5 @@
# portage.py -- core Portage functionality
-# Copyright 1998-2014 Gentoo Foundation
+# Copyright 1998-2018 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
@@ -10,6 +10,7 @@ import platform
import signal
import socket
import struct
+import subprocess
import sys
import traceback
import os as _os
@@ -222,7 +223,7 @@ def spawn(mycommand, env={}, opt_name=None, fd_pipes=None,
returnpid=False,
uid=None, gid=None, groups=None, umask=None, logfile=None,
path_lookup=True, pre_exec=None,
close_fds=(sys.version_info < (3, 4)), unshare_net=False,
- unshare_ipc=False, cgroup=None):
+ unshare_ipc=False, unshare_mount=False, cgroup=None):
"""
Spawns a given command.
@@ -260,6 +261,9 @@ def spawn(mycommand, env={}, opt_name=None, fd_pipes=None,
returnpid=False,
@type unshare_net: Boolean
@param unshare_ipc: If True, IPC will be unshared from the spawned
process
@type unshare_ipc: Boolean
+ @param unshare_mount: If True, mount namespace will be unshared and
mounts will
+ be private to the namespace
+ @type unshare_mount: Boolean
@param cgroup: CGroup path to bind the process to
@type cgroup: String
@@ -328,7 +332,7 @@ def spawn(mycommand, env={}, opt_name=None, fd_pipes=None,
returnpid=False,
# This caches the libc library lookup in the current
# process, so that it's only done once rather than
# for each child process.
- if unshare_net or unshare_ipc:
+ if unshare_net or unshare_ipc or unshare_mount:
find_library("c")
# Force instantiation of portage.data.userpriv_groups before the
@@ -344,7 +348,7 @@ def spawn(mycommand, env={}, opt_name=None, fd_pipes=None,
