Bug#804624: please improve support for installing foreign packages to chroots and add DPKG_ROOT

[Guillem Jover]

Hi!

On Wed, 2016-03-30 at 08:48:45 +0200, Helmut Grohne wrote:
> On Wed, Mar 30, 2016 at 01:29:15AM +0200, Guillem Jover wrote:
> > > b) Packages that do not "set -u" (nounset), can now prepend $DPKG_ROOT
> > >to any file they operate on. With old versions $DPKG_ROOT will be
> > >unset and with change a) $DPKG_ROOT will be empty. Thus this change
> > >is backwards-compatible.
> > 
> > Right, and these can always be set like ": ${DPKG_ROOT:=}". We could
> > even recommend this as part of some doc/howto/spec for the initial
> > deployments, before packages can assume a recent enough dpkg.
> 
> I like the recommendation, but I note that only about 30 packages employ
> "set -u" (i.e. about 0.1%).

I've added this for now to the wiki spec.

  

> > > c) dpkg should gain a new force flag. I call it --force-remote-configure
> > >for now. It is supposed to force dpkg into running maintainer scripts
> > >without chroot even when the package in question did not declare that
> > >its maintainer scripts support this mode of operation. Note that we
> > >currently have no way to express whether a package supports running
> > >maintainer scripts without chroot. The flag is being added by
> > >0002-add-force-remote-scripts.patch and the behavior is implemented
> > >by 0003-inhibit-chroot-when-force-remote-scripts.patch. Packages can
> > >only reasonably support this mode after implementing b).
> > 
> > I don't quite like the name, as remote to me implies on some other
> > machine. Perhaps foreign, host or extern(al), although some of these
> > are a bit overloaded terms already.
> 
> Of course, I am not attached to the name. I just needed some string that
> remotely made sense. What about "chrootless-configure" to make it
> crystal clear? It should go hand in hand with d) if possible. What is
> your preference?

Hmm, that's always difficult. :) "configure" probably not, as this
involves several maintainer scripts not all of them being configure
related. "chrootless" while quite clear is a bit long, but if there's
nothing better then that does. Things that come to mind perhaps which
are also pretty long, but other proposals welcome:

  maintscript-chrootless
  maintscript-jailbreak
  maintscript-detached

Another option could be to add:

  maintscript-chroot

and make that the default. The problem I can see with this is that it
feels like it is promoting a bit the chrootless case as perhaps safer
or similar.

> > Also I'm not sure this makes sense as a force option (instead of its own
> > proper option), as it is a behavior change that we want to always be safe
> > to use, in contrast to a force option that just forces the behavior.
> > Having a force option that would not chroot some times is a bit
> > strange. We might still want to have both though, and there was a
> > related bug report for that (#614126).
> 
> For the same reasons that you bring forward against making this a force
> option, I think it should be a force option: No package in the archive
> is currently known to be safe to configure from outside the chroot. So
> doing that is inherently unsafe. We generally use force options to do
> unsafe things. I see this force option as a development tool and not as
> switch being used regularly. I think it is similar to --force-depends:
> Proceed even though the package declared that it doesn't support the
> mode of operation. The declaration is the absence of a support field as
> in d). Does this make sense to you?

Yes, but as I mention not as the primary user-facing interface.

> Possibly another switch is needed to enable this feature at all? I
> thought we could just make it default (for supported packages), but
> maybe that has downsides as well.

I don't think this behavior should be made the default, it'd be very
unexpected and can cause issues if the external environment is not
"compliant".

> > > d) Once a) is accepted and b) starts getting implemented, we need to
> > >think about a way for packages to tell that they support "remote
> > >scripts". One way to do so would be to add a header "Remote-Scripts:
> > >yes" to the binary package stanza. Packages thus marked would be
> > >required to honour DPKG_ROOT in all maintainer scripts. This flag
> > >makes no provisions yet on what programs can be assumed to be
> > >installed outside the chroot that is operated on.
> > 
> > Yes, ideally only packages marked as such would get a chroot-less
> > environmemt when requested with the new option, because expecting the
> > user to know when a package supports this mode and on what specific
> > version is a terrible interface IMO, as it requires the user to
> > analyze the .debs before unpacking them.
> 
> This also seems in support of the force option to override.
> 
> > The other thing, that ISTR you brought up on IRC are triggers. I'm not
> > sure how we'd handle those either. :/
> 
> I 

Bug#804624: please improve support for installing foreign packages to chroots and add DPKG_ROOT

[Helmut Grohne]

Hi Guillem,

On Wed, Mar 30, 2016 at 01:29:15AM +0200, Guillem Jover wrote:
> I don't quite like the name, as remote to me implies on some other
> machine. Perhaps foreign, host or extern(al), although some of these
> are a bit overloaded terms already.

Given that I didn't get any further feedback on the name from you, I
left it as is, but I still don't mind you changing it.

> Thanks for the patches! I think 2 and 3 need to be merged together,
> but see my comments on the name and if this deserves to be a force
> option or something else. And they need man page updates.

I am attaching updated patches that address all of these points except
for the renaming.

Helmut
>From 1d6cfda8e2a7ab5dfac3ea32b3ed6d957677fdda Mon Sep 17 00:00:00 2001
From: Helmut Grohne 
Date: Mon, 9 Nov 2015 22:07:52 +0100
Subject: [PATCH 1/2] export a variable DPKG_ROOT

This variable holds the value of instdir. It is supposed to be used in
maintainer scripts. It should be prepended to all paths that are
operated on. Currently, dpkg chroots to the instdir before invoking
maintainer scripts, so when it does that DPKG_ROOT is set to the empty
string. Thus currently, DPKG_ROOT is always empty.
---
 man/dpkg.1   | 10 ++
 src/main.c   |  2 ++
 src/script.c |  2 ++
 3 files changed, 14 insertions(+)

diff --git a/man/dpkg.1 b/man/dpkg.1
index b47c848..86b5160 100644
--- a/man/dpkg.1
+++ b/man/dpkg.1
@@ -866,6 +866,16 @@ Defined by \fBdpkg\fP on the maintainer script environment to a value
 (\(oq\fB0\fP\(cq or \(oq\fB1\fP\(cq) noting whether debugging has been
 requested (with the \fB\-\-debug\fP option) for the maintainer scripts
 (since dpkg 1.18.4).
+.TP
+.B DPKG_ROOT
+Defined by \fBdpkg\fP on the maintainer script environment to indicate
+which installation to act on. The value is supposed to be prepended to
+any path maintainer scripts operate on. During normal operation, this
+variable is empy. When installing packages into a chroot, dpkg
+normally invokes maintainer scripts using chroot and leaves this
+variable empty. Only when installing into a chroot and the package
+being installed declares support for using this variabe, the chroot
+call is skipped and this variable is non-empty (since dpkg 1.18.5).
 .
 .SH FILES
 .TP
diff --git a/src/main.c b/src/main.c
index 0ff04f1..6440e79 100644
--- a/src/main.c
+++ b/src/main.c
@@ -879,6 +879,8 @@ int main(int argc, const char *const *argv) {
   /* Always set environment, to avoid possible security risks. */
   if (setenv("DPKG_ADMINDIR", admindir, 1) < 0)
 ohshite(_("unable to setenv for subprocesses"));
+  if (setenv("DPKG_ROOT", instdir, 1) < 0)
+ohshite(_("unable to setenv for subprocesses"));
 
   if (!f_triggers)
 f_triggers = (cipaction->arg_int == act_triggers && *argv) ? -1 : 1;
diff --git a/src/script.c b/src/script.c
index d9514da..6bb121e 100644
--- a/src/script.c
+++ b/src/script.c
@@ -105,6 +105,8 @@ maintscript_pre_exec(struct command *cmd)
 			ohshit(_("admindir must be inside instdir for dpkg to work properly"));
 		if (setenv("DPKG_ADMINDIR", admindir + instdirl, 1) < 0)
 			ohshite(_("unable to setenv for subprocesses"));
+		if (setenv("DPKG_ROOT", "", 1) < 0)
+			ohshite(_("unable to setenv for subprocesses"));
 
 		if (chroot(instdir))
 			ohshite(_("failed to chroot to '%.250s'"), instdir);
-- 
2.8.0.rc3

>From 12ab016704b2b8e775f012c301587b49fe2e0701 Mon Sep 17 00:00:00 2001
From: Helmut Grohne 
Date: Mon, 9 Nov 2015 22:16:10 +0100
Subject: [PATCH 2/2] add --force-remote-scripts

Currently, dpkg chroots to the instdir before invoking maintainer
scripts. The new force flag is supposed to inhibit the chroot call. The
user is supposed to know that the packages he is operating on do support
this new mode of operation. Thus the force flag is marked as dangerous.
---
 man/dpkg.1   | 4 
 src/main.c   | 3 +++
 src/main.h   | 1 +
 src/script.c | 8 
 4 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/man/dpkg.1 b/man/dpkg.1
index 86b5160..1e8bc8e 100644
--- a/man/dpkg.1
+++ b/man/dpkg.1
@@ -611,6 +611,10 @@ Try to (de)install things even when not root.
 \fBbad\-verify\fP:
 Install a package even if it fails authenticity check.
 
+\fBremote\-scripts\fP:
+Run maintainer scrips without chroot despite the package not declaring
+support for this mode of operation (since dpkg 1.18.5).
+
 .TP
 \fB\-\-ignore\-depends\fP=\fIpackage\fP,...
 Ignore dependency-checking for specified packages (actually, checking is
diff --git a/src/main.c b/src/main.c
index 6440e79..025b48b 100644
--- a/src/main.c
+++ b/src/main.c
@@ -196,6 +196,7 @@ int fc_conff_ask = 0;
 int fc_unsafe_io = 0;
 int fc_badverify = 0;
 int fc_badversion = 0;
+int fc_remote_scripts = 0;
 
 int errabort = 50;
 static const char *admindir = ADMINDIR;
@@ -275,6 +276,8 @@ static const struct forceinfo {
 '!', N_("Remove packages which require installation") },
   { "remove-essential",_removeessential,
 '!', N_("Remove an essential 

Bug#804624: please improve support for installing foreign packages to chroots and add DPKG_ROOT

[Helmut Grohne]

Hi Guillem,

Thanks for taking the time to process this crazy idea!

On Wed, Mar 30, 2016 at 01:29:15AM +0200, Guillem Jover wrote:
> > b) Packages that do not "set -u" (nounset), can now prepend $DPKG_ROOT
> >to any file they operate on. With old versions $DPKG_ROOT will be
> >unset and with change a) $DPKG_ROOT will be empty. Thus this change
> >is backwards-compatible.
> 
> Right, and these can always be set like ": ${DPKG_ROOT:=}". We could
> even recommend this as part of some doc/howto/spec for the initial
> deployments, before packages can assume a recent enough dpkg.

I like the recommendation, but I note that only about 30 packages employ
"set -u" (i.e. about 0.1%).

> > c) dpkg should gain a new force flag. I call it --force-remote-configure
> >for now. It is supposed to force dpkg into running maintainer scripts
> >without chroot even when the package in question did not declare that
> >its maintainer scripts support this mode of operation. Note that we
> >currently have no way to express whether a package supports running
> >maintainer scripts without chroot. The flag is being added by
> >0002-add-force-remote-scripts.patch and the behavior is implemented
> >by 0003-inhibit-chroot-when-force-remote-scripts.patch. Packages can
> >only reasonably support this mode after implementing b).
> 
> I don't quite like the name, as remote to me implies on some other
> machine. Perhaps foreign, host or extern(al), although some of these
> are a bit overloaded terms already.

Of course, I am not attached to the name. I just needed some string that
remotely made sense. What about "chrootless-configure" to make it
crystal clear? It should go hand in hand with d) if possible. What is
your preference?

> Also I'm not sure this makes sense as a force option (instead of its own
> proper option), as it is a behavior change that we want to always be safe
> to use, in contrast to a force option that just forces the behavior.
> Having a force option that would not chroot some times is a bit
> strange. We might still want to have both though, and there was a
> related bug report for that (#614126).

For the same reasons that you bring forward against making this a force
option, I think it should be a force option: No package in the archive
is currently known to be safe to configure from outside the chroot. So
doing that is inherently unsafe. We generally use force options to do
unsafe things. I see this force option as a development tool and not as
switch being used regularly. I think it is similar to --force-depends:
Proceed even though the package declared that it doesn't support the
mode of operation. The declaration is the absence of a support field as
in d). Does this make sense to you?

Possibly another switch is needed to enable this feature at all? I
thought we could just make it default (for supported packages), but
maybe that has downsides as well.

> > d) Once a) is accepted and b) starts getting implemented, we need to
> >think about a way for packages to tell that they support "remote
> >scripts". One way to do so would be to add a header "Remote-Scripts:
> >yes" to the binary package stanza. Packages thus marked would be
> >required to honour DPKG_ROOT in all maintainer scripts. This flag
> >makes no provisions yet on what programs can be assumed to be
> >installed outside the chroot that is operated on.
> 
> Yes, ideally only packages marked as such would get a chroot-less
> environmemt when requested with the new option, because expecting the
> user to know when a package supports this mode and on what specific
> version is a terrible interface IMO, as it requires the user to
> analyze the .debs before unpacking them.

This also seems in support of the force option to override.

> One tiny concern is that if eventually we end up with the whole archive
> (haha? :) supporting this, then the field seems a bit of bloat.

I considered this as well, but thought of it as the lesser evil:
 * I do not think that the whole archive will support DPKG_ROOT in a
   reasonable time frame. The major benefit comes to essential and then
   motivation to process further packages drops.
 * We can still require DPKG_ROOT via policy and drop the field after
   all oldstable packages support it.

> The other thing, that ISTR you brought up on IRC are triggers. I'm not
> sure how we'd handle those either. :/

I thought about this some more and the only sensible approach for
triggers I could come up with is handling them on their own: The
triggered package must declare support for this new mode or its trigger
processing will proceed in the old way (unless forced). The major
downside here is that you cannot tell beforehand whether you can
configure a package without chroot, but I don't see that changing
without breaking the flexibility of triggers.

> > e) Once a) is accepted and b) starts getting implemented, we need to
> >think about what programs 

Bug#804624: please improve support for installing foreign packages to chroots and add DPKG_ROOT

[Guillem Jover]

Hi!

On Tue, 2015-11-10 at 00:01:48 +0100, Helmut Grohne wrote:
> Package: dpkg
> Version: 1.18.3
> Severity: wishlist
> Tags: patch
> User: helm...@debian.org
> Usertags: rebootstrap

> Thank you very much for discussing the idea of DPKG_ROOT and recording
> some results at https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap
> already. By now, I am convinced that this idea is worth exploring and
> have thus prepared a small patch set implementing some of the first
> steps.

Some of the stuff I'll cover below I think I probably covered in our
conversation on IRC at the time, so it might not come as new stuff.

> a) dpkg should export DPKG_ROOT. DPKG_ROOT should be a string that
>should be prepended to "/" to arrive at the current installation
>root (instdir). Notably, when dpkg invokes chroot(), DPKG_ROOT
>becomes empty. At the moment, DPKG_ROOT is always empty, but this
>should not be relied upon. This is
>0001-export-a-variable-DPKG_ROOT.patch.

This part is fine in principle, and that's what I envisioned when
thinking about how to merge the dpkg functional test suite into the
dpkg git repo proper. But there are several reasons I've always
hesitated to add this support even though it would be useful for
dpkg itself.

> b) Packages that do not "set -u" (nounset), can now prepend $DPKG_ROOT
>to any file they operate on. With old versions $DPKG_ROOT will be
>unset and with change a) $DPKG_ROOT will be empty. Thus this change
>is backwards-compatible.

Right, and these can always be set like ": ${DPKG_ROOT:=}". We could
even recommend this as part of some doc/howto/spec for the initial
deployments, before packages can assume a recent enough dpkg.

> c) dpkg should gain a new force flag. I call it --force-remote-configure
>for now. It is supposed to force dpkg into running maintainer scripts
>without chroot even when the package in question did not declare that
>its maintainer scripts support this mode of operation. Note that we
>currently have no way to express whether a package supports running
>maintainer scripts without chroot. The flag is being added by
>0002-add-force-remote-scripts.patch and the behavior is implemented
>by 0003-inhibit-chroot-when-force-remote-scripts.patch. Packages can
>only reasonably support this mode after implementing b).

I don't quite like the name, as remote to me implies on some other
machine. Perhaps foreign, host or extern(al), although some of these
are a bit overloaded terms already.

Also I'm not sure this makes sense as a force option (instead of its own
proper option), as it is a behavior change that we want to always be safe
to use, in contrast to a force option that just forces the behavior.
Having a force option that would not chroot some times is a bit
strange. We might still want to have both though, and there was a
related bug report for that (#614126).

> d) Once a) is accepted and b) starts getting implemented, we need to
>think about a way for packages to tell that they support "remote
>scripts". One way to do so would be to add a header "Remote-Scripts:
>yes" to the binary package stanza. Packages thus marked would be
>required to honour DPKG_ROOT in all maintainer scripts. This flag
>makes no provisions yet on what programs can be assumed to be
>installed outside the chroot that is operated on.

Yes, ideally only packages marked as such would get a chroot-less
environmemt when requested with the new option, because expecting the
user to know when a package supports this mode and on what specific
version is a terrible interface IMO, as it requires the user to
analyze the .debs before unpacking them.

One tiny concern is that if eventually we end up with the whole archive
(haha? :) supporting this, then the field seems a bit of bloat.

The other thing, that ISTR you brought up on IRC are triggers. I'm not
sure how we'd handle those either. :/

> e) Once a) is accepted and b) starts getting implemented, we need to
>think about what programs maintainer scripts can assume to be
>available outside the chroot. Some ways to handle that:
> * Packages may only assume "common unix functionality". Such a set
>   would have to be defined somehow and roughly equates what
>   debootstrap requires.

Defining this in terms of strict POSIX compliance (or a subset of the
utilities defined within) would be the easiest/best I think.

> * Packages may only assume essential packages to be available.

That would resitrict the external system to be a Debian system. Which
might be fine I guess but is more limited than what debootstrap requires.
It has the problem that even then you need to specify essential of what
Debian version?

> * A new set of headers Maint-{Depends,Conflicts,...} is added to
>   request tools to be installed. These new relations would be
>   checked outside the chroot (if any).
> 
>   A full Debian release or two needs to pass 

Bug#804624: please improve support for installing foreign packages to chroots and add DPKG_ROOT

[Helmut Grohne]

Package: dpkg
Version: 1.18.3
Severity: wishlist
Tags: patch
User: helm...@debian.org
Usertags: rebootstrap

Hi Guillem,

Thank you very much for discussing the idea of DPKG_ROOT and recording
some results at https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap
already. By now, I am convinced that this idea is worth exploring and
have thus prepared a small patch set implementing some of the first
steps.

a) dpkg should export DPKG_ROOT. DPKG_ROOT should be a string that
   should be prepended to "/" to arrive at the current installation
   root (instdir). Notably, when dpkg invokes chroot(), DPKG_ROOT
   becomes empty. At the moment, DPKG_ROOT is always empty, but this
   should not be relied upon. This is
   0001-export-a-variable-DPKG_ROOT.patch.

b) Packages that do not "set -u" (nounset), can now prepend $DPKG_ROOT
   to any file they operate on. With old versions $DPKG_ROOT will be
   unset and with change a) $DPKG_ROOT will be empty. Thus this change
   is backwards-compatible.

c) dpkg should gain a new force flag. I call it --force-remote-configure
   for now. It is supposed to force dpkg into running maintainer scripts
   without chroot even when the package in question did not declare that
   its maintainer scripts support this mode of operation. Note that we
   currently have no way to express whether a package supports running
   maintainer scripts without chroot. The flag is being added by
   0002-add-force-remote-scripts.patch and the behavior is implemented
   by 0003-inhibit-chroot-when-force-remote-scripts.patch. Packages can
   only reasonably support this mode after implementing b).

d) Once a) is accepted and b) starts getting implemented, we need to
   think about a way for packages to tell that they support "remote
   scripts". One way to do so would be to add a header "Remote-Scripts:
   yes" to the binary package stanza. Packages thus marked would be
   required to honour DPKG_ROOT in all maintainer scripts. This flag
   makes no provisions yet on what programs can be assumed to be
   installed outside the chroot that is operated on.

e) Once a) is accepted and b) starts getting implemented, we need to
   think about what programs maintainer scripts can assume to be
   available outside the chroot. Some ways to handle that:
* Packages may only assume "common unix functionality". Such a set
  would have to be defined somehow and roughly equates what
  debootstrap requires.
* Packages may only assume essential packages to be available.
* A new set of headers Maint-{Depends,Conflicts,...} is added to
  request tools to be installed. These new relations would be
  checked outside the chroot (if any).

  A full Debian release or two needs to pass before such headers can
  be used in the archive. This also poses the problem that a user
  can remove packages required for removing other packages and thus
  revoking the ability to remove certain packages. It is not clear
  how the absence of Maint-Depends is supposed to be handled. It is
  not clear whether dpkg needs to lock the dpkg database outside the
  chroot.

f) Once a), b) and d) are implemented and some version of e) is agreed
   upon, debootstrap can be changed to prefer configuring packages that
   support "remote scripts" to break dependency cycles.

g) At the same time as f), tools like multistrap can start making use of
   this new functionality.

With the above patches, I verified that I can install a package with a
maintainer script into a chroot that has an empty database (in
particular no essential package is unpacked in the chroot). The
maintainer script is run without chroot and has DPKG_ROOT set up
properly.

bash's preinst will become a problem with this scheme. It is a binary
due to earlier breakage when it was a script, but the "remote scripts"
approach cannot work with binaries, because we cannot know whether the
cpu supports executing such binaries. I don't have a plan for
bash.preinst.

I understand that what I propose herein is a steep change with wide
ranging implications. It needs sincere thought to avoid creating a
situation that is hard to fix up. Yet it seems pretty round to me
already. Despite there being open questions, we can handle at least a)
today.

Helmut
>From b10ba1186394ae880a5c1421ddb07afa1f3c9d20 Mon Sep 17 00:00:00 2001
From: Helmut Grohne 
Date: Mon, 9 Nov 2015 22:07:52 +0100
Subject: [PATCH 1/3] export a variable DPKG_ROOT

This variable holds the value of instdir. It is supposed to be used in
maintainer scripts. It should be prepended to all paths that are
operated on. Currently, dpkg chroots to the instdir before invoking
maintainer scripts, so when it does that DPKG_ROOT is set to the empty
string. Thus currently, DPKG_ROOT is always empty.
---
 src/main.c   | 2 ++
 src/script.c | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/src/main.c b/src/main.c
index f16dc0a..4ac701c 100644
--- a/src/main.c
+++