Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Sat, Jan 06, 2018 at 02:25:55AM +0100, Manuel A. Fernandez Montecelo wrote:
> 2018-01-02 03:10 Theodore Ts'o:
> > My only real concern is whether this might complicate building the
> > latest version of e2fsprogs for stable and old-stable for
> > debian-backports.
> 
> I think that it's fine, they've been supported for a long time and
> stable and old-stable should be covered, not sure about old-old-stable
> without backports.

It's mostly fine.  The one problem I found was that I tried to add the
*-dbg packages to the control file such that they would only be built
if the build profile pkg.e2fsprogs.legacy-dbg was active.
Unfortunately, when I tried doing a source-only upload of e2fsprogs,
the DAK software complained that "e2fsprogs-dbg" et.al. were new
packages that weren't supported with a source-only upload.

That's apparently because it was parsing the control file, found the
package declaration, and assumed that build profiles will only
suppress packages (e.g., such as fuse2fs or documentation pckages).
It didn't assume that a non-standard build profile such as
pkg.e2fsprogs.legacy-dbg would *enable* a package, and that under
normal builds, the *-dbg packages wouldn't be built.  Which is fair
enough, there's no way DAK could determine that without actually
building the source-only upload.

So I had to move the *-dbg package definitions into a
debian/control.legacy-dbg file, and mutate debian/control in order to
both support jessie backports, and keep DAK happy.

Which is sad, but it's a solution which works --- and I understand why
there really isn't any other way for DAK to handle this case.

Cheers,

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Manuel A. Fernandez Montecelo]

2018-01-02 03:10 Theodore Ts'o:

My only real concern is whether this might complicate building the
latest version of e2fsprogs for stable and old-stable for
debian-backports.


I think that it's fine, they've been supported for a long time and
stable and old-stable should be covered, not sure about old-old-stable
without backports.


2018-01-03 23:59 Theodore Ts'o:


Would you accept patches to achieve this in e2fsprogs?  It would
probably be quite clean, not complicating/obfuscating the packaging
files too much, usually only 2~10 lines (but I didn't look specifically
into this package yet).


With some help from Simon McVittie, you should be able to use the
build profile pkg.e2fsprogs.no-fuse2fs in the just-uploaded 1.43.8-2
version of e2fsprogs.  It seems to work for me; please let me know if
does what you need.


Not immediately but I will do soon, and possibly send fixes or more
profiles if they're convenient for bootstrapping :)

Thanks, and thanks to Simon too for beating me to it!

--
Manuel A. Fernandez Montecelo 

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Tue, Jan 02, 2018 at 12:38:55AM +0100, Manuel A. Fernandez Montecelo wrote:
> 
> Lately architectures tend to use automatic bootstrapping at least for
> some of the initial dependencies.  Adding support for build profiles
> (would be something like pkg.e2fsprogs.nofuse in this case) can help to
> build just by using env variables when invoking dpkg-buildpackage or
> other build tools.
> 
> Would you accept patches to achieve this in e2fsprogs?  It would
> probably be quite clean, not complicating/obfuscating the packaging
> files too much, usually only 2~10 lines (but I didn't look specifically
> into this package yet).

With some help from Simon McVittie, you should be able to use the
build profile pkg.e2fsprogs.no-fuse2fs in the just-uploaded 1.43.8-2
version of e2fsprogs.  It seems to work for me; please let me know if
does what you need.

Cheers,

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Wed, Jan 03, 2018 at 08:16:35PM +, Simon McVittie wrote:
> > Has there been any thought about having the build
> > profiles framework support for having the rules file autoselect a
> > build profile based on the build environment?
> 
> I suspect that might be a "considered and rejected" sort of thing,
> because toolchain maintainers want the same command to always do more
> or less the same thing. If you want some automation for enabling special
> build profiles, I'd suggest wrapping it around the outside instead.
> That also means it's allowed to edit debian/control if it needs to.

Actually, after doing some experimentation, I was able to make this
work.  From the debian/rules file:

USE_DBGSYM ?= $(shell if dpkg --compare-versions $(DH_VERSION) ">=" 9.20160114 
; then echo yes ; fi)

ifeq ($(USE_DBGSYM),yes)
dh_strip_args = -p$(1) --dbgsym-migration='$(1)-dbg (<= 1.43-1)'
dh_strip_args2 = -p$(1) --dbgsym-migration='$(2)-dbg (<= 1.43-1)'
else
dh_strip_args = -p$(1) --dbg-package=$(1)-dbg
dh_strip_args2 = -p$(1) --dbg-package=$(2)-dbg
DBG_PACKAGES += -pe2fsprogs-dbg -pe2fslibs-dbg -plibcomerr2-dbg -plibss2-dbg
export DEB_BUILD_PROFILES += pkg.e2fsprogs.legacy-dbg
endif

Which is actually cool, because it means you can do an "apt-get source
e2fsprogs", "schroot -c jessie-amd64", and then run "dpkg-buildpackage
-us -uc --changes-option=-S" and have the right thing happen
automagically.

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Simon McVittie]

On Wed, 03 Jan 2018 at 12:40:45 -0500, Theodore Ts'o wrote:
> From the BuildProfile web page and
> looking at the package versions, support for it appears to be in
> Jessie, or at least "preliminary support" is present.

jessie's apt/dpkg at least don't choke on the new syntax (I
tried backporting flatpak from buster, which has , for
jessie-backports-sloppy and it seemed OK), although I didn't actually
try doing a nodoc build for jessie. I'd suggest using jessie-backports'
debhelper (and dh-exec if you use that) if targeting jessie.

(buster's flatpak isn't actually present in jessie-backports-sloppy
because I'm not sure I can commit to having the time or feasibility
to patch it into working on jessie for the entire lifetime of buster,
which seems to be the bar set by the backports maintainers.)

> Since the autodetection isn't there, I would have to just manually
> build with some kind of pkg.e2fsprogs.old-dbg build profile, or some
> such, instead.  I guess it's about the same level of inconvenience of
> needing to run ./debian/rules in order to generate the control file
> from control.in.

Right. If you're doing special package-specific runes anyway, you could
add a `debian/build-backport` that activates the right build profiles
for you instead?

The examples of build profiles in my packages are all "negative"
(Build-Profiles: ), which I think are much more common; but
it's also possible to have "positive" build profiles which enable
extra packages that are not normally built
(Build-Profiles:  or something).

> My "./debian-rules debian-files" scheme used to do a lot more,
> including rewriting several other files in debian/ using m4

dh-exec covers a lot of the uses for that sort of trick, although (like
everything invoked at build time) it can't touch debian/control.

> we had just migrated libuuid and libblkid from e2fsprogs to
> util-linux, and I wanted to support backports to stable, old-stable,
> and Ubuntu LTS.

For the Flatpak family (some of which need non-trivial patching to
support jessie's older GLib) I've been happy with `git merge` into a
separate branch, tbh...

> Has there been any thought about having the build
> profiles framework support for having the rules file autoselect a
> build profile based on the build environment?

I suspect that might be a "considered and rejected" sort of thing,
because toolchain maintainers want the same command to always do more
or less the same thing. If you want some automation for enabling special
build profiles, I'd suggest wrapping it around the outside instead.
That also means it's allowed to edit debian/control if it needs to.

smcv

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Mon, Jan 01, 2018 at 11:43:23PM +, Simon McVittie wrote:
> 
> Perhaps you could convert this into a pkg.e2fsprogs.nofuse build profile?
> 
> This would look something like the attached (untested!) patches.

Thanks, I'll give this a try.  From the BuildProfile web page and
looking at the package versions, support for it appears to be in
Jessie, or at least "preliminary support" is present.  Is that
correct?  Is there any gotchas I should be aware of when backporting
to Jessie?

It looks like one thing which my scheme supports which build profiles
does not is that when backporting to Jessie, I can just check out the
tree from git, and then run:

./debian/rules debian-files
dpkg-buildpackage

... and it will autodetect that the system doesn't support *-dbgsym
packages, and create e2fsprogs-dbg, e2fslibs-dbg, et.al packages for
Jessie instead.

Since the autodetection isn't there, I would have to just manually
build with some kind of pkg.e2fsprogs.old-dbg build profile, or some
such, instead.  I guess it's about the same level of inconvenience of
needing to run ./debian/rules in order to generate the control file
from control.in.

My "./debian-rules debian-files" scheme used to do a lot more,
including rewriting several other files in debian/ using m4, back when
we had just migrated libuuid and libblkid from e2fsprogs to
util-linux, and I wanted to support backports to stable, old-stable,
and Ubuntu LTS.  I did like the fact that it could detect which build
option (now, "build profile") to use automatically, so the folks
building backports for various Debian derivitives didn't need to do
anything special.  Has there been any thought about having the build
profiles framework support for having the rules file autoselect a
build profile based on the build environment?

Cheers,

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Tue, Jan 02, 2018 at 12:38:55AM +0100, Manuel A. Fernandez Montecelo wrote:
> Lately architectures tend to use automatic bootstrapping at least for
> some of the initial dependencies.  Adding support for build profiles
> (would be something like pkg.e2fsprogs.nofuse in this case) can help to
> build just by using env variables when invoking dpkg-buildpackage or
> other build tools.
> 
> Would you accept patches to achieve this in e2fsprogs?  It would
> probably be quite clean, not complicating/obfuscating the packaging
> files too much, usually only 2~10 lines (but I didn't look specifically
> into this package yet).

Sure.  The debian/rules.custom support that I put in predates the
Debian build profiles spec, but it's basically the same idea.  So it
should be pretty simple to migrate things to use the build profile
concept; it's just a matter of wiring up things slightly differently
in debian/rules.

I haven't looked at build profiles spec super-closely yet, but since
it looks like there is core infrastructural support for it in dpkg,
debhelpers, and friends, it's no longer necessary to run
debian/control.in through m4 to generate debian/control (which is how
I did things sans dpkg support).

Send me a patch, either sent to linux-e...@vger.kernel.org, or as
Debian bug, and I'll happily take it.

My only real concern is whether this might complicate building the
latest version of e2fsprogs for stable and old-stable for
debian-backports.

Cheers,

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Simon McVittie]

On Mon, 01 Jan 2018 at 16:51:45 -0500, Theodore Ts'o wrote:
> This probably doesn't help much, but for people who are doing things
> by hand, you can skip the dependency on fuse by unpacking the
> e2fsprogs source packaging, adding the file debian/rules.custom which
> contains the single line, "SKIP_FUSE2FS=yes", and building by hand.

Perhaps you could convert this into a pkg.e2fsprogs.nofuse build profile?

This would look something like the attached (untested!) patches.

The intention seems to be that build profiles are how this is done
automatically or near-automatically in future. dbus and glib2.0 are
examples of packages where I've made use of build profiles to cut off
non-essential functionality for bootstrapping (they're significantly
further up the stack than e2fsprogs, but were involved in a cycle in
the past).

Having nodoc and noudeb build profiles (assuming the package has
documentation and udebs) also makes local test-builds a lot faster for
many packages, which is very convenient when testing functional changes.

> It currently doesn't automatically fix up the control file

You can skip binary packages on a per-build-profile basis without having
to generate debian/control from debian/control.in. After doing that,
it's often convenient to write build profile checks in debian/rules as
a check for "are we building package foo?" rather than "are we using
build profile foo?", like this in dbus:

binaries := $(shell dh_listpackages)

[...]

ifneq ($(filter dbus-udeb,$(binaries)),)
# udeb build
make -C debian/build-udeb install-exec 
DESTDIR=$(CURDIR)/debian/tmp-udeb
endif

> (The rules.custom infrastructure in e2fsprogs's debian/rules file was
> something I had put in a while ago to support building subsets of
> e2fsprogs for certain specialized use cases at $WORK.  It was also
> used way back when to support building new versions of e2fsprogs on
> extremely ancient old-old-old-old-stable.)

These sound like excellent use-cases for build profiles.

> Yeah, it's horribly manual, but when you need to bootstrap a newn
> architecture, it's all manual *anyway*.

Less so these days - I believe there are tools being developed that can
automatically break cycles by building cut-down packages (no documentation,
no FUSE, etc.) and later replacing them with a fully-featured version.

Regards,
smcv
>From 1d50ce0d72d3c4d9397b7e8e13f52bbb19bb5b61 Mon Sep 17 00:00:00 2001
From: Simon McVittie 
Date: Mon, 1 Jan 2018 23:23:36 +
Subject: [PATCH 1/2] [UNTESTED] Add a build profile to skip the FUSE
 dependency

This is a non-standard build profile, but could be used during
bootstrapping.
---
 debian/control | 3 ++-
 debian/rules   | 4 
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/debian/control b/debian/control
index 779f107..6ab004f 100644
--- a/debian/control
+++ b/debian/control
@@ -2,11 +2,12 @@ Source: e2fsprogs
 Section: admin
 Priority: required
 Maintainer: Theodore Y. Ts'o 
-Build-Depends: gettext, texinfo, pkg-config, libfuse-dev [linux-any kfreebsd-any], libattr1-dev, debhelper (>= 9.0), libblkid-dev, uuid-dev, m4
+Build-Depends: gettext, texinfo, pkg-config, libfuse-dev [linux-any kfreebsd-any] , libattr1-dev, debhelper (>= 9.0), libblkid-dev, uuid-dev, m4
 Standards-Version: 4.1.1
 Homepage: http://e2fsprogs.sourceforge.net
 
 Package: fuse2fs
+Build-Profiles: 
 Priority: optional
 Depends: ${shlibs:Depends}, ${misc:Depends}
 Architecture: linux-any kfreebsd-any
diff --git a/debian/rules b/debian/rules
index 841715d..a6592dc 100755
--- a/debian/rules
+++ b/debian/rules
@@ -31,6 +31,10 @@ ifeq ($(DEB_HOST_ARCH_OS), hurd)
 SKIP_FUSE2FS=yes
 endif
 
+ifneq ($(filter pkg.e2fsprogs.nofuse,$(DEB_BUILD_PROFILES)),)
+SKIP_FUSE2FS=yes
+endif
+
 DH_VERSION := $(shell dpkg-query -W -f '$${Version}' debhelper)
 
 # USE_DBGSYM :=
-- 
2.15.1

>From 9be2f6e9e61efa603e970fcd9167320068b9fb6c Mon Sep 17 00:00:00 2001
From: Simon McVittie 
Date: Mon, 1 Jan 2018 23:24:03 +
Subject: [PATCH 2/2] [UNTESTED] Add a build profile to skip the udeb

This is a standard build profile. Not building udebs speeds up test
builds if only the full-fat version is going to be tested anyway.
---
 debian/control | 1 +
 debian/rules   | 4 
 2 files changed, 5 insertions(+)

diff --git a/debian/control b/debian/control
index 6ab004f..c55bfa2 100644
--- a/debian/control
+++ b/debian/control
@@ -103,6 +103,7 @@ Description: command-line interface parsing library - headers and static librari
 
 
 Package: e2fsprogs-udeb
+Build-Profiles: 
 Package-Type: udeb
 Section: debian-installer
 Priority: optional
diff --git a/debian/rules b/debian/rules
index a6592dc..3eb65f1 100755
--- a/debian/rules
+++ b/debian/rules
@@ -35,6 +35,10 @@ ifneq ($(filter pkg.e2fsprogs.nofuse,$(DEB_BUILD_PROFILES)),)
 SKIP_FUSE2FS=yes
 endif
 
+ifneq ($(filter 

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Manuel A. Fernandez Montecelo]

Hi!

2018-01-01 22:51 Theodore Ts'o:


This probably doesn't help much, but for people who are doing things
by hand, you can skip the dependency on fuse by unpacking the
e2fsprogs source packaging, adding the file debian/rules.custom which
contains the single line, "SKIP_FUSE2FS=yes", and building by hand.

It currently doesn't automatically fix up the control file, but I can
set things up so that adding the rules.custom file with
SKIP_FUSE2FS=yes, and running "./debian/rules debian-files" will
automatically rewrite the control file dropping the fuse2fs
dependencies and the "fuse2fs" package from the control.in file.

Which might not matter that much, since when bootstrapping a new
architecture, it's all done manually anyway, so having a properly
update debian/control file might not matter that much.


Lately architectures tend to use automatic bootstrapping at least for
some of the initial dependencies.  Adding support for build profiles
(would be something like pkg.e2fsprogs.nofuse in this case) can help to
build just by using env variables when invoking dpkg-buildpackage or
other build tools.

Would you accept patches to achieve this in e2fsprogs?  It would
probably be quite clean, not complicating/obfuscating the packaging
files too much, usually only 2~10 lines (but I didn't look specifically
into this package yet).

(We can continue off-list, I thought that it's useful for other people
if they are not away of the above).



Yeah, it's horribly manual, but when you need to bootstrap a newn
architecture, it's all manual *anyway*.  And yes, it's a workaround
compared to dropping e2fsprogs from the essential set (for which I
still support), but it's a workaround that works today.  I suppose the
real problem is that a random developer who is trying to bootstrap
Debian on a new architecture won't know about this trick, but in case
it's helpful, I thought I would mention it.


Thanks, it's useful to know :)



(Waving to the RISC-V folks.)


o/


--
Manuel A. Fernandez Montecelo 

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Mon, Nov 13, 2017 at 08:35:10PM +0100, Helmut Grohne wrote:
> > To to be clear, the key metric for your specific goal is the reduction
> > of the _source_ package count since the goal is to reduce the number
> > of packages which have to be built by "hand" (or by script), before
> > you can create a sbuild/pbuild build chroot, correct?
> 
> Correct. Unless I am mistaken, removing e2fsprogs from the build set
> also removes fuse.

Apologies for the thread necromancy, but I was going through old bugs
and old todo items for e2fsprogs debian package, and I was rereading
this thread as part of that.

This probably doesn't help much, but for people who are doing things
by hand, you can skip the dependency on fuse by unpacking the
e2fsprogs source packaging, adding the file debian/rules.custom which
contains the single line, "SKIP_FUSE2FS=yes", and building by hand.

It currently doesn't automatically fix up the control file, but I can
set things up so that adding the rules.custom file with
SKIP_FUSE2FS=yes, and running "./debian/rules debian-files" will
automatically rewrite the control file dropping the fuse2fs
dependencies and the "fuse2fs" package from the control.in file.

Which might not matter that much, since when bootstrapping a new
architecture, it's all done manually anyway, so having a properly
update debian/control file might not matter that much.

(The rules.custom infrastructure in e2fsprogs's debian/rules file was
something I had put in a while ago to support building subsets of
e2fsprogs for certain specialized use cases at $WORK.  It was also
used way back when to support building new versions of e2fsprogs on
extremely ancient old-old-old-old-stable.)

Yeah, it's horribly manual, but when you need to bootstrap a newn
architecture, it's all manual *anyway*.  And yes, it's a workaround
compared to dropping e2fsprogs from the essential set (for which I
still support), but it's a workaround that works today.  I suppose the
real problem is that a random developer who is trying to bootstrap
Debian on a new architecture won't know about this trick, but in case
it's helpful, I thought I would mention it.  (Waving to the RISC-V
folks.)

Cheers,

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Helmut Grohne]

On Mon, Nov 13, 2017 at 11:59:46AM -0500, Theodore Ts'o wrote:
> But if you really are focused on getting to Essential: no, and not
> necessarily changing the priority field, that certainly is a much more
> easily achievable goal.

Of course, reducing minbase would be nice as well, but I agree with your
assertion that there is lower hanging fruit to achieve that. I wouldn't
rule out the super-exciting idea as a later optimization step though.

> Yeah, I think the bigger question is whether any of a reduced minbase
> needs lsattr/chattr in the first place.

That's a very good question. You'll notice that packages such as
resolvconf (lsattr/chattr), initramfs-core-tools (logsave) or
initscripts (logsave) rely on e2fsprogs. Yet, I did find cases where
removing e2fsprogs (with --allow-remove-essential) works. But the
--allow-remove-essential flag feels super-exciting. :-(

> To to be clear, the key metric for your specific goal is the reduction
> of the _source_ package count since the goal is to reduce the number
> of packages which have to be built by "hand" (or by script), before
> you can create a sbuild/pbuild build chroot, correct?

Correct. Unless I am mistaken, removing e2fsprogs from the build set
also removes fuse.

Helmut

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Mon, Nov 13, 2017 at 03:28:32PM +0100, Helmut Grohne wrote:
> 
> On Sun, Nov 12, 2017 at 02:18:45PM -0500, Theodore Ts'o wrote:
> > 1)  If people really want to make e2fsprogs non-essential, I'm not
> > going to object seriously.  It's the downgrade of e2fsprogs from
> > Priority: required to Priority: important which where things get
> > super-exciting.

By the way, when I said "super-exciting", that was a reference to an
management euphemism "uncomfortably excited" which generally refers to
the excitement one feels when "working without a net while crossing
the Grand Canyon on a tightrope" :-)

But if you really are focused on getting to Essential: no, and not
necessarily changing the priority field, that certainly is a much more
easily achievable goal.

> > 3) Lsattr/chattr et.al depend on the e2fsprogs shared libraries, so
> > moving them into a separate binary package isn't going to save as much
> > space as you would like.  So it's not at all clear the complexity is
> > worth it.
> 
> I'm not enthusiastic about moving lsattr either for precisely the reasons
> you name.

Yeah, I think the bigger question is whether any of a reduced minbase
needs lsattr/chattr in the first place.

> Reducing the package count lowers the complexity of the bootstrap
> problem. If e2fsprogs (or anything else) can be moved to the native
> phase, that's a win.

To to be clear, the key metric for your specific goal is the reduction
of the _source_ package count since the goal is to reduce the number
of packages which have to be built by "hand" (or by script), before
you can create a sbuild/pbuild build chroot, correct?

Cheers,

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Helmut Grohne]

Hi Theodore,

Thanks for speaking up.

On Sun, Nov 12, 2017 at 02:18:45PM -0500, Theodore Ts'o wrote:
> 1)  If people really want to make e2fsprogs non-essential, I'm not
> going to object seriously.  It's the downgrade of e2fsprogs from
> Priority: required to Priority: important which where things get
> super-exciting.

It's true that merely making e2fsprogs non-essential will not shrink
minbase. There are other chroot generators such as multistrap that
permit making smaller chroots though. From my pov, the important bit in
making e2fsprogs non-essential is to get the relevant dependencies in
place to figure out when it is safe to remove it. (See below.)

> 2) I'm at this point I'm not really enthuiastic to move lsattr out of
> e2fsprogs.  We are still adding new features to ext4, some of which
> will require new flags, and chattr/lsattr et.al. *were* originally
> designed to be only for ext 2/3/4.  Other file systems have decided to
> use the same ioctl, which is fine, but I've always considered
> chattr/lsattr to be an ext 2/3/4 utility first, and more generic file
> system utility second.  Moving lsattr out of e2fsprogs to some other
> package (e.g., util-linux) will make my kernel development much more
> inconvenient.
> 
> 3) Lsattr/chattr et.al depend on the e2fsprogs shared libraries, so
> moving them into a separate binary package isn't going to save as much
> space as you would like.  So it's not at all clear the complexity is
> worth it.

I'm not enthusiastic about moving lsattr either for precisely the reasons
you name.

> 4) If the real goal is reduce the size of minbase, there is a much
> more effective thing we can do first, or at least, in parallel.  And
> that is to move the l10n files to a separate foo-l10n package.  The
> last time I did this analysis was with Debian Jessie, but I don't
> think the numbers have changed that much.  Of the 201 MB i386 minbase
> chroot, 33MB, or over 16% can be found in /usr/local/locale.  The
> breakdown (using Debian Jessie numbers) are:

My goal here is a twofold shrinking. You can shrink the package count
and the installation size.

Reducing the package count lowers the complexity of the bootstrap
problem. If e2fsprogs (or anything else) can be moved to the native
phase, that's a win.

Using Debian for embedded typically requires moving beyond minbase.
Removing locales is typically done with --path-exclude or similar,
because it is mostly riskless. Following your proposal (from your next
reply) to simply remove binaries is not riskless, as anything might use
them (due to being essential). Removing a whole package is less risky as
apt will complain when you do something stupid.

So if you go beyond the locale and /usr/share/doc removal, e2fsprogs
suddenly looks like low hanging fruit after perl, but perl is much
harder to remove. In a sense, e2fsprogs might be the lowest hanging
fruit. (And that always depends on what the goal is.)

> Does this mean trying to get to Essential: no for e2fsprogs is a bad
> thing?  No, but if your goal is reduce the size of minbase for Debian,
> I just want to point out that there is **much** lower hanging fruit
> that folks might want to consider harvesting first.

For reducing e.g. docker images, your idea to tackle coreutils is a good
one. I leave that for others to work on and note that the second step
(after splitting coreutils and others) will become easier due to making
e2fsprogs non-essential.

I note that I tend to ignore the Priority field. ;)

Helmut

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Simon McVittie]

On Sun, 12 Nov 2017 at 14:18:45 -0500, Theodore Ts'o wrote:
> P.S.  In case it isn't obvious, the reason why it's interesting to
> shrink the size of minbase is that it makes Debian much lighter-weight
> for Docker

The same is true for Flatpak runtimes, which are purely for user-level
apps and so in principle shouldn't need anything from /{usr/,}sbin or
man page section 8 at all. (Although in practice they do need at least
ldconfig, so scripts to make a Flatpak runtime can't just delete /sbin.)

At the moment my approach to creating Flatpak runtimes involves a lot of
"dpkg --force-remove-essential --force-depends" (most notably to remove
dpkg and apt, which don't make sense in a read-only runtime that can
be replaced with a newer version from the outside but cannot be updated
from the inside) but not much manual deletion.

smcv

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Mon, Nov 13, 2017 at 01:14:01AM +0100, Guillem Jover wrote:
> I think that trying to trim down the pseudo-Essential set is an
> extremely worthwhile goal, because it has visible effects on several
> areas, at least:
> 
>  - Possibly making bootstrapping a port way easier.
>  - Making it possible and easier to use Debian on (very) embedded systems.
>  - Reducing footprint for minbase setups, such as VM images, buildds,
>chroots, and similar.

Except for a port, you will need *some* file system, so simply
removing all file system utilities from the minbase set doesn't
necessarily make it *easier* per se.

And most minbase setups aren't necessarily manually removing locale
files today, because debootstrap doesn't support this.  I'm just
pointing out that *just* simply splitting out coreutils into coreutils
and coreutils-l10n will shrink the minbase set by roughly as much as
what is listed at the EssentialOnDiet page.

This is not an argument to not do the other things on the
EssentialOnDiet page.  I'm just pointing out there's quite a lot of
low-hanging fruit that can also be harvested if the priamry goal is
reduction of minbase for VM images, chroots, buildds, etc.  And I
don't think it should be neglected.

I will certainly grant that if the goal is to make Debian work on
super-tiny embedded systems we will need to eject a lot of things from
minbase, including bash, tar, perl-base, etc.  And if the super-tiny
embedded system is going to use squashfs, and is not any other on-disk
file system, then sure, that's certainly a case where removing
e2fsprogs makes sense.

But there are *plenty* of use cases where people are using a minbase
created using debootstrap where there is some lower-hanging fruit that
we might want to pick first.

Cheers,

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Guillem Jover]

On Sun, 2017-11-12 at 18:27:16 -0500, Theodore Ts'o wrote:
> On Sun, Nov 12, 2017 at 09:13:42PM +0100, Mathieu Parent wrote:
> > There is another way to trim the locales: Use dpkg's "--path-exclude=".
> > 
> > This also allows one to keep some locales. This is what we use at work
> > [1]. The problem is that debootstrap doesn't handle those options, so
> > we need to hack a bit [2].
> > 
> > [1]: 
> > https://github.com/nantesmetropole/docker-debian/blob/master/templates/etc/dpkg/dpkg.cfg.d/01_save-space
> > [2]: 
> > https://github.com/nantesmetropole/docker-debian/blob/master/templates/post-debootstrap.sh
> 
> You can always manually delete binaries afterwards or by using
> --path-exclude, but that has always seemed to be a hack to me.  By
> that argument there's no point making e2fsprogs Essential: no /
> Priority: important, since you could just remove the files you don't
> want (mke2fs, e2fsck, etc.) afterwards.  :-)

I don't think that's comparable at all. Excluding documentation, locales
and similar stuff is perfectly fine (and even permitted by policy), as
those are not supposed to be expected even by the same package, and are
not part of the interface provided by the package towards other packages.

Doing the same with binaries or any other interface-providing file
means that dependency system is undermined, more so for an Essential
package, because as the dependencies are implicit, it means *any*
package (its content and maintainer scripts) might become broken,
perhaps in a silent way.


I think that trying to trim down the pseudo-Essential set is an
extremely worthwhile goal, because it has visible effects on several
areas, at least:

 - Possibly making bootstrapping a port way easier.
 - Making it possible and easier to use Debian on (very) embedded systems.
 - Reducing footprint for minbase setups, such as VM images, buildds,
   chroots, and similar.

And most importantly, please do not take this personally, this is not
some concerted effort against e2fsprogs! :) Personally I (and I think
others too) would like to see many other packages out from the
pseudo-Essential set, such as, to name a few: bash, perl-base, tar,
login (any init and login thing really).

For more info please see https://wiki.debian.org/Proposals/EssentialOnDiet
and https://wiki.debian.org/BusterPriorityRequalification.

Thanks,
Guillem

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Sun, Nov 12, 2017 at 09:13:42PM +0100, Mathieu Parent wrote:
> 
> There is another way to trim the locales: Use dpkg's "--path-exclude=".
> 
> This also allows one to keep some locales. This is what we use at work
> [1]. The problem is that debootstrap doesn't handle those options, so
> we need to hack a bit [2].
> 
> [1]: 
> https://github.com/nantesmetropole/docker-debian/blob/master/templates/etc/dpkg/dpkg.cfg.d/01_save-space
> [2]: 
> https://github.com/nantesmetropole/docker-debian/blob/master/templates/post-debootstrap.sh

You can always manually delete binaries afterwards or by using
--path-exclude, but that has always seemed to be a hack to me.  By
that argument there's no point making e2fsprogs Essential: no /
Priority: important, since you could just remove the files you don't
want (mke2fs, e2fsck, etc.) afterwards.  :-)

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Mathieu Parent]

Hi,

2017-11-12 20:18 GMT+01:00 Theodore Ts'o :
[...]
> 4) If the real goal is reduce the size of minbase, there is a much
> more effective thing we can do first, or at least, in parallel.  And
> that is to move the l10n files to a separate foo-l10n package.  The
> last time I did this analysis was with Debian Jessie, but I don't
> think the numbers have changed that much.  Of the 201 MB i386 minbase
> chroot, 33MB, or over 16% can be found in /usr/local/locale.  The
> breakdown (using Debian Jessie numbers) are:
[...]
> P.S.  In case it isn't obvious, the reason why it's interesting to
> shrink the size of minbase is that it makes Debian much lighter-weight
> for Docker --- you don't need e2fsck or mke2fs in most docker
> containers based on Docker; neither do you need the translations into
> Turkish, German, Spanish, Chinese, etc., for e2fsprogs, coreutils,
> dpkg, etc., for most Docker containers.

There is another way to trim the locales: Use dpkg's "--path-exclude=".

This also allows one to keep some locales. This is what we use at work
[1]. The problem is that debootstrap doesn't handle those options, so
we need to hack a bit [2].

[1]: 
https://github.com/nantesmetropole/docker-debian/blob/master/templates/etc/dpkg/dpkg.cfg.d/01_save-space
[2]: 
https://github.com/nantesmetropole/docker-debian/blob/master/templates/post-debootstrap.sh

Regards

-- 
Mathieu Parent

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Mon, Oct 02, 2017 at 01:34:47PM +0200, Adam Borowski wrote:
> But, we're discussing changes to e2fsprogs behind its maintainer's back.  I
> believe he reads debian-devel, but, being nowhere like a frequent poster,
> apparently doesn't watch new threads immediately as they appear (and this
> one started as a response to a 2011 post).
> 
> Ted: could you please chime in?  In case you unsubscribed d-devel, it starts
> at https://lists.debian.org/debian-devel/2017/09/msg00449.html

Apologies for not responding sooner.  Between a crazy travel schedule
(kernel summit in Prague, teaching a tutorial at LISA 2017 in San
Francisco, and recovering from a killer case of Flu), I just hadn't
really gotten to this earlier.

1)  If people really want to make e2fsprogs non-essential, I'm not
going to object seriously.  It's the downgrade of e2fsprogs from
Priority: required to Priority: important which where things get
super-exciting.

2) I'm at this point I'm not really enthuiastic to move lsattr out of
e2fsprogs.  We are still adding new features to ext4, some of which
will require new flags, and chattr/lsattr et.al. *were* originally
designed to be only for ext 2/3/4.  Other file systems have decided to
use the same ioctl, which is fine, but I've always considered
chattr/lsattr to be an ext 2/3/4 utility first, and more generic file
system utility second.  Moving lsattr out of e2fsprogs to some other
package (e.g., util-linux) will make my kernel development much more
inconvenient.

3) Lsattr/chattr et.al depend on the e2fsprogs shared libraries, so
moving them into a separate binary package isn't going to save as much
space as you would like.  So it's not at all clear the complexity is
worth it.

4) If the real goal is reduce the size of minbase, there is a much
more effective thing we can do first, or at least, in parallel.  And
that is to move the l10n files to a separate foo-l10n package.  The
last time I did this analysis was with Debian Jessie, but I don't
think the numbers have changed that much.  Of the 201 MB i386 minbase
chroot, 33MB, or over 16% can be found in /usr/local/locale.  The
breakdown (using Debian Jessie numbers) are:

Package Savings Cummulative
(kB)Savings (kB)  Percentage
coreutils   8052 805224.91
dpkg46201267239.20
bash37441641650.78
gnupg   34241984061.37
e2fsprogs   17762161666.86
tar 16802329672.06
shadow  16322492877.11
apt 15282645681.84
libapt-pkg4.12  10522750885.09
Linux-PAM   796 2830487.55
findutils   756 2906089.89
grep636 2969691.86
diffutils   620 3031693.78
debconf 596 3091295.62
adduser 444 3135696.99
sed 428 3178498.32
libgpg-error388 3217299.52
systemd 84  3225699.78
acl 72  32328   100.00

In Debian Stretch, I've already done this separation for e2fsprogs, so
the installed size of e2fsprogs is only 1309kB.  And so I've already
made harvested way more than half of the savings of getting rid of
e2fsprogs from the minbase set by the simple expedient of moving the
/usr/share/locale files to e2fsprogs-l10n.

Simply splitting coreutils into coreutils and coreutils-l10n would
reduce minbase by a factor of *six* over getting rid of e2fsprogs from
minbase.

Does this mean trying to get to Essential: no for e2fsprogs is a bad
thing?  No, but if your goal is reduce the size of minbase for Debian,
I just want to point out that there is **much** lower hanging fruit
that folks might want to consider harvesting first.

Cheers,

- Ted

P.S.  In case it isn't obvious, the reason why it's interesting to
shrink the size of minbase is that it makes Debian much lighter-weight
for Docker --- you don't need e2fsck or mke2fs in most docker
containers based on Docker; neither do you need the translations into
Turkish, German, Spanish, Chinese, etc., for e2fsprogs, coreutils,
dpkg, etc., for most Docker containers.

When I asked the question *why* is it worth it to spend the effort to
reduce the size of Essential: yes, I was told it was it was a
prerequisite of reducing the set of packages where Priority is
"required", and *that* was because it allowed for a reduction of the
minbase set, which in turn is intersting for reducing the size of
Docker images.  If there are other reasons why people want to make
e2fsprogs no longer essential: yes, it's good to have those out on the
table, since it may be that there are other things we can do first, or
in parallel, that might be *far* more effective towards the real end
goal that people have.