Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Sat, Jan 06, 2018 at 02:25:55AM +0100, Manuel A. Fernandez Montecelo wrote:
> 2018-01-02 03:10 Theodore Ts'o:
> > My only real concern is whether this might complicate building the
> > latest version of e2fsprogs for stable and old-stable for
> > debian-backports.
> 
> I think that it's fine, they've been supported for a long time and
> stable and old-stable should be covered, not sure about old-old-stable
> without backports.

It's mostly fine.  The one problem I found was that I tried to add the
*-dbg packages to the control file such that they would only be built
if the build profile pkg.e2fsprogs.legacy-dbg was active.
Unfortunately, when I tried doing a source-only upload of e2fsprogs,
the DAK software complained that "e2fsprogs-dbg" et.al. were new
packages that weren't supported with a source-only upload.

That's apparently because it was parsing the control file, found the
package declaration, and assumed that build profiles will only
suppress packages (e.g., such as fuse2fs or documentation pckages).
It didn't assume that a non-standard build profile such as
pkg.e2fsprogs.legacy-dbg would *enable* a package, and that under
normal builds, the *-dbg packages wouldn't be built.  Which is fair
enough, there's no way DAK could determine that without actually
building the source-only upload.

So I had to move the *-dbg package definitions into a
debian/control.legacy-dbg file, and mutate debian/control in order to
both support jessie backports, and keep DAK happy.

Which is sad, but it's a solution which works --- and I understand why
there really isn't any other way for DAK to handle this case.

Cheers,

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Manuel A. Fernandez Montecelo]

2018-01-02 03:10 Theodore Ts'o:

My only real concern is whether this might complicate building the
latest version of e2fsprogs for stable and old-stable for
debian-backports.


I think that it's fine, they've been supported for a long time and
stable and old-stable should be covered, not sure about old-old-stable
without backports.


2018-01-03 23:59 Theodore Ts'o:


Would you accept patches to achieve this in e2fsprogs?  It would
probably be quite clean, not complicating/obfuscating the packaging
files too much, usually only 2~10 lines (but I didn't look specifically
into this package yet).


With some help from Simon McVittie, you should be able to use the
build profile pkg.e2fsprogs.no-fuse2fs in the just-uploaded 1.43.8-2
version of e2fsprogs.  It seems to work for me; please let me know if
does what you need.


Not immediately but I will do soon, and possibly send fixes or more
profiles if they're convenient for bootstrapping :)

Thanks, and thanks to Simon too for beating me to it!

--
Manuel A. Fernandez Montecelo 

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Tue, Jan 02, 2018 at 12:38:55AM +0100, Manuel A. Fernandez Montecelo wrote:
> 
> Lately architectures tend to use automatic bootstrapping at least for
> some of the initial dependencies.  Adding support for build profiles
> (would be something like pkg.e2fsprogs.nofuse in this case) can help to
> build just by using env variables when invoking dpkg-buildpackage or
> other build tools.
> 
> Would you accept patches to achieve this in e2fsprogs?  It would
> probably be quite clean, not complicating/obfuscating the packaging
> files too much, usually only 2~10 lines (but I didn't look specifically
> into this package yet).

With some help from Simon McVittie, you should be able to use the
build profile pkg.e2fsprogs.no-fuse2fs in the just-uploaded 1.43.8-2
version of e2fsprogs.  It seems to work for me; please let me know if
does what you need.

Cheers,

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Wed, Jan 03, 2018 at 08:16:35PM +, Simon McVittie wrote:
> > Has there been any thought about having the build
> > profiles framework support for having the rules file autoselect a
> > build profile based on the build environment?
> 
> I suspect that might be a "considered and rejected" sort of thing,
> because toolchain maintainers want the same command to always do more
> or less the same thing. If you want some automation for enabling special
> build profiles, I'd suggest wrapping it around the outside instead.
> That also means it's allowed to edit debian/control if it needs to.

Actually, after doing some experimentation, I was able to make this
work.  From the debian/rules file:

USE_DBGSYM ?= $(shell if dpkg --compare-versions $(DH_VERSION) ">=" 9.20160114 
; then echo yes ; fi)

ifeq ($(USE_DBGSYM),yes)
dh_strip_args = -p$(1) --dbgsym-migration='$(1)-dbg (<= 1.43-1)'
dh_strip_args2 = -p$(1) --dbgsym-migration='$(2)-dbg (<= 1.43-1)'
else
dh_strip_args = -p$(1) --dbg-package=$(1)-dbg
dh_strip_args2 = -p$(1) --dbg-package=$(2)-dbg
DBG_PACKAGES += -pe2fsprogs-dbg -pe2fslibs-dbg -plibcomerr2-dbg -plibss2-dbg
export DEB_BUILD_PROFILES += pkg.e2fsprogs.legacy-dbg
endif

Which is actually cool, because it means you can do an "apt-get source
e2fsprogs", "schroot -c jessie-amd64", and then run "dpkg-buildpackage
-us -uc --changes-option=-S" and have the right thing happen
automagically.

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Simon McVittie]

On Wed, 03 Jan 2018 at 12:40:45 -0500, Theodore Ts'o wrote:
> From the BuildProfile web page and
> looking at the package versions, support for it appears to be in
> Jessie, or at least "preliminary support" is present.

jessie's apt/dpkg at least don't choke on the new syntax (I
tried backporting flatpak from buster, which has , for
jessie-backports-sloppy and it seemed OK), although I didn't actually
try doing a nodoc build for jessie. I'd suggest using jessie-backports'
debhelper (and dh-exec if you use that) if targeting jessie.

(buster's flatpak isn't actually present in jessie-backports-sloppy
because I'm not sure I can commit to having the time or feasibility
to patch it into working on jessie for the entire lifetime of buster,
which seems to be the bar set by the backports maintainers.)

> Since the autodetection isn't there, I would have to just manually
> build with some kind of pkg.e2fsprogs.old-dbg build profile, or some
> such, instead.  I guess it's about the same level of inconvenience of
> needing to run ./debian/rules in order to generate the control file
> from control.in.

Right. If you're doing special package-specific runes anyway, you could
add a `debian/build-backport` that activates the right build profiles
for you instead?

The examples of build profiles in my packages are all "negative"
(Build-Profiles: ), which I think are much more common; but
it's also possible to have "positive" build profiles which enable
extra packages that are not normally built
(Build-Profiles:  or something).

> My "./debian-rules debian-files" scheme used to do a lot more,
> including rewriting several other files in debian/ using m4

dh-exec covers a lot of the uses for that sort of trick, although (like
everything invoked at build time) it can't touch debian/control.

> we had just migrated libuuid and libblkid from e2fsprogs to
> util-linux, and I wanted to support backports to stable, old-stable,
> and Ubuntu LTS.

For the Flatpak family (some of which need non-trivial patching to
support jessie's older GLib) I've been happy with `git merge` into a
separate branch, tbh...

> Has there been any thought about having the build
> profiles framework support for having the rules file autoselect a
> build profile based on the build environment?

I suspect that might be a "considered and rejected" sort of thing,
because toolchain maintainers want the same command to always do more
or less the same thing. If you want some automation for enabling special
build profiles, I'd suggest wrapping it around the outside instead.
That also means it's allowed to edit debian/control if it needs to.

smcv

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Mon, Jan 01, 2018 at 11:43:23PM +, Simon McVittie wrote:
> 
> Perhaps you could convert this into a pkg.e2fsprogs.nofuse build profile?
> 
> This would look something like the attached (untested!) patches.

Thanks, I'll give this a try.  From the BuildProfile web page and
looking at the package versions, support for it appears to be in
Jessie, or at least "preliminary support" is present.  Is that
correct?  Is there any gotchas I should be aware of when backporting
to Jessie?

It looks like one thing which my scheme supports which build profiles
does not is that when backporting to Jessie, I can just check out the
tree from git, and then run:

./debian/rules debian-files
dpkg-buildpackage

... and it will autodetect that the system doesn't support *-dbgsym
packages, and create e2fsprogs-dbg, e2fslibs-dbg, et.al packages for
Jessie instead.

Since the autodetection isn't there, I would have to just manually
build with some kind of pkg.e2fsprogs.old-dbg build profile, or some
such, instead.  I guess it's about the same level of inconvenience of
needing to run ./debian/rules in order to generate the control file
from control.in.

My "./debian-rules debian-files" scheme used to do a lot more,
including rewriting several other files in debian/ using m4, back when
we had just migrated libuuid and libblkid from e2fsprogs to
util-linux, and I wanted to support backports to stable, old-stable,
and Ubuntu LTS.  I did like the fact that it could detect which build
option (now, "build profile") to use automatically, so the folks
building backports for various Debian derivitives didn't need to do
anything special.  Has there been any thought about having the build
profiles framework support for having the rules file autoselect a
build profile based on the build environment?

Cheers,

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Tue, Jan 02, 2018 at 12:38:55AM +0100, Manuel A. Fernandez Montecelo wrote:
> Lately architectures tend to use automatic bootstrapping at least for
> some of the initial dependencies.  Adding support for build profiles
> (would be something like pkg.e2fsprogs.nofuse in this case) can help to
> build just by using env variables when invoking dpkg-buildpackage or
> other build tools.
> 
> Would you accept patches to achieve this in e2fsprogs?  It would
> probably be quite clean, not complicating/obfuscating the packaging
> files too much, usually only 2~10 lines (but I didn't look specifically
> into this package yet).

Sure.  The debian/rules.custom support that I put in predates the
Debian build profiles spec, but it's basically the same idea.  So it
should be pretty simple to migrate things to use the build profile
concept; it's just a matter of wiring up things slightly differently
in debian/rules.

I haven't looked at build profiles spec super-closely yet, but since
it looks like there is core infrastructural support for it in dpkg,
debhelpers, and friends, it's no longer necessary to run
debian/control.in through m4 to generate debian/control (which is how
I did things sans dpkg support).

Send me a patch, either sent to linux-e...@vger.kernel.org, or as
Debian bug, and I'll happily take it.

My only real concern is whether this might complicate building the
latest version of e2fsprogs for stable and old-stable for
debian-backports.

Cheers,

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Simon McVittie]

On Mon, 01 Jan 2018 at 16:51:45 -0500, Theodore Ts'o wrote:
> This probably doesn't help much, but for people who are doing things
> by hand, you can skip the dependency on fuse by unpacking the
> e2fsprogs source packaging, adding the file debian/rules.custom which
> contains the single line, "SKIP_FUSE2FS=yes", and building by hand.

Perhaps you could convert this into a pkg.e2fsprogs.nofuse build profile?

This would look something like the attached (untested!) patches.

The intention seems to be that build profiles are how this is done
automatically or near-automatically in future. dbus and glib2.0 are
examples of packages where I've made use of build profiles to cut off
non-essential functionality for bootstrapping (they're significantly
further up the stack than e2fsprogs, but were involved in a cycle in
the past).

Having nodoc and noudeb build profiles (assuming the package has
documentation and udebs) also makes local test-builds a lot faster for
many packages, which is very convenient when testing functional changes.

> It currently doesn't automatically fix up the control file

You can skip binary packages on a per-build-profile basis without having
to generate debian/control from debian/control.in. After doing that,
it's often convenient to write build profile checks in debian/rules as
a check for "are we building package foo?" rather than "are we using
build profile foo?", like this in dbus:

binaries := $(shell dh_listpackages)

[...]

ifneq ($(filter dbus-udeb,$(binaries)),)
# udeb build
make -C debian/build-udeb install-exec 
DESTDIR=$(CURDIR)/debian/tmp-udeb
endif

> (The rules.custom infrastructure in e2fsprogs's debian/rules file was
> something I had put in a while ago to support building subsets of
> e2fsprogs for certain specialized use cases at $WORK.  It was also
> used way back when to support building new versions of e2fsprogs on
> extremely ancient old-old-old-old-stable.)

These sound like excellent use-cases for build profiles.

> Yeah, it's horribly manual, but when you need to bootstrap a newn
> architecture, it's all manual *anyway*.

Less so these days - I believe there are tools being developed that can
automatically break cycles by building cut-down packages (no documentation,
no FUSE, etc.) and later replacing them with a fully-featured version.

Regards,
smcv
>From 1d50ce0d72d3c4d9397b7e8e13f52bbb19bb5b61 Mon Sep 17 00:00:00 2001
From: Simon McVittie 
Date: Mon, 1 Jan 2018 23:23:36 +
Subject: [PATCH 1/2] [UNTESTED] Add a build profile to skip the FUSE
 dependency

This is a non-standard build profile, but could be used during
bootstrapping.
---
 debian/control | 3 ++-
 debian/rules   | 4 
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/debian/control b/debian/control
index 779f107..6ab004f 100644
--- a/debian/control
+++ b/debian/control
@@ -2,11 +2,12 @@ Source: e2fsprogs
 Section: admin
 Priority: required
 Maintainer: Theodore Y. Ts'o 
-Build-Depends: gettext, texinfo, pkg-config, libfuse-dev [linux-any kfreebsd-any], libattr1-dev, debhelper (>= 9.0), libblkid-dev, uuid-dev, m4
+Build-Depends: gettext, texinfo, pkg-config, libfuse-dev [linux-any kfreebsd-any] , libattr1-dev, debhelper (>= 9.0), libblkid-dev, uuid-dev, m4
 Standards-Version: 4.1.1
 Homepage: http://e2fsprogs.sourceforge.net
 
 Package: fuse2fs
+Build-Profiles: 
 Priority: optional
 Depends: ${shlibs:Depends}, ${misc:Depends}
 Architecture: linux-any kfreebsd-any
diff --git a/debian/rules b/debian/rules
index 841715d..a6592dc 100755
--- a/debian/rules
+++ b/debian/rules
@@ -31,6 +31,10 @@ ifeq ($(DEB_HOST_ARCH_OS), hurd)
 SKIP_FUSE2FS=yes
 endif
 
+ifneq ($(filter pkg.e2fsprogs.nofuse,$(DEB_BUILD_PROFILES)),)
+SKIP_FUSE2FS=yes
+endif
+
 DH_VERSION := $(shell dpkg-query -W -f '$${Version}' debhelper)
 
 # USE_DBGSYM :=
-- 
2.15.1

>From 9be2f6e9e61efa603e970fcd9167320068b9fb6c Mon Sep 17 00:00:00 2001
From: Simon McVittie 
Date: Mon, 1 Jan 2018 23:24:03 +
Subject: [PATCH 2/2] [UNTESTED] Add a build profile to skip the udeb

This is a standard build profile. Not building udebs speeds up test
builds if only the full-fat version is going to be tested anyway.
---
 debian/control | 1 +
 debian/rules   | 4 
 2 files changed, 5 insertions(+)

diff --git a/debian/control b/debian/control
index 6ab004f..c55bfa2 100644
--- a/debian/control
+++ b/debian/control
@@ -103,6 +103,7 @@ Description: command-line interface parsing library - headers and static librari
 
 
 Package: e2fsprogs-udeb
+Build-Profiles: 
 Package-Type: udeb
 Section: debian-installer
 Priority: optional
diff --git a/debian/rules b/debian/rules
index a6592dc..3eb65f1 100755
--- a/debian/rules
+++ b/debian/rules
@@ -35,6 +35,10 @@ ifneq ($(filter pkg.e2fsprogs.nofuse,$(DEB_BUILD_PROFILES)),)
 SKIP_FUSE2FS=yes
 endif
 
+ifneq ($(filter 

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Manuel A. Fernandez Montecelo]

Hi!

2018-01-01 22:51 Theodore Ts'o:


This probably doesn't help much, but for people who are doing things
by hand, you can skip the dependency on fuse by unpacking the
e2fsprogs source packaging, adding the file debian/rules.custom which
contains the single line, "SKIP_FUSE2FS=yes", and building by hand.

It currently doesn't automatically fix up the control file, but I can
set things up so that adding the rules.custom file with
SKIP_FUSE2FS=yes, and running "./debian/rules debian-files" will
automatically rewrite the control file dropping the fuse2fs
dependencies and the "fuse2fs" package from the control.in file.

Which might not matter that much, since when bootstrapping a new
architecture, it's all done manually anyway, so having a properly
update debian/control file might not matter that much.


Lately architectures tend to use automatic bootstrapping at least for
some of the initial dependencies.  Adding support for build profiles
(would be something like pkg.e2fsprogs.nofuse in this case) can help to
build just by using env variables when invoking dpkg-buildpackage or
other build tools.

Would you accept patches to achieve this in e2fsprogs?  It would
probably be quite clean, not complicating/obfuscating the packaging
files too much, usually only 2~10 lines (but I didn't look specifically
into this package yet).

(We can continue off-list, I thought that it's useful for other people
if they are not away of the above).



Yeah, it's horribly manual, but when you need to bootstrap a newn
architecture, it's all manual *anyway*.  And yes, it's a workaround
compared to dropping e2fsprogs from the essential set (for which I
still support), but it's a workaround that works today.  I suppose the
real problem is that a random developer who is trying to bootstrap
Debian on a new architecture won't know about this trick, but in case
it's helpful, I thought I would mention it.


Thanks, it's useful to know :)



(Waving to the RISC-V folks.)


o/


--
Manuel A. Fernandez Montecelo 

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Mon, Nov 13, 2017 at 08:35:10PM +0100, Helmut Grohne wrote:
> > To to be clear, the key metric for your specific goal is the reduction
> > of the _source_ package count since the goal is to reduce the number
> > of packages which have to be built by "hand" (or by script), before
> > you can create a sbuild/pbuild build chroot, correct?
> 
> Correct. Unless I am mistaken, removing e2fsprogs from the build set
> also removes fuse.

Apologies for the thread necromancy, but I was going through old bugs
and old todo items for e2fsprogs debian package, and I was rereading
this thread as part of that.

This probably doesn't help much, but for people who are doing things
by hand, you can skip the dependency on fuse by unpacking the
e2fsprogs source packaging, adding the file debian/rules.custom which
contains the single line, "SKIP_FUSE2FS=yes", and building by hand.

It currently doesn't automatically fix up the control file, but I can
set things up so that adding the rules.custom file with
SKIP_FUSE2FS=yes, and running "./debian/rules debian-files" will
automatically rewrite the control file dropping the fuse2fs
dependencies and the "fuse2fs" package from the control.in file.

Which might not matter that much, since when bootstrapping a new
architecture, it's all done manually anyway, so having a properly
update debian/control file might not matter that much.

(The rules.custom infrastructure in e2fsprogs's debian/rules file was
something I had put in a while ago to support building subsets of
e2fsprogs for certain specialized use cases at $WORK.  It was also
used way back when to support building new versions of e2fsprogs on
extremely ancient old-old-old-old-stable.)

Yeah, it's horribly manual, but when you need to bootstrap a newn
architecture, it's all manual *anyway*.  And yes, it's a workaround
compared to dropping e2fsprogs from the essential set (for which I
still support), but it's a workaround that works today.  I suppose the
real problem is that a random developer who is trying to bootstrap
Debian on a new architecture won't know about this trick, but in case
it's helpful, I thought I would mention it.  (Waving to the RISC-V
folks.)

Cheers,

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Helmut Grohne]

On Mon, Nov 13, 2017 at 11:59:46AM -0500, Theodore Ts'o wrote:
> But if you really are focused on getting to Essential: no, and not
> necessarily changing the priority field, that certainly is a much more
> easily achievable goal.

Of course, reducing minbase would be nice as well, but I agree with your
assertion that there is lower hanging fruit to achieve that. I wouldn't
rule out the super-exciting idea as a later optimization step though.

> Yeah, I think the bigger question is whether any of a reduced minbase
> needs lsattr/chattr in the first place.

That's a very good question. You'll notice that packages such as
resolvconf (lsattr/chattr), initramfs-core-tools (logsave) or
initscripts (logsave) rely on e2fsprogs. Yet, I did find cases where
removing e2fsprogs (with --allow-remove-essential) works. But the
--allow-remove-essential flag feels super-exciting. :-(

> To to be clear, the key metric for your specific goal is the reduction
> of the _source_ package count since the goal is to reduce the number
> of packages which have to be built by "hand" (or by script), before
> you can create a sbuild/pbuild build chroot, correct?

Correct. Unless I am mistaken, removing e2fsprogs from the build set
also removes fuse.

Helmut

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Mon, Nov 13, 2017 at 03:28:32PM +0100, Helmut Grohne wrote:
> 
> On Sun, Nov 12, 2017 at 02:18:45PM -0500, Theodore Ts'o wrote:
> > 1)  If people really want to make e2fsprogs non-essential, I'm not
> > going to object seriously.  It's the downgrade of e2fsprogs from
> > Priority: required to Priority: important which where things get
> > super-exciting.

By the way, when I said "super-exciting", that was a reference to an
management euphemism "uncomfortably excited" which generally refers to
the excitement one feels when "working without a net while crossing
the Grand Canyon on a tightrope" :-)

But if you really are focused on getting to Essential: no, and not
necessarily changing the priority field, that certainly is a much more
easily achievable goal.

> > 3) Lsattr/chattr et.al depend on the e2fsprogs shared libraries, so
> > moving them into a separate binary package isn't going to save as much
> > space as you would like.  So it's not at all clear the complexity is
> > worth it.
> 
> I'm not enthusiastic about moving lsattr either for precisely the reasons
> you name.

Yeah, I think the bigger question is whether any of a reduced minbase
needs lsattr/chattr in the first place.

> Reducing the package count lowers the complexity of the bootstrap
> problem. If e2fsprogs (or anything else) can be moved to the native
> phase, that's a win.

To to be clear, the key metric for your specific goal is the reduction
of the _source_ package count since the goal is to reduce the number
of packages which have to be built by "hand" (or by script), before
you can create a sbuild/pbuild build chroot, correct?

Cheers,

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Helmut Grohne]

Hi Theodore,

Thanks for speaking up.

On Sun, Nov 12, 2017 at 02:18:45PM -0500, Theodore Ts'o wrote:
> 1)  If people really want to make e2fsprogs non-essential, I'm not
> going to object seriously.  It's the downgrade of e2fsprogs from
> Priority: required to Priority: important which where things get
> super-exciting.

It's true that merely making e2fsprogs non-essential will not shrink
minbase. There are other chroot generators such as multistrap that
permit making smaller chroots though. From my pov, the important bit in
making e2fsprogs non-essential is to get the relevant dependencies in
place to figure out when it is safe to remove it. (See below.)

> 2) I'm at this point I'm not really enthuiastic to move lsattr out of
> e2fsprogs.  We are still adding new features to ext4, some of which
> will require new flags, and chattr/lsattr et.al. *were* originally
> designed to be only for ext 2/3/4.  Other file systems have decided to
> use the same ioctl, which is fine, but I've always considered
> chattr/lsattr to be an ext 2/3/4 utility first, and more generic file
> system utility second.  Moving lsattr out of e2fsprogs to some other
> package (e.g., util-linux) will make my kernel development much more
> inconvenient.
> 
> 3) Lsattr/chattr et.al depend on the e2fsprogs shared libraries, so
> moving them into a separate binary package isn't going to save as much
> space as you would like.  So it's not at all clear the complexity is
> worth it.

I'm not enthusiastic about moving lsattr either for precisely the reasons
you name.

> 4) If the real goal is reduce the size of minbase, there is a much
> more effective thing we can do first, or at least, in parallel.  And
> that is to move the l10n files to a separate foo-l10n package.  The
> last time I did this analysis was with Debian Jessie, but I don't
> think the numbers have changed that much.  Of the 201 MB i386 minbase
> chroot, 33MB, or over 16% can be found in /usr/local/locale.  The
> breakdown (using Debian Jessie numbers) are:

My goal here is a twofold shrinking. You can shrink the package count
and the installation size.

Reducing the package count lowers the complexity of the bootstrap
problem. If e2fsprogs (or anything else) can be moved to the native
phase, that's a win.

Using Debian for embedded typically requires moving beyond minbase.
Removing locales is typically done with --path-exclude or similar,
because it is mostly riskless. Following your proposal (from your next
reply) to simply remove binaries is not riskless, as anything might use
them (due to being essential). Removing a whole package is less risky as
apt will complain when you do something stupid.

So if you go beyond the locale and /usr/share/doc removal, e2fsprogs
suddenly looks like low hanging fruit after perl, but perl is much
harder to remove. In a sense, e2fsprogs might be the lowest hanging
fruit. (And that always depends on what the goal is.)

> Does this mean trying to get to Essential: no for e2fsprogs is a bad
> thing?  No, but if your goal is reduce the size of minbase for Debian,
> I just want to point out that there is **much** lower hanging fruit
> that folks might want to consider harvesting first.

For reducing e.g. docker images, your idea to tackle coreutils is a good
one. I leave that for others to work on and note that the second step
(after splitting coreutils and others) will become easier due to making
e2fsprogs non-essential.

I note that I tend to ignore the Priority field. ;)

Helmut

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Simon McVittie]

On Sun, 12 Nov 2017 at 14:18:45 -0500, Theodore Ts'o wrote:
> P.S.  In case it isn't obvious, the reason why it's interesting to
> shrink the size of minbase is that it makes Debian much lighter-weight
> for Docker

The same is true for Flatpak runtimes, which are purely for user-level
apps and so in principle shouldn't need anything from /{usr/,}sbin or
man page section 8 at all. (Although in practice they do need at least
ldconfig, so scripts to make a Flatpak runtime can't just delete /sbin.)

At the moment my approach to creating Flatpak runtimes involves a lot of
"dpkg --force-remove-essential --force-depends" (most notably to remove
dpkg and apt, which don't make sense in a read-only runtime that can
be replaced with a newer version from the outside but cannot be updated
from the inside) but not much manual deletion.

smcv

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Mon, Nov 13, 2017 at 01:14:01AM +0100, Guillem Jover wrote:
> I think that trying to trim down the pseudo-Essential set is an
> extremely worthwhile goal, because it has visible effects on several
> areas, at least:
> 
>  - Possibly making bootstrapping a port way easier.
>  - Making it possible and easier to use Debian on (very) embedded systems.
>  - Reducing footprint for minbase setups, such as VM images, buildds,
>chroots, and similar.

Except for a port, you will need *some* file system, so simply
removing all file system utilities from the minbase set doesn't
necessarily make it *easier* per se.

And most minbase setups aren't necessarily manually removing locale
files today, because debootstrap doesn't support this.  I'm just
pointing out that *just* simply splitting out coreutils into coreutils
and coreutils-l10n will shrink the minbase set by roughly as much as
what is listed at the EssentialOnDiet page.

This is not an argument to not do the other things on the
EssentialOnDiet page.  I'm just pointing out there's quite a lot of
low-hanging fruit that can also be harvested if the priamry goal is
reduction of minbase for VM images, chroots, buildds, etc.  And I
don't think it should be neglected.

I will certainly grant that if the goal is to make Debian work on
super-tiny embedded systems we will need to eject a lot of things from
minbase, including bash, tar, perl-base, etc.  And if the super-tiny
embedded system is going to use squashfs, and is not any other on-disk
file system, then sure, that's certainly a case where removing
e2fsprogs makes sense.

But there are *plenty* of use cases where people are using a minbase
created using debootstrap where there is some lower-hanging fruit that
we might want to pick first.

Cheers,

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Guillem Jover]

On Sun, 2017-11-12 at 18:27:16 -0500, Theodore Ts'o wrote:
> On Sun, Nov 12, 2017 at 09:13:42PM +0100, Mathieu Parent wrote:
> > There is another way to trim the locales: Use dpkg's "--path-exclude=".
> > 
> > This also allows one to keep some locales. This is what we use at work
> > [1]. The problem is that debootstrap doesn't handle those options, so
> > we need to hack a bit [2].
> > 
> > [1]: 
> > https://github.com/nantesmetropole/docker-debian/blob/master/templates/etc/dpkg/dpkg.cfg.d/01_save-space
> > [2]: 
> > https://github.com/nantesmetropole/docker-debian/blob/master/templates/post-debootstrap.sh
> 
> You can always manually delete binaries afterwards or by using
> --path-exclude, but that has always seemed to be a hack to me.  By
> that argument there's no point making e2fsprogs Essential: no /
> Priority: important, since you could just remove the files you don't
> want (mke2fs, e2fsck, etc.) afterwards.  :-)

I don't think that's comparable at all. Excluding documentation, locales
and similar stuff is perfectly fine (and even permitted by policy), as
those are not supposed to be expected even by the same package, and are
not part of the interface provided by the package towards other packages.

Doing the same with binaries or any other interface-providing file
means that dependency system is undermined, more so for an Essential
package, because as the dependencies are implicit, it means *any*
package (its content and maintainer scripts) might become broken,
perhaps in a silent way.


I think that trying to trim down the pseudo-Essential set is an
extremely worthwhile goal, because it has visible effects on several
areas, at least:

 - Possibly making bootstrapping a port way easier.
 - Making it possible and easier to use Debian on (very) embedded systems.
 - Reducing footprint for minbase setups, such as VM images, buildds,
   chroots, and similar.

And most importantly, please do not take this personally, this is not
some concerted effort against e2fsprogs! :) Personally I (and I think
others too) would like to see many other packages out from the
pseudo-Essential set, such as, to name a few: bash, perl-base, tar,
login (any init and login thing really).

For more info please see https://wiki.debian.org/Proposals/EssentialOnDiet
and https://wiki.debian.org/BusterPriorityRequalification.

Thanks,
Guillem

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Sun, Nov 12, 2017 at 09:13:42PM +0100, Mathieu Parent wrote:
> 
> There is another way to trim the locales: Use dpkg's "--path-exclude=".
> 
> This also allows one to keep some locales. This is what we use at work
> [1]. The problem is that debootstrap doesn't handle those options, so
> we need to hack a bit [2].
> 
> [1]: 
> https://github.com/nantesmetropole/docker-debian/blob/master/templates/etc/dpkg/dpkg.cfg.d/01_save-space
> [2]: 
> https://github.com/nantesmetropole/docker-debian/blob/master/templates/post-debootstrap.sh

You can always manually delete binaries afterwards or by using
--path-exclude, but that has always seemed to be a hack to me.  By
that argument there's no point making e2fsprogs Essential: no /
Priority: important, since you could just remove the files you don't
want (mke2fs, e2fsck, etc.) afterwards.  :-)

- Ted

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Mathieu Parent]

Hi,

2017-11-12 20:18 GMT+01:00 Theodore Ts'o :
[...]
> 4) If the real goal is reduce the size of minbase, there is a much
> more effective thing we can do first, or at least, in parallel.  And
> that is to move the l10n files to a separate foo-l10n package.  The
> last time I did this analysis was with Debian Jessie, but I don't
> think the numbers have changed that much.  Of the 201 MB i386 minbase
> chroot, 33MB, or over 16% can be found in /usr/local/locale.  The
> breakdown (using Debian Jessie numbers) are:
[...]
> P.S.  In case it isn't obvious, the reason why it's interesting to
> shrink the size of minbase is that it makes Debian much lighter-weight
> for Docker --- you don't need e2fsck or mke2fs in most docker
> containers based on Docker; neither do you need the translations into
> Turkish, German, Spanish, Chinese, etc., for e2fsprogs, coreutils,
> dpkg, etc., for most Docker containers.

There is another way to trim the locales: Use dpkg's "--path-exclude=".

This also allows one to keep some locales. This is what we use at work
[1]. The problem is that debootstrap doesn't handle those options, so
we need to hack a bit [2].

[1]: 
https://github.com/nantesmetropole/docker-debian/blob/master/templates/etc/dpkg/dpkg.cfg.d/01_save-space
[2]: 
https://github.com/nantesmetropole/docker-debian/blob/master/templates/post-debootstrap.sh

Regards

-- 
Mathieu Parent

Re: e2fsprogs as Essential: yes?: Maybe we should be separating l10n files first?

[Theodore Ts'o]

On Mon, Oct 02, 2017 at 01:34:47PM +0200, Adam Borowski wrote:
> But, we're discussing changes to e2fsprogs behind its maintainer's back.  I
> believe he reads debian-devel, but, being nowhere like a frequent poster,
> apparently doesn't watch new threads immediately as they appear (and this
> one started as a response to a 2011 post).
> 
> Ted: could you please chime in?  In case you unsubscribed d-devel, it starts
> at https://lists.debian.org/debian-devel/2017/09/msg00449.html

Apologies for not responding sooner.  Between a crazy travel schedule
(kernel summit in Prague, teaching a tutorial at LISA 2017 in San
Francisco, and recovering from a killer case of Flu), I just hadn't
really gotten to this earlier.

1)  If people really want to make e2fsprogs non-essential, I'm not
going to object seriously.  It's the downgrade of e2fsprogs from
Priority: required to Priority: important which where things get
super-exciting.

2) I'm at this point I'm not really enthuiastic to move lsattr out of
e2fsprogs.  We are still adding new features to ext4, some of which
will require new flags, and chattr/lsattr et.al. *were* originally
designed to be only for ext 2/3/4.  Other file systems have decided to
use the same ioctl, which is fine, but I've always considered
chattr/lsattr to be an ext 2/3/4 utility first, and more generic file
system utility second.  Moving lsattr out of e2fsprogs to some other
package (e.g., util-linux) will make my kernel development much more
inconvenient.

3) Lsattr/chattr et.al depend on the e2fsprogs shared libraries, so
moving them into a separate binary package isn't going to save as much
space as you would like.  So it's not at all clear the complexity is
worth it.

4) If the real goal is reduce the size of minbase, there is a much
more effective thing we can do first, or at least, in parallel.  And
that is to move the l10n files to a separate foo-l10n package.  The
last time I did this analysis was with Debian Jessie, but I don't
think the numbers have changed that much.  Of the 201 MB i386 minbase
chroot, 33MB, or over 16% can be found in /usr/local/locale.  The
breakdown (using Debian Jessie numbers) are:

Package Savings Cummulative
(kB)Savings (kB)  Percentage
coreutils   8052 805224.91
dpkg46201267239.20
bash37441641650.78
gnupg   34241984061.37
e2fsprogs   17762161666.86
tar 16802329672.06
shadow  16322492877.11
apt 15282645681.84
libapt-pkg4.12  10522750885.09
Linux-PAM   796 2830487.55
findutils   756 2906089.89
grep636 2969691.86
diffutils   620 3031693.78
debconf 596 3091295.62
adduser 444 3135696.99
sed 428 3178498.32
libgpg-error388 3217299.52
systemd 84  3225699.78
acl 72  32328   100.00

In Debian Stretch, I've already done this separation for e2fsprogs, so
the installed size of e2fsprogs is only 1309kB.  And so I've already
made harvested way more than half of the savings of getting rid of
e2fsprogs from the minbase set by the simple expedient of moving the
/usr/share/locale files to e2fsprogs-l10n.

Simply splitting coreutils into coreutils and coreutils-l10n would
reduce minbase by a factor of *six* over getting rid of e2fsprogs from
minbase.

Does this mean trying to get to Essential: no for e2fsprogs is a bad
thing?  No, but if your goal is reduce the size of minbase for Debian,
I just want to point out that there is **much** lower hanging fruit
that folks might want to consider harvesting first.

Cheers,

- Ted

P.S.  In case it isn't obvious, the reason why it's interesting to
shrink the size of minbase is that it makes Debian much lighter-weight
for Docker --- you don't need e2fsck or mke2fs in most docker
containers based on Docker; neither do you need the translations into
Turkish, German, Spanish, Chinese, etc., for e2fsprogs, coreutils,
dpkg, etc., for most Docker containers.

When I asked the question *why* is it worth it to spend the effort to
reduce the size of Essential: yes, I was told it was it was a
prerequisite of reducing the set of packages where Priority is
"required", and *that* was because it allowed for a reduction of the
minbase set, which in turn is intersting for reducing the size of
Docker images.  If there are other reasons why people want to make
e2fsprogs no longer essential: yes, it's good to have those out on the
table, since it may be that there are other things we can do first, or
in parallel, that might be *far* more effective towards the real end
goal that people have.

Re: e2fsprogs as Essential: yes?

[Helmut Grohne]

On Sun, Oct 01, 2017 at 12:45:39AM +0200, Helmut Grohne wrote:
> I think we should revisit this proposal now that it becomes practical.
> 
> To get us going, I have come up with a plan:
> 
> 1) Analyze which packages would need dependencies on e2fsprogs.

Done. See my previous mail.

> 2) File a bug against lintian to stop complaining about e2fsprogs
>dependencies.

#878518 is fixed in lintian/2.5.56.

> 3) MBF those packages that need an e2fsprogs dependency.

I intend to perform said MBF in one week and this mail serves as the MBF
announcement. I will use the following mail template:

--->8---
Subject: %(package)s should depend on e2fsprogs

Package: %(package)s
Version: %(version)s
User: helm...@debian.org
Usertags: nonessentiale2fsprogs

Dear maintainer,

We want to make removing e2fsprogs from installations possible. For standard
installations this is not useful, but embedded applications and chroots benefit
from such an option.  For getting there all packages that use e2fsprogs must be
identified and gain a dependency on it as e2fsprogs currently is essential.

%(package)s was identified as potentially needing such a dependency,
because it mentions tool names from e2fsprogs in the following files:

%(report)s

Please investigate whether these cases are actually uses of a tool from
e2fsprogs. Care has been taken to shrink the number of candidates as much as
possible, but a few false positives will remain. After doing so, do one of the
following:

 * Add e2fsprogs to Depends.
 * Add e2fsprogs to Recommends.
 * Close this bug explaining why e2fsprogs is not used by this package.

Once e2fsprogs drops the "Essential: yes" flag, this bug will be upgraded to RC
severity. Please note that lintian will warn about such a dependency before
lintian 2.5.56.

Thanks for your help

Helmut
---8<---

For a dd-list, affected packages and the report details, see my previous
mail. I will update the data using a fresh analysis run before
submitting bugs.

Please object now.

> 4) Drop Essential: yes from e2fsprogs.

Later, but I hope we can do it in time for buster.

Helmut

Re: e2fsprogs as Essential: yes?

[Felipe Sateler]

On Tue, 03 Oct 2017 22:51:08 +0200, Michael Biebl wrote:

> Am 03.10.2017 um 19:37 schrieb Felipe Sateler:
>> I don't think that's necessary as init is Important: yes, and both
>> systemd and sysvinit will have to gain a dep on e2fsprogs.
> 
> Why exactly? To fsck/mount ext234 file systems?
> Keep in mind that sysvinit/systemd don't depend on xfsprogs, etc either.

I was going by Helmut's dd-list. Both might be a false positive though. 
Further inspection of systemd sources suggests at least systemd is a 
false positive. I have not checked sysvinit.

I guess e2fsprogs should gain the Important: yes flag then.

-- 
Saludos,
Felipe Sateler

Re: e2fsprogs as Essential: yes?

[Michael Biebl]

Am 03.10.2017 um 19:37 schrieb Felipe Sateler:
> I don't think that's necessary as init is Important: yes, and both
> systemd and sysvinit will have to gain a dep on e2fsprogs.

Why exactly? To fsck/mount ext234 file systems?
Keep in mind that sysvinit/systemd don't depend on xfsprogs, etc either.



-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?



signature.asc
Description: OpenPGP digital signature

Re: e2fsprogs as Essential: yes?

[Felipe Sateler]

On Tue, Oct 3, 2017 at 5:31 AM, Andreas Henriksson  wrote:
> Hello Felipe, Helmut,
>
> On Mon, Oct 02, 2017 at 01:20:55PM +, Felipe Sateler wrote:
>> Hi,
>>
>> On Sun, 01 Oct 2017 00:45:39 +0200, Helmut Grohne wrote:
> [...]
>> Thanks for resuming this work.
>
> +1
>
>> > To get us going, I have come up with a plan:
> [...]
>> > 2) File a bug against lintian to stop complaining about e2fsprogs
>> >dependencies.
>>
>> +1
>
> For an example of a package (where I recently added e2fsprogs
> dependency) that currently triggers this lintian warning, see udisks2.
>
> https://lintian.debian.org/maintainer/pkg-utopia-maintain...@lists.alioth.debian.org.html#udisks2
>
>>
>> > 3) MBF those packages that need an e2fsprogs dependency.
>> > 4) Drop Essential: yes from e2fsprogs.
>>
>> As Adam mentioned, we will need to wait one release to drop the
>> Essential: yes bit :( . Alternatively, e2fsck would have to gain Breaks:
>> against all unfixed rdeps. For such a core package I think this might be
>> problematic for upgrades, but I haven't tested.
>
> I disagree.
>
> I don't see any practical problem with dropping it since the Priority
> field will still have it as part of any (normal) installation. Potentially
> something with a Conflicts/Breaks could motivate apt into attempting
> uninstalling it during upgrades, but I don't see anyone having reported
> such an issue so no need to assume the worst yet.

Currently the only negative dependency is initramfs-tools, which has a
Breaks against versions older than stable (but not oldstable).

I note that both systemd and sysvinit would have to gain a Depends, so
at least bootable systems should be fine. The intersection of
"non-bootable systems" and "systems that need e2fsprogs" should be
small. (Please enlighten me if I'm lacking in imagination)
Would backporting this dependency to stable be possible/feasible?

> If people really think the theoretical is so important a stop-gap
> solution could be to use (XB-)Important: yes. Maybe it should even
> be used permanently.

I don't think that's necessary as init is Important: yes, and both
systemd and sysvinit will have to gain a dep on e2fsprogs.

-- 

Saludos,
Felipe Sateler

Re: e2fsprogs as Essential: yes?

[Andreas Henriksson]

Hello Felipe, Helmut,

On Mon, Oct 02, 2017 at 01:20:55PM +, Felipe Sateler wrote:
> Hi,
> 
> On Sun, 01 Oct 2017 00:45:39 +0200, Helmut Grohne wrote:
[...]
> Thanks for resuming this work.

+1

> > To get us going, I have come up with a plan:
[...]
> > 2) File a bug against lintian to stop complaining about e2fsprogs
> >dependencies.
> 
> +1

For an example of a package (where I recently added e2fsprogs
dependency) that currently triggers this lintian warning, see udisks2.

https://lintian.debian.org/maintainer/pkg-utopia-maintain...@lists.alioth.debian.org.html#udisks2

> 
> > 3) MBF those packages that need an e2fsprogs dependency.
> > 4) Drop Essential: yes from e2fsprogs.
> 
> As Adam mentioned, we will need to wait one release to drop the 
> Essential: yes bit :( . Alternatively, e2fsck would have to gain Breaks: 
> against all unfixed rdeps. For such a core package I think this might be 
> problematic for upgrades, but I haven't tested.

I disagree.

I don't see any practical problem with dropping it since the Priority
field will still have it as part of any (normal) installation. Potentially
something with a Conflicts/Breaks could motivate apt into attempting
uninstalling it during upgrades, but I don't see anyone having reported
such an issue so no need to assume the worst yet.

If people really think the theoretical is so important a stop-gap
solution could be to use (XB-)Important: yes. Maybe it should even
be used permanently.

See the (new) fdisk package (src:util-linux) as an example.

> 
> 
> > So I thought, "how hard can it be?" [...]

Famous last words. ;)
Thanks again for enduring through to the final end.

Regards,
Andreas Henriksson

PS. I'd personally see it natural for e2fsprogs to be
Priority: important and (XB-)Important: yes. ie. Part of a normal (but
not minbase) install, plus safety guarded against deinstallation.

Re: e2fsprogs as Essential: yes?

[Andreas Henriksson]

On Mon, Oct 02, 2017 at 10:49:56AM +0200, Helmut Grohne wrote:
> On Sun, Oct 01, 2017 at 10:45:20PM +0200, Simon Richter wrote:
> > > lsattr, chattr
> > 
> > These I'd expect to be present in a sane system, including inside
> > containers.
> 
> I argue that a minbase debootstrap is not a sane system.
[...]

I'm going to be more explicit here, but basically saying the same as
Helmut:

"sane system" == Priority: important (or higher)

Anything with Priority >= important will be part of any (bare) default
installation, as performed by debian-installer. (Also by a normal
debootstrap, without custom arguments explicitly requesting minbase.)

"Minimal system" (== Priority: required) != "sane system"

Please note how Essential: yes is not part of any of the above
definitions!

In a system like Debian, we want dependency tracking. A package
with Essential: yes prevents that. (It's explicitly forbidden in policy.)
Thus packages should not use Essential: yes unless they have a very
good reason for it (and sometimes that reason is that because of
historical Essential: yes usage it's VERY hard to get rid of it).

Regards,
Andreas Henriksson

Re: e2fsprogs as Essential: yes?

[Ben Hutchings]

On Mon, 2017-10-02 at 13:20 +, Felipe Sateler wrote:
[...]
> for preinst/postrm and thus may be problematic:
> 
> 1. e2fsck-static (appears to be false positive)
> 2. lilo (uses kernel preinst hook)
> 3. blktrace (appears false positive)  
> 
> I don't know how the kernel hook works, is it problematic?
[...]

Kernel hook scripts are run by the corresponding maintainer scripts of
kernel packages (in all package implementations I know of).  The
postinst hook scripts are also run by 'make install' if the target is
/.

The lilo hack using chattr to prevent kernel images moving around on
xfs seems like a policy violation to me, though I can't find an
explicit ban on modifying files belonging to other packages (other than
configuration files).  It is, at least, opt-in.

(But this hack isn't complete, because lilo doesn't make-immutable the
kernel images that were installed before lilo, or to undo this if it is
removed.  So far as I can see, that would make all kernel packages un-
removable!)

Ben.

-- 
Ben Hutchings
Life is what happens to you while you're busy making other plans.
  - John Lennon



signature.asc
Description: This is a digitally signed message part

Re: e2fsprogs as Essential: yes?

[Felipe Sateler]

Hi,

On Sun, 01 Oct 2017 00:45:39 +0200, Helmut Grohne wrote:

> In 2011 Steve Langasek proposed dropping Essential: yes from e2fsprogs.
> 
> On Sat, Mar 26, 2011 at 11:47:08AM -0700, Steve Langasek wrote:
>> Currently the e2fsprogs package is marked Essential: yes in the
>> archive.  Is this a historical holdover?  I believe e2fsprogs used to
>> ship /sbin/fsck, but since 2009 (i.e., util-linux (>= 2.15~rc1-1),
>> which e2fsprogs has a pre-depends on), this has been provided by
>> util-linux instead.
>>
>> The remaining programs provided by e2fsprogs are all specific to the
>> ext* family of filesystems, so I don't think meet the definition of
>> Essential any longer - their presence is certainly important if you
>> have an ext[234] filesystem, but while this is the default, you can
>> have systems that don't use ext* at all, which makes e2fsprogs no more
>> essential in nature than the other per-filesystem fsck tools.
>>
>> Now that the transition to util-linux is done in a stable release, is
>> it time for us to drop the Essential: yes flag from e2fsprogs?  This
>> will benefit those targetting embedded systems that don't use ext,
>> where the package will be dead weight; the risk of any packages
>> assuming availability of these e2fs-specific interfaces without a
>> dependency is quite low; and we're at the right point in the cycle to
>> make changes to the Essential set, where we have time to deal with any
>> unexpected fallout.
> 
> Since then we have fully transitioned to systemd and init has become
> non-essential. The issue around pulling e2fsprogs into essential via
> logsave has thus solved itself.
> 
> I think we should revisit this proposal now that it becomes practical.

Thanks for resuming this work.

> 
> To get us going, I have come up with a plan:
> 
> 1) Analyze which packages would need dependencies on e2fsprogs.

>From what I gather in the files you attached, these packages are flagged 
for preinst/postrm and thus may be problematic:

1. e2fsck-static (appears to be false positive)
2. lilo (uses kernel preinst hook)
3. blktrace (appears false positive)

I don't know how the kernel hook works, is it problematic?

> 2) File a bug against lintian to stop complaining about e2fsprogs
>dependencies.

+1

> 3) MBF those packages that need an e2fsprogs dependency.
> 4) Drop Essential: yes from e2fsprogs.

As Adam mentioned, we will need to wait one release to drop the 
Essential: yes bit :( . Alternatively, e2fsck would have to gain Breaks: 
against all unfixed rdeps. For such a core package I think this might be 
problematic for upgrades, but I haven't tested.


> So I thought, "how hard can it be?" All we need to do is grep the
> archive for those tools and add those dependencies. So I unpacked sid
> main amd64 and grepped[1] each and every file (potentially decompressing
> gzip) for those e2fsprogs. The results[2] are 5666 occurrences in 1250
> binary packages. From there, I started looking[3] into the actual uses
> and filtered common non-uses such as documentation, debug symbols,
> kernel images, locales and other stuff. I manually checked the remaining
> packages and thus went down[4] to 318 occurrences in 133 binary
> packages. Thus I arrive at the final dd-list (attached) for an MBF. We
> can now say that our package lists will increase by less than 1.5kb
> uncompressed if we make e2fsprogs non-essential. In comparison, the
> average binary package weighs 767 bytes. I believe that the method used
> is mostly free from false negatives (by looking for bare program names
> in each and every file) and has a manageable number of false positives.
> 
> I think we can check off the analysis part. How about proceeding with
> the lintian bug and the MBF now? Did I miss anything?

I think this is OK. Thanks again for the work, and I think we should 
proceed. The earlier, the better!

-- 
Saludos,
Felipe Sateler

Re: e2fsprogs as Essential: yes?

[Adam Borowski]

On Mon, Oct 02, 2017 at 12:10:21PM +0100, Ian Campbell wrote:
> On Mon, 2017-10-02 at 06:57 -0400, Michael Stone wrote:
> > On Mon, Oct 02, 2017 at 11:50:37AM +0100, Ian Campbell wrote:
> > > On Mon, 2017-10-02 at 06:27 -0400, Michael Stone wrote:
> > > > On Mon, Oct 02, 2017 at 11:03:18AM +0200, Helmut Grohne wrote:
> > > > > This is a fair point, but I think the perfect is the enemy of
> > > > > the good.
> > > > > 
> > > > > I agree that moving badblocks, lsattr and chattr to another
> > > > > package or inside src:util-linux is something worth to consider. 
> > > > > Yet, it feels like a secondary thought to me.  It is not something
> > > > > I will drive.
> > > > 
> > > > It seems far easier to move them before explicitly dependencies are
> > > > added, rather than after, no?
> > > 
> > > Perhaps `Provides: ` would be an appropriate stop-gap?
> > 
> > I'm honestly not sure what you're suggesting would provide what.
> 
> The e2fsprogs package could "Provides: lsattr", then users can
> "Depends: lsattr" instead of "Depends: e2fsprogs".
> 
> At a later date if/when lsattr is split out into its own package then
> the dependencies do not need updating a second time.

Doing such changes twice would be a huge waste of time.  As it's not such a
time-pressing issue, why won't we do it right from the start?

But, we're discussing changes to e2fsprogs behind its maintainer's back.  I
believe he reads debian-devel, but, being nowhere like a frequent poster,
apparently doesn't watch new threads immediately as they appear (and this
one started as a response to a 2011 post).

Ted: could you please chime in?  In case you unsubscribed d-devel, it starts
at https://lists.debian.org/debian-devel/2017/09/msg00449.html

TL;DR: Helmut wants to make e2fsprogs non-essential, he has a plan to make
dependencies explicit, but one identified issue is that lsattr, chattr, and
less importantly badblocks, filefrag are useful on filesystems other than
ext*, thus it might be good to have them moved out of e2fsprogs.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ We domesticated dogs 36000 years ago; together we chased
⣾⠁⢰⠒⠀⣿⡁ animals, hung out and licked or scratched our private parts.
⢿⡄⠘⠷⠚⠋⠀ Cats domesticated us 9500 years ago, and immediately we got
⠈⠳⣄ agriculture, towns then cities. -- whitroth on /.

Re: e2fsprogs as Essential: yes?

[Ian Campbell]

On Mon, 2017-10-02 at 06:57 -0400, Michael Stone wrote:
> On Mon, Oct 02, 2017 at 11:50:37AM +0100, Ian Campbell wrote:
> > On Mon, 2017-10-02 at 06:27 -0400, Michael Stone wrote:
> > > On Mon, Oct 02, 2017 at 11:03:18AM +0200, Helmut Grohne wrote:
> > > > This is a fair point, but I think the perfect is the enemy of
> > > > the good.
> > > > 
> > > > I agree that moving badblocks, lsattr and chattr to another
> > > > package or
> > > > inside src:util-linux is something worth to consider. Yet, it
> > > > feels like
> > > > a secondary thought to me. It is not something I will drive.
> > > 
> > > It seems far easier to move them before explicitly dependencies
> > > are added,
> > > rather than after, no?
> > 
> > Perhaps `Provides: ` would be an appropriate stop-gap?
> 
> I'm honestly not sure what you're suggesting would provide what.

The e2fsprogs package could "Provides: lsattr", then users can
"Depends: lsattr" instead of "Depends: e2fsprogs".

At a later date if/when lsattr is split out into its own package then
the dependencies do not need updating a second time.

Ian.

Re: e2fsprogs as Essential: yes?

[Michael Stone]

On Mon, Oct 02, 2017 at 11:50:37AM +0100, Ian Campbell wrote:

On Mon, 2017-10-02 at 06:27 -0400, Michael Stone wrote:

On Mon, Oct 02, 2017 at 11:03:18AM +0200, Helmut Grohne wrote:
> This is a fair point, but I think the perfect is the enemy of the good.
>
> I agree that moving badblocks, lsattr and chattr to another package or
> inside src:util-linux is something worth to consider. Yet, it feels like
> a secondary thought to me. It is not something I will drive.

It seems far easier to move them before explicitly dependencies are added,
rather than after, no?


Perhaps `Provides: ` would be an appropriate stop-gap?


I'm honestly not sure what you're suggesting would provide what.

Mike Stone

Re: e2fsprogs as Essential: yes?

[Ian Campbell]

On Mon, 2017-10-02 at 06:27 -0400, Michael Stone wrote:
> On Mon, Oct 02, 2017 at 11:03:18AM +0200, Helmut Grohne wrote:
> > This is a fair point, but I think the perfect is the enemy of the good.
> > 
> > I agree that moving badblocks, lsattr and chattr to another package or
> > inside src:util-linux is something worth to consider. Yet, it feels like
> > a secondary thought to me. It is not something I will drive.
> 
> It seems far easier to move them before explicitly dependencies are added, 
> rather than after, no?

Perhaps `Provides: ` would be an appropriate stop-gap?

Ian.

Re: e2fsprogs as Essential: yes?

[Michael Stone]

On Mon, Oct 02, 2017 at 10:53:57AM +0800, Paul Wise wrote:

On Mon, Oct 2, 2017 at 5:01 AM, Michael Stone wrote:

If e2fsprogs goes non-essential I'd rather see a new package for the
filesystem-indpendent parts than have random packages depending on
"ext2/ext3/ext4 file system utilities" because they want chattr. (Side note:
if the fs-independent programs aren't spun off to a new package, the
description really should be updated to make it clear that there's stuff in
there that isn't specific to ext2/3/4.)


I wonder why these tools weren't put into util-linux/etc in the first
place, perhaps they should move to a less filesystem-specific source
package too?


Well, in the first place chattr/lsattr were filesystem specific--other 
filesystems adopted the ext2 interface over the years. badblocks was 
designed to generate a block list for mke2fs. (And really it's 
functionally obsolete with modern drives.)


Mike Stone

Re: e2fsprogs as Essential: yes?

[Michael Stone]

On Mon, Oct 02, 2017 at 11:03:18AM +0200, Helmut Grohne wrote:

This is a fair point, but I think the perfect is the enemy of the good.

I agree that moving badblocks, lsattr and chattr to another package or
inside src:util-linux is something worth to consider. Yet, it feels like
a secondary thought to me. It is not something I will drive.


It seems far easier to move them before explicitly dependencies are added, 
rather than after, no?


Mike Stone

Re: e2fsprogs as Essential: yes?

[Helmut Grohne]

On Sun, Oct 01, 2017 at 05:01:41PM -0400, Michael Stone wrote:
> If e2fsprogs goes non-essential I'd rather see a new package for the
> filesystem-indpendent parts than have random packages depending on
> "ext2/ext3/ext4 file system utilities" because they want chattr. (Side note:
> if the fs-independent programs aren't spun off to a new package, the
> description really should be updated to make it clear that there's stuff in
> there that isn't specific to ext2/3/4.)

This is a fair point, but I think the perfect is the enemy of the good.

I agree that moving badblocks, lsattr and chattr to another package or
inside src:util-linux is something worth to consider. Yet, it feels like
a secondary thought to me. It is not something I will drive.

Once e2fsprogs is non-essential, you'll know exactly which packages to
look at for finding users of these tools, so we can simply do in two
steps.

Look, we started considering optional e2fsprogs in 2011. Work is ongoing
for 6 years now. If we ever want to finish, we need to do it in small
steps rather than pulling more issues into it.

The question should be: "Do we face negative consequences in doing so?"
rather than "What else can we fix about e2fsprogs?".

Helmut

Re: e2fsprogs as Essential: yes?

[Helmut Grohne]

On Sun, Oct 01, 2017 at 10:45:20PM +0200, Simon Richter wrote:
> > lsattr, chattr
> 
> These I'd expect to be present in a sane system, including inside
> containers.

I argue that a minbase debootstrap is not a sane system. I guess you
also expect init and init is not part of a minbase. We're not talking
about removing "Priority: required" from e2fsprogs. The standard
debootstrap will install e2fsprogs as the majority of systems will need
e2fsprogs, yes. All I want is to make it opt-out.

If you are worried about whether these tools are available on "normal"
systems, come back when we talk about changing the Priority field.

Helmut

Re: e2fsprogs as Essential: yes?

[Paul Wise]

On Mon, Oct 2, 2017 at 5:01 AM, Michael Stone wrote:

> If e2fsprogs goes non-essential I'd rather see a new package for the
> filesystem-indpendent parts than have random packages depending on
> "ext2/ext3/ext4 file system utilities" because they want chattr. (Side note:
> if the fs-independent programs aren't spun off to a new package, the
> description really should be updated to make it clear that there's stuff in
> there that isn't specific to ext2/3/4.)

I wonder why these tools weren't put into util-linux/etc in the first
place, perhaps they should move to a less filesystem-specific source
package too?

-- 
bye,
pabs

https://wiki.debian.org/PaulWise

Re: e2fsprogs as Essential: yes?

[Adam Borowski]

On Sun, Oct 01, 2017 at 05:01:41PM -0400, Michael Stone wrote:
> On Sun, Oct 01, 2017 at 01:29:54AM +0200, Adam Borowski wrote:
> > At least the following are universal:
> > lsattr, chattr
> > badblocks
> > filefrag
> 
> If e2fsprogs goes non-essential I'd rather see a new package for the
> filesystem-indpendent parts than have random packages depending on
> "ext2/ext3/ext4 file system utilities" because they want chattr.

+1.  I guess this is a good point for the MBF Helmut wants to do.

shlib dependencies are:
* lsattr, chattr, badblocks:
  • e2fslibs
  • libcomerr2
  • libblkid1
  • libuuid1
* filefrag
  • libblkid1
  • libuuid1

Of these, libblkid1 and libuuid1 are transitively essential; libcomerr2
(87.0 kB inst) is very popular: no rdeps above priority:standard but you'll
have a hard time finding an useful system without one of those rdeps;
this leaves e2fslibs (466 kB inst) whose other users are mc and a handful
of less-used packages.

Current e2fsprogs is 1340 kB inst + libss2 (99.3 kB inst).


On the other hand, looking at unstable-capable physical machines I own or
admin, despite none of them using ext4 for actual system or data, every
single one has ext4 for u-boot, /boot or often sees SD cards from some SoC,
thus I personally would benefit only on containers.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ We domesticated dogs 36000 years ago; together we chased
⣾⠁⢰⠒⠀⣿⡁ animals, hung out and licked or scratched our private parts.
⢿⡄⠘⠷⠚⠋⠀ Cats domesticated us 9500 years ago, and immediately we got
⠈⠳⣄ agriculture, towns then cities. -- whitroth on /.

Re: e2fsprogs as Essential: yes?

[Michael Stone]

On Sun, Oct 01, 2017 at 01:29:54AM +0200, Adam Borowski wrote:

At least the following are universal:
lsattr, chattr
badblocks
/usr/sbin/filefrag (really shouldn't be in sbin)


If e2fsprogs goes non-essential I'd rather see a new package for the 
filesystem-indpendent parts than have random packages depending on 
"ext2/ext3/ext4 file system utilities" because they want chattr. 
(Side note: if the fs-independent programs aren't spun off to a new 
package, the description really should be updated to make it clear that 
there's stuff in there that isn't specific to ext2/3/4.)


Mike Stone

Re: e2fsprogs as Essential: yes?

[Simon Richter]

Hi,

On 01.10.2017 01:29, Adam Borowski wrote:

> lsattr, chattr

These I'd expect to be present in a sane system, including inside
containers.

> badblocks

I need that in rescue systems at least. Is d-i independent from
Essential here?

Container overhead is negligible, BTW — all Debian based containers
share a single copy of the Essential set.

   Simon

Re: e2fsprogs as Essential: yes?

[Adam Borowski]

On Sun, Oct 01, 2017 at 12:45:39AM +0200, Helmut Grohne wrote:
> In 2011 Steve Langasek proposed dropping Essential: yes from e2fsprogs.
> 
> On Sat, Mar 26, 2011 at 11:47:08AM -0700, Steve Langasek wrote:
> > Currently the e2fsprogs package is marked Essential: yes in the archive.  Is
> > this a historical holdover?  I believe e2fsprogs used to ship /sbin/fsck,
> > but since 2009 (i.e., util-linux (>= 2.15~rc1-1), which e2fsprogs has a
> > pre-depends on), this has been provided by util-linux instead.
> >
> > The remaining programs provided by e2fsprogs are all specific to the ext*
> > family of filesystems

At least the following are universal:
lsattr, chattr
badblocks
/usr/sbin/filefrag (really shouldn't be in sbin)

> > so I don't think meet the definition of Essential any
> > longer - their presence is certainly important if you have an ext[234]
> > filesystem, but while this is the default, you can have systems that don't
> > use ext* at all, which makes e2fsprogs no more essential in nature than the
> > other per-filesystem fsck tools.

I'm not sure if _host_ systems with space so low to care about getting rid
of e2fsprogs are real use cases (such extreme embedded are not going to run
unmodified Debian).  Containers on the other hand have little use for fsck
or badblocks, and only rarely for {ls,ch}attr or filefrag.  Thus, requiring
users of other filesystems to have e2fsprogs installed is not a big burden,
even if we'd still want the option to not have any such tools at all.

> I think we should revisit this proposal now that it becomes practical.
> 
> To get us going, I have come up with a plan:
> 
> 1) Analyze which packages would need dependencies on e2fsprogs.
> 2) File a bug against lintian to stop complaining about e2fsprogs
>dependencies.
> 3) MBF those packages that need an e2fsprogs dependency.
> 4) Drop Essential: yes from e2fsprogs.

Alas, this plan would break partial upgrades from stretch to buster.  Thus,
we'd need to do 3) in buster then 4) only in bullseye.

> So I thought, "how hard can it be?" All we need to do is grep the
> archive for those tools and add those dependencies.

> 318 occurrences in 133 binary packages.

> I think we can check off the analysis part. How about proceeding with
> the lintian bug and the MBF now? Did I miss anything?

Sounds good to me.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀ We domesticated dogs 36000 years ago; together we chased
⣾⠁⢰⠒⠀⣿⡁ animals, hung out and licked or scratched our private parts.
⢿⡄⠘⠷⠚⠋⠀ Cats domesticated us 9500 years ago, and immediately we got
⠈⠳⣄ agriculture, towns then cities. -- whitroth on /.

Re: e2fsprogs as Essential: yes?

[Helmut Grohne]

In 2011 Steve Langasek proposed dropping Essential: yes from e2fsprogs.

On Sat, Mar 26, 2011 at 11:47:08AM -0700, Steve Langasek wrote:
> Currently the e2fsprogs package is marked Essential: yes in the archive.  Is
> this a historical holdover?  I believe e2fsprogs used to ship /sbin/fsck,
> but since 2009 (i.e., util-linux (>= 2.15~rc1-1), which e2fsprogs has a
> pre-depends on), this has been provided by util-linux instead.
>
> The remaining programs provided by e2fsprogs are all specific to the ext*
> family of filesystems, so I don't think meet the definition of Essential any
> longer - their presence is certainly important if you have an ext[234]
> filesystem, but while this is the default, you can have systems that don't
> use ext* at all, which makes e2fsprogs no more essential in nature than the
> other per-filesystem fsck tools.
>
> Now that the transition to util-linux is done in a stable release, is it
> time for us to drop the Essential: yes flag from e2fsprogs?  This will
> benefit those targetting embedded systems that don't use ext, where the
> package will be dead weight; the risk of any packages assuming availability
> of these e2fs-specific interfaces without a dependency is quite low; and
> we're at the right point in the cycle to make changes to the Essential set,
> where we have time to deal with any unexpected fallout.

Since then we have fully transitioned to systemd and init has become
non-essential. The issue around pulling e2fsprogs into essential via
logsave has thus solved itself.

I think we should revisit this proposal now that it becomes practical.

To get us going, I have come up with a plan:

1) Analyze which packages would need dependencies on e2fsprogs.
2) File a bug against lintian to stop complaining about e2fsprogs
   dependencies.
3) MBF those packages that need an e2fsprogs dependency.
4) Drop Essential: yes from e2fsprogs.

So I thought, "how hard can it be?" All we need to do is grep the
archive for those tools and add those dependencies. So I unpacked sid
main amd64 and grepped[1] each and every file (potentially decompressing
gzip) for those e2fsprogs. The results[2] are 5666 occurrences in 1250
binary packages. From there, I started looking[3] into the actual uses
and filtered common non-uses such as documentation, debug symbols,
kernel images, locales and other stuff. I manually checked the remaining
packages and thus went down[4] to 318 occurrences in 133 binary
packages. Thus I arrive at the final dd-list (attached) for an MBF. We
can now say that our package lists will increase by less than 1.5kb
uncompressed if we make e2fsprogs non-essential. In comparison, the
average binary package weighs 767 bytes. I believe that the method used
is mostly free from false negatives (by looking for bare program names
in each and every file) and has a manageable number of false positives.

I think we can check off the analysis part. How about proceeding with
the lintian bug and the MBF now? Did I miss anything?

Again, we are at the right point in the cycle to handle unexpected
fallout. :)

Helmut

[1] Attached e2fsprogsimport.py + the software behind
https://dedup.debian.net/
[2] http://subdivi.de/~helmut/e2fsprogsimport.result.xz
[3] Attached e2fsprogsanalyze.py
[4] http://subdivi.de/~helmut/e2fsprogsanalyze.result.xz
#!/usr/bin/python3

import argparse
from contextlib import closing
import logging
import multiprocessing
try:
import queue
except ImportError:
import Queue as queue
import sys
try:
from urllib.request import urlopen
except ImportError:
from urllib2 import urlopen
import yaml
import zlib

from debian import deb822
from debian.debian_support import version_compare

from dedup.compression import GzipDecompressor
from dedup.debpkg import DebExtractor, decodetarname
from dedup.filemagic import FileDigester
from dedup.hashing import DecompressedHash, hash_file, SuppressingHash
from dedup.utils import open_compressed_mirror_url

class E2fsprogsAnalyzer:
"""Search for words in a stream of bytes.
@ivar found: is a set of discovered words
"""
def __init__(self, words):
"""
@param words: an iterable object yielding bytes
"""
self.found = set()
self.missing = set(words)
self.overlap = max(map(len, words)) - 1
self.buff = b""

def update(self, data):
self.buff += data
for word in tuple(self.missing):
if word in self.buff:
self.found.add(word)
self.missing.discard(word)
self.buff = self.buff[-self.overlap:]

def hexdigest(self):
return self.found

def copy(self):
new = E2fsprogsAnalyzer(self.missing)
new.found = self.found.copy()
new.overlap = self.overlap
new.buff = self.buff
return new

def make_tuple(*args):
return args

class MultiDigest(object):
def __init__(self, *digesters, postproc=make_tuple):
self.digesters = digesters

Re: e2fsprogs as Essential: yes?

[Mark Hymers]

On Sat, 26, Mar, 2011 at 11:47:08AM -0700, Steve Langasek spoke thus..
 Hi folks,
 
 Currently the e2fsprogs package is marked Essential: yes in the archive.  Is
 this a historical holdover?  I believe e2fsprogs used to ship /sbin/fsck,
 but since 2009 (i.e., util-linux (= 2.15~rc1-1), which e2fsprogs has a
 pre-depends on), this has been provided by util-linux instead.

Hi,

The only other thing I can see is that e2fsprogs contains lsattr and
chattr - a quick grep through my local /var/lib/dpkg/info shows that
chattr is used in the postfix postinst without an explicit dependency.
I wonder if there are more instances of that?

Mark

-- 
Mark Hymers mhy at debian dot org

But Yossarian *still* didn't understand either how Milo could buy eggs
 in Malta for seven cents apiece and sell them at a profit in Pianosa
 for five cents.
 Catch 22, Joseph Heller


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110326224209.ga11...@hymers.org.uk

Re: e2fsprogs as Essential: yes?

[Ted Ts'o]

On Sat, Mar 26, 2011 at 10:42:09PM +, Mark Hymers wrote:
 
 The only other thing I can see is that e2fsprogs contains lsattr and
 chattr - a quick grep through my local /var/lib/dpkg/info shows that
 chattr is used in the postfix postinst without an explicit dependency.
 I wonder if there are more instances of that?

What to do with lsattr and chattr is actually a somewhat interesting
question.  They are most definitely ext2/3/4 specific, and the
ext2/3/4 development team still adds new flags to it from time to
time, so we had no plans moving those two commands out of e2fsprogs
any time soon.  On the other hand, other file systems, including
reiserfs and btrfs, have used the same ioctl and command line
interface, and we do coordinate flags to prevent conflicts.  So other
users of other file systems will still need lsattr and chattr.

There are similar, although less serious, issues with filefrag -v
(which will work on other file systems), but which also has some
ext2/3/4 specific code it in.

Another binary which is used by other packages includes the logsave
utility, which is also in e2fsprogs, and which is used by
/etc/init.d/checkfs.sh and /etc/init.d/checkroot.sh in the initscripts
package.

Regards,

- Ted


-- 
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/20110326230729.gg2...@thunk.org