Re: Illegal Instruction Using sudo in Bookworm on i686

[Paul Gevers]

Hi,

On 09-06-2024 1:56 p.m., rhys wrote:

So given that these no longer fit the "old and busted" description, is Debian 
going to stick with the decision to not support them?  Or is Debian going to continue to 
support this processor, since it is still apparently a viable product, enough that new 
systems are using it?


https://lists.debian.org/debian-devel-announce/2023/12/msg3.html 
still stands. Debian basically isn't going to support any i386 CPU 
anymore in the sense that the Release Team gave the teams the wildcard 
to drop kernel, installer and images support for i386 (I don't have the 
details of the current status).


"""
Following that, there are two routes into running i386:

1. as a multi-arch option on an otherwise amd64 system
2. as an i386 chroot on another architecture system
"""

That's what we intend to do as Debian. If you want an Debian 
architecture to keep Debian running on i386 CPU's, you can create a 
port. But unless you find enough volunteers, I'm a bit skeptical you'll 
succeed. There are enough problems to cover, of which upstream kernel 
support and the 32 bit memory build problem are on the top of my mind.


Paul


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Seeking consensus on file conflict between python3-proto-plus and nanopb

[Simon Richter]

Hi,

On 6/11/24 07:40, Yogeswaran Umasankar wrote:


It appears that nanopb's use of the module name "proto" does not align
with the conventional identification of a Python module. Given this, it
might be plausible to make this module private within the nanopb
package. This adjustment could potentially resolve the conflict without
affecting the dependent packages.


From a quick glance at the package, not completely. It seems the 
"proto" module is used by the generator only, which is called as a 
plugin from Google protoc, so I'm not sure it can be taken fully private.


I think it should be possible to rename this module though with little 
adverse effect.


The standard invocation certainly is convoluted[1], and given that CMake 
is usually not the first choice for cross compiling in embedded 
environments, I also expect that there will be a lot of users rolling 
their own -- but I don't see the name of the module used on that command 
line, only the name of the plugin.


The script tries to do a relative import and provides fallback code if 
that is not yet supported in the Python interpreter (which it is since 
2007), so that might explain why the module is not up to current 
conventions.


   Simon

[1] https://github.com/nanopb/nanopb/blob/master/extra/FindNanopb.cmake

Re: Seeking consensus on file conflict between python3-proto-plus and nanopb

[Helmut Grohne]

Hi Laszlo and Yogeswaran,

I'm explicitly adding Laszlo to Cc to increase the chances of him
chiming in.

On Mon, Jun 10, 2024 at 06:40:02PM -0400, Yogeswaran Umasankar wrote:
> There is a file conflict between python3-proto-plus and nanopb. The
> conflict arises due to both packages has a file at
> /usr/lib/python3/dist-packages/proto/__init__.py [0]. I am maintaining
> python3-proto-plus, and I am seeking guidance.

Thank you for going the extra mile and resolving this constructively and
consensually.

> The module name "proto" is an integral part of the python3-proto-plus
> package. Renaming the "proto" module in python3-proto-plus would
> significantly impact future dependent packages.

I agree with this assessment. At the same time, I note that "proto" is a
fairly generic name. Even though it seems unlikely that upstream would
want to change it, I think telling them would be useful still. It
definitely is conceivable that another project would later try to also
use this module name and therefore it is best to avoid it.

> It appears that nanopb's use of the module name "proto" does not align
> with the conventional identification of a Python module. Given this, it
> might be plausible to make this module private within the nanopb
> package. This adjustment could potentially resolve the conflict without
> affecting the dependent packages.

Yes. In particular, I could not locate external uses of nanopb's proto
module. chromium and firefox also use this module name though my
impression is that they have another conflict on this name again arguing
in favour of not using it at all.

> I have attempted to reach out to the nanopb maintainer to discuss this
> issue, but I have not yet received a response. In case the maintainer is
> MIA, should I proceed with renaming the "proto" module in nanopb to
> "nanopb-proto"? As one of the team members, I am willing to implement
> this change if it is deemed the best solution.

I recommend that you send a patch to the bug and give Laszlo two weeks
before proceeding to NMU in the absence of a reply as the proposed
change is a bit intrusive.

Helmut

Re: Enabling some -Werror=format* by default?

[Guillem Jover]

Hi!

On Mon, 2024-06-10 at 18:01:58 +0200, Helmut Grohne wrote:
> Ideally, we'd not just do a rebuild with the flags, but also do a
> rebuild without and then compare the binary .debs. In the event that we
> misguide configure, we expect the .debs to differ and otherwise to equal
> due to the work of the reproducible builds folks. That equality has a
> really annoying difference in practice though: Build ids are dependent
> on the compiler flags, so the comparison would have to magically ignore
> changes in build id and this is where things become quite difficult.

A quick check seems to indicate warning flags do not get recorded in
the object files, which make sense as these are not supposed to change
the emitted objects.

(This can be seen with «dwarfdump .../*.debug | grep DW_AT_producer»
on the debug files from the -dbgsym packages.)

Thanks,
Guillem

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Simon Richter]

Hi,

On 6/11/24 00:26, Simon McVittie wrote:


Reproducibility outside of sterile environments is however a problem for us
as a distribution, because it affects how well people are able to contribute
to packages they are not directly maintaining



If our package-building entry point sets up aspects of its desired
normalized (or "sterile") environment itself, surely it's equally easy
for those contributors to build every package in that way, whether they
maintain this particular package or not?


Yes, but building the package is not the hard part in making a useful 
contribution -- anything but trivial changes will need iterative 
modifications and testing, and the package building entrypoint is 
limited to "clean and build entire package" and "build package without 
cleaning first", with the latter being untested and likely broken for a 
lot of packages -- both meson and cmake utterly dislike being asked to 
configure an existing build directory as if it were new.


For my own packages, I roughly know how far I can deviate from the clean 
environment and still get meaningful test results, but for anything 
else, I will still need to deep-dive into the build system to get 
something that is capable of incremental builds.



if my package is not required to work outside
of a very controlled environment, that is also an impediment to
co-maintenance



I'm not sure that follows. If the only thing we require is that it works
in a controlled environment, and the controlled environment is documented
and easy to achieve, then surely every prospective co-maintainer is in
an equally good position to be contributing? That seems better, to me, than
having unwritten rules about what environment is "close enough" and what
environment doesn't actually work.


I will need to deviate from the clean environment, because the clean 
environment does not have vim installed. Someone else might need to 
deviate further and have a graphical environment and a lot of dbus 
services available because their preferred editor requires it.


Adding a global expectation about the environment that a package build 
can rely on *creates* an unwritten per-package rule whether it is 
permissible to deviate from this expectation during development.


I expect that pretty much no one uses the C.UTF-8 locale for their 
normal login session, so adding this as a requirement to the build 
environment creates a pretty onerous rule: "if you want to test your 
changes, you need to remember to call make/ninja with LC_ALL=C.UTF-8."


Of course we know that this rule is bullshit, because the majority of 
packages will build and test fine without it, but this knowledge is 
precisely one of the "unwritten rules" that we're trying to avoid here.



We already do expect maintainers to be building in a specified
environment: Debian unstable (not stable, and not Ubuntu, for example).


I develop mostly on Debian or Devuan stable, then do a pbuilder build 
right before upload to make sure it also builds in a clean unstable 
environment. The original requirement was mostly about uploading binary 
packages, which we (almost) don't do anymore.



(I also do agree that it is an anti-pattern if we have a specified
environment where tests or QA will be run, and serious consequences for
failures in that environment, without it being reasonably straightforward
for contributors to repeat the testing in a sufficiently similar
controlled environment that they have a decent chance at being able to
reproduce the failure. But, again, that isn't this thread.)


This is largely what I think is this thread -- narrowing the environment 
where builds, tests and QA will be run, and narrowing what will be 
considered a bug.



Indeed -- however this class of bugs has already been solved because
reproducible-builds.org have filed bugs wherever this happened, and
maintainers have added workarounds where it was impossible to fix.



Someone (actually, quite a lot of someones) had to do that testing,
and those fixes or workarounds. Who did it benefit, and would they have
received the same benefit if we had said "building in a locale other than
C.UTF-8 is unsupported", or in some cases "building in a non-UTF-8 locale
is unsupported", and made it straightforward to build in the supported
locales?


I'd say that developers who don't have English as their first language 
have directly benefited from this, and would not have benefited if it 
was not seen as a problem if a package didn't build on their machines 
without the use of a controlled environment.


I also think that we have indirectly benefited from better test coverage.


Turning this workaround into boilerplate code was a mistake already, so the
answer to the complaint about having to copy boilerplate code that should be
moved into the framework is "do not copy boilerplate code."



If you don't want package-specific code to be responsible for forcing
a "reasonable" locale where necessary, then what layer do you want to
be 

Re: Seeking consensus on file conflict between python3-proto-plus and nanopb

[Andrey Rakhmatullin]

On Mon, Jun 10, 2024 at 06:40:02PM -0400, Yogeswaran Umasankar wrote:
> There is a file conflict between python3-proto-plus and nanopb. The
> conflict arises due to both packages has a file at
> /usr/lib/python3/dist-packages/proto/__init__.py [0]. I am maintaining
> python3-proto-plus, and I am seeking guidance.
> 
> The module name "proto" is an integral part of the python3-proto-plus
> package. Renaming the "proto" module in python3-proto-plus would
> significantly impact future dependent packages.
> 
> It appears that nanopb's use of the module name "proto" does not align
> with the conventional identification of a Python module. 

What do you mean? It does look like a Python module to me. It's very
likely that it should be a private one though, but I see you already asked
the maintainer about that.

> I have attempted to reach out to the nanopb maintainer to discuss this
> issue, but I have not yet received a response. In case the maintainer is
> MIA, should I proceed with renaming the "proto" module in nanopb to
> "nanopb-proto"? 

No, it will make it useless.
I don't see a good way forward that doesn't involve NMUing nanopb.


-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Enabling some -Werror=format* by default?

[Fay Stegerman]

* Helmut Grohne  [2024-06-10 18:01]:
> Ideally, we'd not just do a rebuild with the flags, but also do a
> rebuild without and then compare the binary .debs. In the event that we
> misguide configure, we expect the .debs to differ and otherwise to equal
> due to the work of the reproducible builds folks. That equality has a
> really annoying difference in practice though: Build ids are dependent
> on the compiler flags, so the comparison would have to magically ignore
> changes in build id and this is where things become quite difficult.

Indeed build IDs can be a very annoying source of unreproducible builds (though
I mostly encounter those working with Android toolchains, not Debian packages).

But build IDs can be disabled via appropriate linker flags (e.g.
-Wl,--build-id=none).  Is there some reason simply doing that for both rebuilds
is not an option?

- Fay

Re: Enabling some -Werror=format* by default?

[Helmut Grohne]

On Mon, Jun 10, 2024 at 04:06:13PM +0500, Andrey Rakhmatullin wrote:
> Do you think it makes sense to add this a flag that enables -Werror=format
> to dpkg-buildflags(1), before, or after a test rebuild, before, or after
> the MBF if we do one?

I think that a test rebuild and the MBF are reasonable preconditions to
extend the default build flags (and with default I mean changing
hardening=+all).

As multiple people pointed out, the effects of the flags are hard to
predict and they can even cause misbuilds (via failing configure
checks), so these flags do have a non-trivial cost (and benefits).

Ideally, we'd not just do a rebuild with the flags, but also do a
rebuild without and then compare the binary .debs. In the event that we
misguide configure, we expect the .debs to differ and otherwise to equal
due to the work of the reproducible builds folks. That equality has a
really annoying difference in practice though: Build ids are dependent
on the compiler flags, so the comparison would have to magically ignore
changes in build id and this is where things become quite difficult.

> Another related question: if not via dpkg-buildflags, how do we do
> rebuilds with changed default flags?

If you export DEB_CFLAGS_APPEND=-Werror=format=2 and
DEB_CXXFLAGS_APPEND=-Werror=format=2 (not to be confused with
DEB_*_MAINT_APPEND which is often set in d/rules), you should get most
packages to pick up these flags.

Possibly debusine.debian.net can be used to perform such a rebuild
rather than using your own resources. Steering it to do this is a
non-trivial task at present, but I my impression is that you will
receive support in doing so and it can do native armhf builds
(eliminating the need for cross builds). Your mileage will vary.

In any case, my impression is that the first step towards changing
hardening flags is actually performing test builds in whatever form.

Helmut

Re: Enabling some -Werror=format* by default?

[Guillem Jover]

Hi!

On Mon, 2024-06-10 at 20:09:21 +0500, Andrey Rakhmatullin wrote:
> On Mon, Jun 10, 2024 at 04:24:54PM +0200, Guillem Jover wrote:
> > > Do you think it makes sense to add this a flag that enables -Werror=format
> > > to dpkg-buildflags(1), before, or after a test rebuild, before, or after
> > > the MBF if we do one?
> > 
> > If by adding you mean adding a new feature flag that is disabled by
> > default, then depending on the feature area, that might unfortunately
> > be equivalent to enable it by default (due to the «all» keyword).
> 
> Another related question: if not via dpkg-buildflags, how do we do
> rebuilds with changed default flags?

Ah, you can still use its mechanism (just not its policy) where a
rebuild should be able to use DEB__APPEND for that, or set
them in one of its config files.

If having a modified dpkg-dev emitting those by default would be less
cumbersome for the rebuilder, I'm happy to provide an out of archive
package for that for example, which I think we have done for some
rebuilds.

Thanks,
Guillem

Re: Suggestions about i386 support

[Simon McVittie]

On Mon, 10 Jun 2024 at 20:17:27 +0500, Andrey Rakhmatullin wrote:
> On Mon, Jun 10, 2024 at 04:02:54PM +0100, Simon McVittie wrote:
> > several important upstreams no longer consider i386 to be useful (and
> > especially i386-without-SSE2), so so the burden of supporting 32-bit
> > CPUs in modern software will fall on the downstream developers who have
> > chosen that their aim is to support 32-bit CPUs.
> 
> I assume such software already has this status on Debian i386 (and so is
> either not built there or supported only by the maintainer, or maybe built
> with a raised baseline) so there should be no regression here

Some concrete examples:

* Packages built with -fcf-protection (Intel CET) require CPU support
  for "long NOP" opcodes which are, or used to be, non-baseline.
  This is a security hardening measure, so if we disable it in order
  to respect the documented baseline, the result is that our binaries
  are less secure than they could be (fewer mitigations for exploits)
  on CPUs where those opcodes actually *are* supported. (That's what
  started this thread: sudo enables this hardening.)

* nodeJS, Qt5WebEngine and others have a known baseline violation by using
  and requiring SSE2 internally, which they document by depending on
  sse2-support. I thought Chromium did this too, but maybe it either
  doesn't do that any more, or still has the baseline violation but
  doesn't document it.

* Packages built with rustc (notably Firefox and librsvg) had a known
  baseline violation in the past (they would crash with an illegal
  instruction on i586) which was made officially not-a-bug by raising the
  baseline to i686.

* In packages with test suites that compare the results of floating-point
  calculations at a high precision (such as librsvg), the fact that we
  target the legacy i387 FPU (which has 80-bit excess precision) rather
  than SSE2 (which has 64-bit IEEE precision) has caused a lot of extra
  work for maintainers in the past, due to tests getting a result on i386
  that differs from the result on every other platform we support.

* Some third-party software like Steam does not follow our baselines,
  and unconditionally requires newer CPU features. (Not a concern for
  Steam on i386 any more, because Steam now requires a 64-bit CPU, so
  bookworm's steam-installer is intentionally not installable on purely
  32-bit systems - but Steam will not actually work on a baseline *64-bit*
  CPU any more.)

smcv

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Simon McVittie]

On Sat, 08 Jun 2024 at 02:14:36 +0900, Simon Richter wrote:
> Reproducibility outside of sterile environments is however a problem for us
> as a distribution, because it affects how well people are able to contribute
> to packages they are not directly maintaining

If our package-building entry point sets up aspects of its desired
normalized (or "sterile") environment itself, surely it's equally easy
for those contributors to build every package in that way, whether they
maintain this particular package or not?

> if my package is not required to work outside
> of a very controlled environment, that is also an impediment to
> co-maintenance

I'm not sure that follows. If the only thing we require is that it works
in a controlled environment, and the controlled environment is documented
and easy to achieve, then surely every prospective co-maintainer is in
an equally good position to be contributing? That seems better, to me, than
having unwritten rules about what environment is "close enough" and what
environment doesn't actually work.

If I want to contribute to (let's say) both GNOME and KDE, but the GNOME
team expects me to be building in one controlled environment, and the KDE
team expects me to be building in a *different* controlled environment,
then sure, that would be a barrier to contribution: I'd have to do that
setup once per team, and maybe they'd be mutually incompatible. But that
isn't going to be the case if we're setting a policy for the whole distro,
which only needs to happen once?

We already do expect maintainers to be building in a specified
environment: Debian unstable (not stable, and not Ubuntu, for example).

I can see that if our policy was something like "must build in a schroot",
then that would be making us vulnerable to a lock-in where we can't
move to Podman or systemd-nspawn or Incus or whatever is the flavour of
the month because our policy says we use schroot, and then we end up
shackled to schroot's particular properties and limitations. (Indeed,
to an extent, we already have that problem by using schroot on official
buildds, and as a result being unable to gain much benefit from work
done on container technologies outside the Debian bubble.)

But that's not what was proposed by this thread: this thread is about
locales. Now that glibc has C.UTF-8 built-in and non-optional, you can
set a normalized or sterile locale regardless of whether you're building
on bare metal, in a VM, in a schroot, in Docker, or whatever, and it's
very easy to do that in a tool (or even an interactive shell) and have
it inherit down through the build? So I'm not sure I see the problem?

If you're making a wider point about use of containers etc. that is
orthogonal to setting the locale, then that would be a valid objection
to someone saying "we should standardize on building in Docker" (and I
would make a similar objection myself), but that's not this thread.

(I also do agree that it is an anti-pattern if we have a specified
environment where tests or QA will be run, and serious consequences for
failures in that environment, without it being reasonably straightforward
for contributors to repeat the testing in a sufficiently similar
controlled environment that they have a decent chance at being able to
reproduce the failure. But, again, that isn't this thread.)

> a lot of the debates we've had in the past years is who gets to
> decide what is in scope

Yes, that's always going to be the case for a community that doesn't
have an authority figure telling us "the scope is what I say it is". We
have debates when we don't all agree, and the scope of our collective
project is one of the foundations for all the other decisions we make,
so it's certainly something that we can't expect to be unanimous. (Insert
wise words from Russ Allbery about the difference between unanimity and
consensus here...)

I hope we can come close enough to a consensus that we're all generally
willing to accept it, though, even if that means sometimes accepting a
narrower or wider scope than I would personally prefer.

> > What Giole proposed at the beginning of this thread can be rephrased as
> > declaring that "FTBFS when locale is not C.UTF-8" and "non-reproducible
> > when locale is varied" are non-bugs, and therefore they are not only
> > wontfix, but they should be closed altogether as being out-of-scope.
> 
> Indeed -- however this class of bugs has already been solved because
> reproducible-builds.org have filed bugs wherever this happened, and
> maintainers have added workarounds where it was impossible to fix.

Someone (actually, quite a lot of someones) had to do that testing,
and those fixes or workarounds. Who did it benefit, and would they have
received the same benefit if we had said "building in a locale other than
C.UTF-8 is unsupported", or in some cases "building in a non-UTF-8 locale
is unsupported", and made it straightforward to build in the supported
locales?

I think there is a danger that we sink time 

Re: Suggestions about i386 support

[Andrey Rakhmatullin]

On Mon, Jun 10, 2024 at 04:02:54PM +0100, Simon McVittie wrote:
> On Mon, 10 Jun 2024 at 12:43:27 +, Stefano Rivera wrote:
> > The point here is that the Debian project is not intending to support
> > new hardware on the i386 architecture. The architecture is being kept
> > around primarily to support running old i386 binaries.
> 
> ... and the most appropriate answers to some technical questions are not
> going to be the same for "i386 to run legacy 32-bit binaries on 64-bit
> CPUs" and "i386 to run on 32-bit CPUs", so we cannot simply support
> both equally.

Yeah, it should be made clear that if some people want to bring back
proper support for i386 hardware, they will need to make a new port.
Which is of course more complicated than fixing an existing one (but at
least bootstrapping it should be easier than bootstrapping some non-x86
port).

> If people want a distribution to run on 32-bit x86 CPUs badly enough,
> one possible route would be to start a new port (perhaps called ia32,
> i386t64 or something similar) for that use-case; it would have a baseline
> that is as low as its maintainers want it to be (i586?), and a 64-bit
> time_t for post-2038 future-proofing.
> 
> As far as I'm aware, nobody is yet putting effort into doing this. Also,
> several important upstreams no longer consider i386 to be useful (and
> especially i386-without-SSE2), so so the burden of supporting 32-bit
> CPUs in modern software will fall on the downstream developers who have
> chosen that their aim is to support 32-bit CPUs.

I assume such software already has this status on Debian i386 (and so is
either not built there or supported only by the maintainer, or maybe built
with a raised baseline) so there should be no regression here, though
additional packages will get the same status in the future.
Similarly, we already don't build a noticeable number of packages on armel
(and some of them also on armhf) when we build them on arm64.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Enabling some -Werror=format* by default?

[Andrey Rakhmatullin]

On Mon, Jun 10, 2024 at 04:24:54PM +0200, Guillem Jover wrote:
> > Do you think it makes sense to add this a flag that enables -Werror=format
> > to dpkg-buildflags(1), before, or after a test rebuild, before, or after
> > the MBF if we do one?
> 
> If by adding you mean adding a new feature flag that is disabled by
> default, then depending on the feature area, that might unfortunately
> be equivalent to enable it by default (due to the «all» keyword).

Another related question: if not via dpkg-buildflags, how do we do
rebuilds with changed default flags?

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Enabling some -Werror=format* by default?

[Simon McVittie]

On Mon, 10 Jun 2024 at 16:24:54 +0200, Guillem Jover wrote:
> In general I think it's good (in principle) to enable -Werror flags
> that detect actual bugs in code, the problem is always going to be
> the amount of fallout and work that creates

Yes. The difficult thing here is balancing "detect actual (important)
bugs in code" against "detect non-bugs or unimportant bugs, and make
them be treated as if they were important".

Unfortunately, because compiler warnings are mostly heuristics involving
a reasonable guess about what the developer had intended to happen, most
compiler warnings do a mixture of both.

smcv

Re: Suggestions about i386 support

[Simon McVittie]

On Mon, 10 Jun 2024 at 12:43:27 +, Stefano Rivera wrote:
> The point here is that the Debian project is not intending to support
> new hardware on the i386 architecture. The architecture is being kept
> around primarily to support running old i386 binaries.

... and the most appropriate answers to some technical questions are not
going to be the same for "i386 to run legacy 32-bit binaries on 64-bit
CPUs" and "i386 to run on 32-bit CPUs", so we cannot simply support
both equally.

> We didn't bring 64bit time_t to the architecture, because of this goal.

This is a good example of a decision that goes differently for those two
purposes. If we want to run legacy i386 binaries, 64-bit time_t would
be counterproductive, because it will break some of those binaries *now*
(not just in 2038).

Raising the baseline (to i686 or beyond) is another decision that would
go differently for those two purposes. For legacy i386 binaries on an
x86_64 CPU, the baseline could raise as far as i686 + SSE + SSE2 without
losing any 64-bit CPU support, because SSE and SSE2 are part of the
x86_64 baseline; but for genuinely 32-bit CPUs, as discussed in this
thread and elsewhere, even i686 is sometimes too high.

> There isn't a team working on a modern 32 bit x86 port. We're just
> trying to keep the old one going for as long as we can. You're welcome
> to try to form such a team, of course :)

The "i386" name is widely hard-coded in third-party software (e.g. Steam)
as being the appropriate architecture to use to run legacy 32-bit
binaries on a 64-bit CPU, so if we were to fork i386 into two separate
ports (one for legacy binaries on 64-bit CPUs, and one for 32-bit CPUs),
the route that would avoid breaking backwards-compatibility would be for
the i386 name to stay with the port that is intended for legacy binaries,
introducing a new name for the port that is intended for 32-bit CPUs.

(This is because by definition legacy binaries are legacy, and are not
going to change to accommodate new decisions, whereas a new port can
produce new binaries with code changes if enough effort is put into
doing so.)

If people want a distribution to run on 32-bit x86 CPUs badly enough,
one possible route would be to start a new port (perhaps called ia32,
i386t64 or something similar) for that use-case; it would have a baseline
that is as low as its maintainers want it to be (i586?), and a 64-bit
time_t for post-2038 future-proofing.

As far as I'm aware, nobody is yet putting effort into doing this. Also,
several important upstreams no longer consider i386 to be useful (and
especially i386-without-SSE2), so so the burden of supporting 32-bit
CPUs in modern software will fall on the downstream developers who have
chosen that their aim is to support 32-bit CPUs.

And, for several larger packages it is becoming a serious problem
to compile and link the software natively on any 32-bit architecture,
because the working set for the compiler or linker is larger than 4 GiB,
but a 32-bit architecture can't possibly address more than 4 GiB of
virtual memory at a time.

Those are problems that any would-be port maintainer for a "modern"
32-bit x86 port would have to deal with somehow, and telling the wider
Debian project "this is important to me, therefore it should be important
to you" is unlikely to result in those problems going away.

Alternatively...

> The cost of supporting a port of Debian is far more than the cost of the
> machines needed to build it. Never mind the cost of 1 user's machine.

As a point of comparison, a used 64-bit laptop of a well-respected
design from 12-13 years ago (Lenovo X220) seems to be readily available
for around £60 on eBay UK, and that price is enough to pay a software
consultant in the UK for somewhere around 1 hour. Of course, volunteer
effort is not directly interchangeable with consultant effort and not
every country is the UK, but the time-cost of maintaining a 32-bit port
of Debian is going to be lots of hours.

I suspect that many of the used 64-bit laptops of that age on eBay
go unsold and will instead be discarded, entering the e-waste stream,
in which case buying one as a replacement for a 32-bit machine is not
a net increase in e-waste.

(No endorsement of eBay intended, other sources of refurbished computers
are available.)

smcv

Re: Enabling some -Werror=format* by default?

[Andrey Rakhmatullin]

On Mon, Jun 10, 2024 at 04:24:54PM +0200, Guillem Jover wrote:
> > > > We recently increased the time_t size on certain architectures and some
> > > > packages started failing to build because they were using a format
> > > > specifier too narrow for the wider time_t, e.g. #1067829.
> > > > But the only reason those package FTBFS is they enable either -Werror or
> > > > some more specific non-default switch like -Werror=format=2, so I 
> > > > suspect
> > > > much more packages contain similar code but gained only a warning. Isn't
> > > > this a bad thing? Should we enable at least some combination of 
> > > > -Wformat*
> > > > switches by default? Should we at least add a new flag to 
> > > > dpkg-buildflags
> > > > and do some test rebuilds with it enabled?
> > > 
> > > It wasn't quite clear to me what -Werror=format=2 actually means.
> > > According to the gcc documentation[1], -Wformat=2 currently means:
> > > 
> > > -Wformat -Wformat-nonliteral -Wformat-security -Wformat-y2k.
> 
> Those are in addition to the ones enabled by -Wformat=1, which are:

(note that -Wformat is already included in that list)

>   -Wnonnull -Wno-format-contains-nul -Wno-format-extra-args
>   -Wno-format-zero-length

(note that -Wformat is not equal to these flags, it just implies -Wnonnull
and those three -Wno-* can be used to turn off some parts of it)


> In general I think it's good (in principle) to enable -Werror flags
> that detect actual bugs in code, the problem is always going to be
> the amount of fallout and work that creates, and whether there's
> consensus about that work commitment being acceptable to maintainers
> in Debian, or whether that can interfere with transitions going on,
> etc.

Yeah.

> > > It also is unclear how this impacts the archive and yes, I'd recommend a
> > > rebuild. Note though that we likely need this rebuild both on a 64bit
> > > architecture and a 32bit architecture that is not i386 (due to how t64
> > > works). A partial archive rebuild may work to gauge the size of the
> > > problem.
> > > 
> > > I note that this kind of change affects cross builds, so performing
> > > cross builds for armhf on amd64 will likely show many of these failures
> > > (in addition to all the regular cross build failures).
> > > 
> > > I recommend doing more research before moving forward with this. In
> > > particular a MBF about introduced problems would be prudent before doing
> > > the switch and even if we don't switch, such a MBF bears value on its
> > > own.
> 
> > Do you think it makes sense to add this a flag that enables -Werror=format
> > to dpkg-buildflags(1), before, or after a test rebuild, before, or after
> > the MBF if we do one?
> 
> If by adding you mean adding a new feature flag that is disabled by
> default, then depending on the feature area, that might unfortunately
> be equivalent to enable it by default (due to the «all» keyword).

That's a good point, though hopefully people who use "all" understand the
implication...

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Enabling some -Werror=format* by default?

[Guillem Jover]

Hi!

On Mon, 2024-06-10 at 16:06:13 +0500, Andrey Rakhmatullin wrote:
> On Mon, Jun 10, 2024 at 07:48:59AM +0200, Helmut Grohne wrote:
> > > We recently increased the time_t size on certain architectures and some
> > > packages started failing to build because they were using a format
> > > specifier too narrow for the wider time_t, e.g. #1067829.
> > > But the only reason those package FTBFS is they enable either -Werror or
> > > some more specific non-default switch like -Werror=format=2, so I suspect
> > > much more packages contain similar code but gained only a warning. Isn't
> > > this a bad thing? Should we enable at least some combination of -Wformat*
> > > switches by default? Should we at least add a new flag to dpkg-buildflags
> > > and do some test rebuilds with it enabled?
> > 
> > It wasn't quite clear to me what -Werror=format=2 actually means.
> > According to the gcc documentation[1], -Wformat=2 currently means:
> > 
> > -Wformat -Wformat-nonliteral -Wformat-security -Wformat-y2k.

Those are in addition to the ones enabled by -Wformat=1, which are:

  -Wnonnull -Wno-format-contains-nul -Wno-format-extra-args
  -Wno-format-zero-length

So all the above would supposedly turn into -Werror too.

> But I don't know if even -Werror=format is too much to enable by default
> globally (I assume it's a good thing to enable it, though).

Hard to say w/o a rebuild I guess. (And take into account this can
also affect configure-style checks, which might become silent errors
but end up disabling features or enabling unexpected code paths or
similar.)

In general I think it's good (in principle) to enable -Werror flags
that detect actual bugs in code, the problem is always going to be
the amount of fallout and work that creates, and whether there's
consensus about that work commitment being acceptable to maintainers
in Debian, or whether that can interfere with transitions going on,
etc.

> > It also is unclear how this impacts the archive and yes, I'd recommend a
> > rebuild. Note though that we likely need this rebuild both on a 64bit
> > architecture and a 32bit architecture that is not i386 (due to how t64
> > works). A partial archive rebuild may work to gauge the size of the
> > problem.
> > 
> > I note that this kind of change affects cross builds, so performing
> > cross builds for armhf on amd64 will likely show many of these failures
> > (in addition to all the regular cross build failures).
> > 
> > I recommend doing more research before moving forward with this. In
> > particular a MBF about introduced problems would be prudent before doing
> > the switch and even if we don't switch, such a MBF bears value on its
> > own.

> Do you think it makes sense to add this a flag that enables -Werror=format
> to dpkg-buildflags(1), before, or after a test rebuild, before, or after
> the MBF if we do one?

If by adding you mean adding a new feature flag that is disabled by
default, then depending on the feature area, that might unfortunately
be equivalent to enable it by default (due to the «all» keyword).

(I started a design to version the build flags interface, but I got
stuck, so I let it brew as a background process, have pending to finish
that up and propose it on d-dpkg.)

Thanks,
Guillem

Re: Suggestions about i386 support

[Stefano Rivera]

Hi The (2024.06.10_12:22:14_+)
> How to get access to the right parts of the waste stream to be able to
> pull out some working 64-bit hardware is another question, and one where
> I don't have an answer that wouldn't involve spending money (which would
> presumably make the proposed alternative insufficiently comparable,
> since presumably you wouldn't have to spend money to keep the existing
> 32-bit machine in service). If Andrey does, I'd be interested to learn
> it.

The point here is that the Debian project is not intending to support
new hardware on the i386 architecture. The architecture is being kept
around primarily to support running old i386 binaries.

We didn't bring 64bit time_t to the architecture, because of this goal.

There isn't a team working on a modern 32 bit x86 port. We're just
trying to keep the old one going for as long as we can. You're welcome
to try to form such a team, of course :)

The cost of supporting a port of Debian is far more than the cost of the
machines needed to build it. Never mind the cost of 1 user's machine.

Stefano

-- 
Stefano Rivera
  http://tumbleweed.org.za/

Re: Suggestions about i386 support

[The Wanderer]

On 2024-06-10 at 08:09, rhys wrote:

> On Jun 10, 2024, at 01:44, Andrey Rakhmatullin 
> wrote:
> 
>> On Sun, Jun 09, 2024 at 08:39:27PM -0500, r...@neoquasar.org
>> wrote:

>>> Reuse is better than recycle for complex things like electronics.
>>> 
>> You were suggested to resuse an old amd64 machine.
> 
> Again, that assumes that I have such a thing.  I don't.  Unless you
> want to provide one?
> 
> Also, that still doesn't explain how that means the existing 32-bit
> machine stays out of the waste stream.  In your solution, it doesn't.
> In my solution, it does.

I think the suggestion was to take an old amd64 machine out of the waste
stream, and put the existing 32-bit machine into the waste stream, so
that the total amount in the waste stream remains the same but you no
longer need software support for the 32-bit machine.

How to get access to the right parts of the waste stream to be able to
pull out some working 64-bit hardware is another question, and one where
I don't have an answer that wouldn't involve spending money (which would
presumably make the proposed alternative insufficiently comparable,
since presumably you wouldn't have to spend money to keep the existing
32-bit machine in service). If Andrey does, I'd be interested to learn
it.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man. -- George Bernard Shaw



signature.asc
Description: OpenPGP digital signature

Re: Suggestions about i386 support

[rhys]

> On Jun 10, 2024, at 01:44, Andrey Rakhmatullin  wrote:
> 
> On Sun, Jun 09, 2024 at 08:39:27PM -0500, r...@neoquasar.org wrote:
 It's not just a matter of "buy something better." That's easy. 
>> 
>>> Indeed, that is easier and cheaper.
>> 
>> Of course, continuing to use a system I already have is cheaper still.
>> 
>>> What's not easy is that a) that adds another machine to the waste 
>>> stream, instead of continuing to get use from it, and b) someone has 
>>> to take the time to set up the new machine, test things, migrate 
>>> services, etc. to functionally replace the old one. That takes time 
>>> and effort, too, multiplied by the number of such systems out there. 
>> 
>>> (a) is false as newer hardware can already be taken from electronic waste, 
>>> so it does not add new waste.
>> 
>> That is a gross overstatement. Electronics are NOT 100% recyclable. A fair 
>> amount of material still goes to waste, and there are all kinds of costs 
>> associated with those processes. 
>> 
>> Reuse is better than recycle for complex things like electronics. 
> You were suggested to resuse an old amd64 machine.

Again, that assumes that I have such a thing.  I don't.  Unless you want to 
provide one?

Also, that still doesn't explain how that means the existing 32-bit machine 
stays out of the waste stream.  In your solution, it doesn't.  In my solution, 
it does.

--J

Re: Enabling some -Werror=format* by default?

[Andrey Rakhmatullin]

On Mon, Jun 10, 2024 at 07:48:59AM +0200, Helmut Grohne wrote:
> > We recently increased the time_t size on certain architectures and some
> > packages started failing to build because they were using a format
> > specifier too narrow for the wider time_t, e.g. #1067829.
> > But the only reason those package FTBFS is they enable either -Werror or
> > some more specific non-default switch like -Werror=format=2, so I suspect
> > much more packages contain similar code but gained only a warning. Isn't
> > this a bad thing? Should we enable at least some combination of -Wformat*
> > switches by default? Should we at least add a new flag to dpkg-buildflags
> > and do some test rebuilds with it enabled?
> 
> It wasn't quite clear to me what -Werror=format=2 actually means.
> According to the gcc documentation[1], -Wformat=2 currently means:
> 
> -Wformat -Wformat-nonliteral -Wformat-security -Wformat-y2k.
> 
> Of these, we already enable -Werror=format-security, but not the other
> ones. It is not clear to me, which of these actually catches the the
> type mismatches. Would you do more research here?

#include 

void foo()
{
printf("%d", 0L);
}

This produces a warning with just -Wformat ("format ‘%d’ expects argument
of type ‘int’, but argument 2 has type ‘long int’ [-Wformat=]"). It
doesn't look like any more narrow options for this exist (-Wno-* ones
listed in the -Wformat description don't silence this one, and they don't
look like they should anyway).

But I don't know if even -Werror=format is too much to enable by default
globally (I assume it's a good thing to enable it, though).

> It also is unclear how this impacts the archive and yes, I'd recommend a
> rebuild. Note though that we likely need this rebuild both on a 64bit
> architecture and a 32bit architecture that is not i386 (due to how t64
> works). A partial archive rebuild may work to gauge the size of the
> problem.
> 
> I note that this kind of change affects cross builds, so performing
> cross builds for armhf on amd64 will likely show many of these failures
> (in addition to all the regular cross build failures).
> 
> I recommend doing more research before moving forward with this. In
> particular a MBF about introduced problems would be prudent before doing
> the switch and even if we don't switch, such a MBF bears value on its
> own.
Do you think it makes sense to add this a flag that enables -Werror=format
to dpkg-buildflags(1), before, or after a test rebuild, before, or after
the MBF if we do one?

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Enabling some -Werror=format* by default?

[Helmut Grohne]

On Fri, Jun 07, 2024 at 12:19:28AM +0500, Andrey Rakhmatullin wrote:
> We recently increased the time_t size on certain architectures and some
> packages started failing to build because they were using a format
> specifier too narrow for the wider time_t, e.g. #1067829.
> But the only reason those package FTBFS is they enable either -Werror or
> some more specific non-default switch like -Werror=format=2, so I suspect
> much more packages contain similar code but gained only a warning. Isn't
> this a bad thing? Should we enable at least some combination of -Wformat*
> switches by default? Should we at least add a new flag to dpkg-buildflags
> and do some test rebuilds with it enabled?

It wasn't quite clear to me what -Werror=format=2 actually means.
According to the gcc documentation[1], -Wformat=2 currently means:

-Wformat -Wformat-nonliteral -Wformat-security -Wformat-y2k.

Of these, we already enable -Werror=format-security, but not the other
ones. It is not clear to me, which of these actually catches the the
type mismatches. Would you do more research here?

It also is unclear how this impacts the archive and yes, I'd recommend a
rebuild. Note though that we likely need this rebuild both on a 64bit
architecture and a 32bit architecture that is not i386 (due to how t64
works). A partial archive rebuild may work to gauge the size of the
problem.

I note that this kind of change affects cross builds, so performing
cross builds for armhf on amd64 will likely show many of these failures
(in addition to all the regular cross build failures).

I recommend doing more research before moving forward with this. In
particular a MBF about introduced problems would be prudent before doing
the switch and even if we don't switch, such a MBF bears value on its
own.

Helmut

[1] https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html

Re: Illegal Instruction Using sudo in Bookworm on i686

[Marc Haber]

On Sun, 09 Jun 2024 20:39:27 -0500, r...@neoquasar.org wrote:
>Based on these NEW i686-class systems being available, are people more willing 
>to spend the time to support them, knowing that the code will be used on 
>hardware still supported by its manufacturer, still under warranty, still in 
>production use, etc.?

It is not enough to be willing. It is necessary to actually do. I
think that once i686 (re)qualifies as release arch, it will be (again)
one.

Greetings
Marc
-- 

Marc Haber |   " Questions are the | Mailadresse im Header
Rhein-Neckar, DE   | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402

Re: Suggestions about i386 support

[Andrey Rakhmatullin]

On Sun, Jun 09, 2024 at 08:39:27PM -0500, r...@neoquasar.org wrote:
> >> It's not just a matter of "buy something better." That's easy.  
> 
> >Indeed, that is easier and cheaper.
> 
> Of course, continuing to use a system I already have is cheaper still.
> 
> > What's not easy is that a) that adds another machine to the waste 
> > stream, instead of continuing to get use from it, and b) someone has 
> > to take the time to set up the new machine, test things, migrate 
> > services, etc. to functionally replace the old one. That takes time 
> > and effort, too, multiplied by the number of such systems out there.  
> 
> > (a) is false as newer hardware can already be taken from electronic waste, 
> > so it does not add new waste.
> 
> That is a gross overstatement. Electronics are NOT 100% recyclable. A fair 
> amount of material still goes to waste, and there are all kinds of costs 
> associated with those processes. 
> 
> Reuse is better than recycle for complex things like electronics. 
You were suggested to resuse an old amd64 machine.


-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Suggestions about i386 support

[rhys]

>> It's not just a matter of "buy something better." That's easy.  

>Indeed, that is easier and cheaper.

Of course, continuing to use a system I already have is cheaper still.

> What's not easy is that a) that adds another machine to the waste 
> stream, instead of continuing to get use from it, and b) someone has 
> to take the time to set up the new machine, test things, migrate 
> services, etc. to functionally replace the old one. That takes time 
> and effort, too, multiplied by the number of such systems out there.  

> (a) is false as newer hardware can already be taken from electronic waste, so 
> it does not add new waste.

That is a gross overstatement. Electronics are NOT 100% recyclable. A fair 
amount of material still goes to waste, and there are all kinds of costs 
associated with those processes. 

Reuse is better than recycle for complex things like electronics. 

>(Also electricity isn't free everywhere.)

Of course. 

> Maintaining support for ancient hardware costs too. And is more expensive per 
> device as the number of systems is lower.

This will always be true if more than one architecture is supported. One will 
be more expensive (in labor) than the other. 

> [support assistance suggestions]

I will keep this (and a couple others) that have listed specific suggestions 
and work on them as I have time. The idea of fundraising is an interesting one. 

> Which leads to the problem: most people who want this, seem to want to 
> continue to use old hardware (T60, N270). However, continuing to support i386 
> has likely costs much higher than the replacement cost of 
said hardware... Which is probably why nobody really seems sufficiently 
motivated to actually invest resources. (Or do you?)

Except that the "replacement cost" is not tied in any way to the "development 
cost" in a practical sense. There is no way to move resources from anywhere 
that I can use for "replacement cost." I'm on my own. 

That's true for development, too, of course. But you make it sound like there 
is money somewhere for replacement and there isn't. 

Also, HARDWARE replacement is the simplest and cheapest part. That doesn't 
address any of the time or expense in setting up and implementing a new system. 
That is far more expensive and time consuming than just buying a new 
motherboard. It's disingenuous to ignore this. 

> (Sadly you previously refused incoming mail as I got a bounce.) 

As you can imagine, I don't personally approve or deny each incoming email. 
It's possible my mail server denied your message based on anti-spam criteria. I 
do get incoming mail in general and I get mail sent to this list. If you want 
to troubleshoot this further, I can send you another account (off list) that we 
might use as a side channel. 

Re: Illegal Instruction Using sudo in Bookworm on i686

[rhys]

> Or is Debian going to continue to support this processor, since it is still 
> apparently a viable product, enough that new systems are using it?
Considering the plans for i386 I don't think it makes sense to even ask
this question?

Of course it makes sense to ask. The plans are based on a faulty premise, as 
evidenced by this product. Therefore, it is wise to revisit the issue. 

Based on these NEW i686-class systems being available, are people more willing 
to spend the time to support them, knowing that the code will be used on 
hardware still supported by its manufacturer, still under warranty, still in 
production use, etc.?

These systems no longer fall under "hasn't been made since 2004." These are NEW 
systems. 

Re: Suggestions about i386 support

[Ansgar ]

Hi,

On Sun, 2024-06-09 at 08:58 -0500, r...@neoquasar.org wrote:
> What it is is functional, and paid for. And likely, already installed
> and in use somewhere (like all of my 32-bit systems). 
> 
> It's not just a matter of "buy something better." That's easy. 

Indeed, that is easier and cheaper.

> What's not easy is that a) that adds another machine to the waste
> stream, instead of continuing to get use from it, and b) someone has
> to take the time to set up the new machine, test things, migrate
> services, etc. to functionally replace the old one. That takes time
> and effort, too, multiplied by the number of such systems out there. 

(a) is false as newer hardware can already be taken from electronic
waste, so it does not add new waste. (Also electricity isn't free
everywhere.)

Maintaining support for ancient hardware costs too. And is more
expensive per device as the number of systems is lower.

> I've asked before and I'll ask again - and perhaps it's time for
> someone to contact me off list to discuss details - how can I help
> with support for i386? I have just enough software training to be
> dangerous and may be able to help carry some of the actual load here,
> instead of just asking for more free support. 

As I said before
(https://lists.debian.org/debian-devel/2024/05/msg00302.html):

If you look at https://release.debian.org/testing/arch_qualify.html
there is at least several things that can be done:

1. Add CPU security mitigations to Linux kernel.
2. Address builds reaching address limit. There were ideas to use
foreign-arch (amd64) compilers to do so.
3. Look at other arch-specific issues (porter); this can also include
baseline violations and other issues for real i386 hardware.

It is also possible to work on finding funding and asking someone else
to do this. I've no idea how much that would cost, but let's say a few
10k USD.

Which leads to the problem: most people who want this, seem to want to
continue to use old hardware (T60, N270). However, continuing to
support i386 has likely costs much higher than the replacement cost of
said hardware... Which is probably why nobody really seems sufficiently
motivated to actually invest resources. (Or do you?)

(Sadly you previously refused incoming mail as I got a bounce.)

Ansgar

Re: Suggestions about i386 support

[rhys]

Characterizing it as "ancient" intentionally misses the point. 

What it is is functional, and paid for. And likely, already installed and in 
use somewhere (like all of my 32-bit systems). 

It's not just a matter of "buy something better." That's easy. 

What's not easy is that a) that adds another machine to the waste stream, 
instead of continuing to get use from it, and b) someone has to take the time 
to set up the new machine, test things, migrate services, etc. to functionally 
replace the old one. That takes time and effort, too, multiplied by the number 
of such systems out there. 


People are saying "you can't trust the popcorn data," but that kind of sounds 
to me like people looking to support the answer they've already chosen and 
discounting evidence to the contrary. 

I understand that continuing to support things isn't free. I've offered to 
contribute, but my offers have all been met with descriptions of how hard 
someone else is working, rather than any description of how I might help. 

Lastly, there was a description of a problem that "needed to be fixed," but 
also "can't be reproduced." Well, if it can't be reproduced, then by 
definition, it can't be fixed. That's basic troubleshooting. If it can't be 
reproduced and doesn't seem to affect most systems, then it's barely more than 
a rumor and deserves no attention until it's substantiated further. 

So it doesn't make sense to throw that on the pile of "look how hard it is to 
support old hardware." Yes, there are challenges. Lots of them. But that's not 
one of them. 

I've asked before and I'll ask again - and perhaps it's time for someone to 
contact me off list to discuss details - how can I help with support for i386? 
I have just enough software training to be dangerous and may be able to help 
carry some of the actual load here, instead of just asking for more free 
support. 

Sent from my mobile device.

From: Marc Haber 
Sent: Monday, May 20, 2024 02:35
To: debian-devel@lists.debian.org
Subject: Re: Suggestions about i386 support

On Sun, 19 May 2024 20:19:12 +0200, Ben Hutchings 
 wrote: 
>The plan is to keep i386 as a partial architecture that can be used as 
>a "foreign architecture" on systems where amd64 is the main 
>architecture. 

Many people using ancient hardware such as a T60 thinkpad say that's 
not enough for them. I can partly understand both sides though. 

Greetings 
Marc 
-- 
 
Marc Haber |   " Questions are the | Mailadresse im Header 
Rhein-Neckar, DE   | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402 

Re: Illegal Instruction Using sudo in Bookworm on i686

[Andrey Rakhmatullin]

On Sun, Jun 09, 2024 at 06:56:00AM -0500, rhys wrote:
> The question right now is:  Is this processor supported at all?
No.

> So given that these no longer fit the "old and busted" description, is Debian 
> going to stick with the decision to not support them?
I'm sure we will, yes, though I'm not in a position to decide that.

> Or is Debian going to continue to support this processor, since it is still 
> apparently a viable product, enough that new systems are using it?
Considering the plans for i386 I don't think it makes sense to even ask
this question?

> Only after that issue is addressed does anyone need to worry about sudo.  
> Depending on the answer, that is.
Yes, this indeed looks like one of those "why you don't support my CPU"
threads not specific to any software.

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Illegal Instruction Using sudo in Bookworm on i686

[rhys]

> On Jun 9, 2024, at 03:02, Marc Haber  wrote:
> 
> On Sat, 08 Jun 2024 07:25:49 +, Laszlo Merenyi
>  wrote:
>> I was able to make sudo (and visudo) executable working on this CPU, by 
>> recompiling the sudo-1.9.15p5 source code package on the target with 
>> manually removed "-fcf_protection" hardening option.
>> 
>> I did not yet met any other program in Bookworm's i386 release having 
>> similar "illegal instruction" issue. So, by using a recompiled sudo, 
>> Bookworm seems to work on Vortex86DX3.
> 
> I am part of the sudo maintainer team in Debian. Sudo is a security
> relevant software, and in the team we decided that it is more
> important to have a maximally hardened binary than to run on hardware
> that doesnt have official support.
> 
> I'd rather not weaken sudo security for all over supporting a tiny
> part of the hardware base. Also, the bug is a toolchain topic in my
> opinion, sudo is just a user of the problematic toolchain features.
> 
> I'm open for arguments though. Please also see #1043281 which has most
> of the technical points there.

That argument puts the cart before the horse.  Changes to the 'sudo' package 
come later... maybe.  (If this were a normal e-mail thread, I would change the 
subject line to make that clearer.)

The question right now is:  Is this processor supported at all?

On the one hand it does not fit the description of currently supported 32-bit 
686-class processors.  It doesn't support the instruction that Intel backported 
into their processors after publishing the specs.  (Which was a shady move, but 
one thing at a time.)

On the other hand, the reasoning behind the decision to not support some of 
these processors was based on these processors being old, rare, off the market, 
etc.  But this processor is still being sold, complete with motherboards for a 
number of applications.

So given that these no longer fit the "old and busted" description, is Debian 
going to stick with the decision to not support them?  Or is Debian going to 
continue to support this processor, since it is still apparently a viable 
product, enough that new systems are using it?

Only after that issue is addressed does anyone need to worry about sudo.  
Depending on the answer, that is.

--J

Re: Illegal Instruction Using sudo in Bookworm on i686

[Marc Haber]

On Sat, 08 Jun 2024 07:25:49 +, Laszlo Merenyi
 wrote:
>I was able to make sudo (and visudo) executable working on this CPU, by 
>recompiling the sudo-1.9.15p5 source code package on the target with manually 
>removed "-fcf_protection" hardening option.
>
>I did not yet met any other program in Bookworm's i386 release having similar 
>"illegal instruction" issue. So, by using a recompiled sudo, Bookworm seems to 
>work on Vortex86DX3.

I am part of the sudo maintainer team in Debian. Sudo is a security
relevant software, and in the team we decided that it is more
important to have a maximally hardened binary than to run on hardware
that doesnt have official support.

I'd rather not weaken sudo security for all over supporting a tiny
part of the hardware base. Also, the bug is a toolchain topic in my
opinion, sudo is just a user of the problematic toolchain features.

I'm open for arguments though. Please also see #1043281 which has most
of the technical points there.

Greetings
Marc
-- 

Marc Haber |   " Questions are the | Mailadresse im Header
Rhein-Neckar, DE   | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402

Re: Illegal Instruction Using sudo in Bookworm on i686

[Sven Mueller]

https://www.compmall.de/VDX3-EITX-75S-505669 is in stock.I found a variety of other shops selling similar boards, some having them in stock, some not.Am 08.06.2024 13:29 schrieb rhys :Yes, this is a known issue.  This is because Bookworm only supports 32-bit CPUs that are fully Intel compatible.  You will find that there are other binaries such as ffmpeg that fail with the same problem.  (This is from memory.  I have a similar system that is "almost" Intel compatible, but cannot run Bookworm due to these issues.)

The fact that these processors are still sold today is interesting, though.  A big part of the argument for limiting 32-bit support has to do with the assumption that they are all "very old" systems.  So a NEW 32-bit processor might change that discussion.

This is perhaps a better link for that, though, since it shows a product based on that CPU, rather than just a discussion about it:

https://icop.com.tw/product/VDX3-PITX#!

(Note:  The site uses a self-signed certificate.  Bleh.)

It doesn't list a price or availability, though, which suggests that it may NOT actually be sold any more.  

Do you have an example of a site where these are available for purchase as new (not used or refurbished)?

--J

> On Jun 8, 2024, at 02:25, Laszlo Merenyi  wrote:
> 
> Message-id: 
> In-reply-to: <529d4728-d26f-43ff-a957-54b29652d...@neoquasar.org>
> References: <529d4728-d26f-43ff-a957-54b29652d...@neoquasar.org>
> 
> Hello,
> I encountered a similar sudo issue with Bookworm installed on a Vortex86DX3 CPU based embedded computer. 
> Vortex86 series chips are less known x86 CPUs, that are manufactured and are available on-the-market still today. Their type detection in Linux Kernel was implemented in 2021. They are 32-bit only and they are stated not to be fully i686 compatible CPUs. 
> See for example: https://www.icop.com.tw/news/858#! 
> 
> I was able to make sudo (and visudo) executable working on this CPU, by recompiling the sudo-1.9.15p5 source code package on the target with manually removed "-fcf_protection" hardening option.
> 
> I did not yet met any other program in Bookworm's i386 release having similar "illegal instruction" issue. So, by using a recompiled sudo, Bookworm seems to work on Vortex86DX3.
> 
> Regards,
> Laszlo
> 
> 
> 

Re: Illegal Instruction Using sudo in Bookworm on i686

[rhys]

Yes, this is a known issue.  This is because Bookworm only supports 32-bit CPUs 
that are fully Intel compatible.  You will find that there are other binaries 
such as ffmpeg that fail with the same problem.  (This is from memory.  I have 
a similar system that is "almost" Intel compatible, but cannot run Bookworm due 
to these issues.)

The fact that these processors are still sold today is interesting, though.  A 
big part of the argument for limiting 32-bit support has to do with the 
assumption that they are all "very old" systems.  So a NEW 32-bit processor 
might change that discussion.

This is perhaps a better link for that, though, since it shows a product based 
on that CPU, rather than just a discussion about it:

https://icop.com.tw/product/VDX3-PITX#!

(Note:  The site uses a self-signed certificate.  Bleh.)

It doesn't list a price or availability, though, which suggests that it may NOT 
actually be sold any more.  

Do you have an example of a site where these are available for purchase as new 
(not used or refurbished)?

--J

> On Jun 8, 2024, at 02:25, Laszlo Merenyi  wrote:
> 
> Message-id: 
> In-reply-to: <529d4728-d26f-43ff-a957-54b29652d...@neoquasar.org>
> References: <529d4728-d26f-43ff-a957-54b29652d...@neoquasar.org>
> 
> Hello,
> I encountered a similar sudo issue with Bookworm installed on a Vortex86DX3 
> CPU based embedded computer. 
> Vortex86 series chips are less known x86 CPUs, that are manufactured and are 
> available on-the-market still today. Their type detection in Linux Kernel was 
> implemented in 2021. They are 32-bit only and they are stated not to be fully 
> i686 compatible CPUs. 
> See for example: https://www.icop.com.tw/news/858#! 
> 
> I was able to make sudo (and visudo) executable working on this CPU, by 
> recompiling the sudo-1.9.15p5 source code package on the target with manually 
> removed "-fcf_protection" hardening option.
> 
> I did not yet met any other program in Bookworm's i386 release having similar 
> "illegal instruction" issue. So, by using a recompiled sudo, Bookworm seems 
> to work on Vortex86DX3.
> 
> Regards,
> Laszlo
> 
> 
> 

Re: Re: Illegal Instruction Using sudo in Bookworm on i686

[Laszlo Merenyi]

Message-id: 
In-reply-to: <529d4728-d26f-43ff-a957-54b29652d...@neoquasar.org>
References: <529d4728-d26f-43ff-a957-54b29652d...@neoquasar.org>

Hello,
I encountered a similar sudo issue with Bookworm installed on a Vortex86DX3 CPU 
based embedded computer. 
Vortex86 series chips are less known x86 CPUs, that are manufactured and are 
available on-the-market still today. Their type detection in Linux Kernel was 
implemented in 2021. They are 32-bit only and they are stated not to be fully 
i686 compatible CPUs. 
See for example: https://www.icop.com.tw/news/858#! 

I was able to make sudo (and visudo) executable working on this CPU, by 
recompiling the sudo-1.9.15p5 source code package on the target with manually 
removed "-fcf_protection" hardening option.

I did not yet met any other program in Bookworm's i386 release having similar 
"illegal instruction" issue. So, by using a recompiled sudo, Bookworm seems to 
work on Vortex86DX3.

Regards,
Laszlo

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Simon Richter]

Hi,

On 6/8/24 00:42, Simon McVittie wrote:


Having an UTF-8 locale available would be a good thing, but allowing
packages to rely on the active locale to be UTF-8 based reduces our testing
scope.



I'm not sure I follow. Are you suggesting that we should build each
package *n* times (in a UTF-8 locale, in a legacy locale, in locales
known to have unique quirks like Turkish and Japanese, ...), just for
its side-effect of *potentially* passing through those locales to the
upstream test suite?


To an extent, that is what reproducible-builds.org testing does. 
According to [1], they test LC_ALL unset vs LC_ALL set to one of 
et_EE.UTF-8, de_CH.UTF-8, nl_BE.UTF-8, or it_CH.UTF-8. They also test 
other locale variables.


What I'm concerned about is not so much tests inside packages behaving 
differently depending on locale, because that is an upstream problem.


Reproducibility outside of sterile environments is however a problem for 
us as a distribution, because it affects how well people are able to 
contribute to packages they are not directly maintaining -- so for me 
this hooks into the salsa discussion as well: if my package is not 
required to work outside of a very controlled environment, that is also 
an impediment to co-maintenance.



What we say is a bug, and what we say is not a bug, is a policy decision
about our scope: we support some things and we do not support others.


Exactly, and a lot of the debates we've had in the past years is who 
gets to decide what is in scope, and what is "legacy" code that should 
be excised to reduce the workload of the people driving change forward.



What Giole proposed at the beginning of this thread can be rephrased as
declaring that "FTBFS when locale is not C.UTF-8" and "non-reproducible
when locale is varied" are non-bugs, and therefore they are not only
wontfix, but they should be closed altogether as being out-of-scope.


Indeed -- however this class of bugs has already been solved because 
reproducible-builds.org have filed bugs wherever this happened, and 
maintainers have added workarounds where it was impossible to fix.


Turning this workaround into boilerplate code was a mistake already, so 
the answer to the complaint about having to copy boilerplate code that 
should be moved into the framework is "do not copy boilerplate code."



For locales and other facets of the execution environment that are
similarly easy to clear/reset/sanitize/normalize, we don't necessarily
need to be saying "if you do a build in this situation, you are doing
it wrong", because we could equally well be saying "if you do a build in
this situation, the build toolchain will automatically fix it for you" -
much more friendly towards anyone who is building packages interactively,
which seems to be the use-case that you're primarily interested in.


No, automatic fixes are absolutely not friendly -- these *add* to my 
mental load because I need to be aware of them if I want to understand 
what is happening. This is already annoying enough for LC_ALL=C.UTF-8 
inside a package, but at least that usually happens in prominent enough 
places that I can find it, and it is part of the package.


Magic code inside the framework that performs an action automatically 
for me is extra debugging effort, because I need to either know the 
exact rules that the framework applies, or I need to debug into the 
framework.


   Simon

[1] https://tests.reproducible-builds.org/debian/index_variations.html


OpenPGP_0xEBF67A846AABE354.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Simon McVittie]

On Fri, 07 Jun 2024 at 14:32:14 +0200, Guillem Jover wrote:
> I'm a non-native speaker, who has been involved
> in l10n for a long time, while at the same time I've pretty much
> always run my systems with either LANG=C.UTF-8 or before that LANG=C,
> LC_CTYPE=ca_ES.UTF-8 and LC_COLLATE=ca_ES.UTF-8.

So diagnostic messages in your non-English language are so important to
you that you ... set your locale environment variables to values that
result in you seeing diagnostic messages in English instead? I'm not
sure I understand the point you're making here :-)

If your point is that people-who-are-not-you place a higher value on
having diagnostic messages come out in their non-English language than
you personally do, then, yes, that's certainly a valid thing for those
people to want.

But I'm not sure that our current package set actually achieves that -
increasingly many of our packages overwrite the locale with "C.UTF-8"
in some layer of their build system, because they cannot guarantee that
the locale they inherit from the environment is anything reasonable (in
particular, it might be "C", which often breaks tools that want to work
with non-ASCII filenames, inputs or outputs). In the enumeration from
my earlier message, you want (1.), but increasingly, what you actually
get is (2½.) instead, and that results in neither you nor Giole getting
the results you would hope for.

The compromise that Alexandre suggested elsewhere in the thread -
requiring the locale to be *something* UTF-8, but leaving it unspecified
exactly which UTF-8 locale, so that a French-speaking developer can ask
for fr_FR.UTF-8 and get their compiler warnings in French - seems like
something that might actually give you what you want in more cases than
the status quo does? If we mandate a UTF-8 locale, then stack layers like
debhelper's meson plugin could probably stop forcing C.UTF-8.

> we make lots of l10n work rather pointless

Surely only if that l10n work was done on tools that are only ever run
from package builds, and never interactively? A lot of localization is
done for end-user-facing tools (GUI, TUI or CLI) which are never relevant
during a package build anyway.

Even for compilers and similar non-interactive development tools, if
a French-speaking developer runs gcc in the fr_FR.UTF-8 locale during
their upstream development, they'll still benefit from its warnings being
localized into French, even if they would never see those same warnings
during a Debian package build of the same software.

(Analogous: I similarly benefit from gcc having ANSI colour highlights
in its output, even though my Debian package build logs don't have those.)

> and if no one is running with different locales then l10n
> bugs might easily creep in

If no one is running (their interactive sessions) with a particular
locale, why do we even support that locale?

If a locale has users, and they find bugs, then of course those bugs are
something to be fixed (subject to triaging and prioritization, because
we have more bugs than time). But I'm not convinced that occasionally
doing package builds in arbitrary locales is something that will find
locale bugs more readily than real users' normal use of the software
that we ship.

The locale issues I've generally seen during package builds are more like
"I've set up this artificial situation, and now the consequences of what
I asked for are considered to be a bug", for instance "if I run this
tool that wants to output UTF-8 in an ASCII-only locale, it fails with
an error message" (well, of course it does, it's being put in a situation
where it can't do its job as-designed). Or building HTML documentation in
an arbitrary locale, and then having reproducible-builds act surprised
that one build mentions the French translation of "table of contents"
and the other mentions the German translation of "table of contents"
(well, of course it does - "you asked for it, you got it").

> I can understand though the sentiment
> of wanting to shrug this problem category off and wanting instead to
> sweep it under the carpet, but that has accessibility consequences.

I am not advocating sweeping this problem category under the carpet!
I'm just not convinced that saying "we support building any package
with an arbitrary locale at entry to the build system" is actually a
good way to detect the sorts of locale issues that cause the sorts of
concrete end-user-facing problems that have accessibility consequences.

If we want to run test-suites under multiple locales, then we should
maybe consider doing that, rather than using the locale of the build
system as a proxy for the (single) locale in which tests will be run for
this particular build. Saying "it's a bug if your test suite fails in
tr_TR.UTF-8" doesn't do anything to guarantee that anyone will actually
ever try that particular build scenario.

And, even if your test suite passes in tr_TR.UTF-8, that doesn't
necessarily mean that the right thing as expected by a Turkish 

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Simon McVittie]

On Fri, 07 Jun 2024 at 23:22:46 +0900, Simon Richter wrote:
> On 6/7/24 22:40, Alexandre Detiste wrote:
> > Maybe a compromise would be to at least mandate some UTF-8 locale.
> 
> Having an UTF-8 locale available would be a good thing, but allowing
> packages to rely on the active locale to be UTF-8 based reduces our testing
> scope.

I'm not sure I follow. Are you suggesting that we should build each
package *n* times (in a UTF-8 locale, in a legacy locale, in locales
known to have unique quirks like Turkish and Japanese, ...), just for
its side-effect of *potentially* passing through those locales to the
upstream test suite?

If we want to run the test suite in each of those locales, then I think
instead we should do just that: run the test suite (and only the test
suite!) in each of those locales. dh_auto_test could grow a way to do
that, if there's demand. Repeating the whole compilation seems like a
sufficiently large waste of time and resources that, in practice, we
are not going to be able to do this routinely for more than a couple
of locales.

Or, better, we should provide packages with a way to guarantee that
certain locales are available[1], and then tests that are known to be
testing locale-sensitive things should explicitly switch into the locales
of interest, to make sure that they are tested every time, not just if
the builder's locale happens to be the interesting one. For example,
glib2.0's test suite temporarily switches to a Japanese locale in order to
test its handling of formatting dates with an era (Japanese is one of the
few locales where that concept exists), and it does this even when built
by a non-Japanese-speaking developer like me. If it relied on the current
locale for its test coverage, then we would never have discovered #1060735
unless it was coincidentally built by a Japanese developer who is using
a big-endian machine, which seems quite unlikely to happen by chance!

Or, when you say "testing", do you really mean "doing the build, for
the side-effect of seeing whether it succeeds or fails"? (That's not
really the same thing as running a test suite.)

Realistically, several important tools require a UTF-8 locale and will
not work reliably otherwise. Meson either is one of these, or was in
the past, as a result of Python's Unicode behaviour; so debhelper sets
LC_ALL=C.UTF-8 when it invokes Meson, ignoring any preference that might
have been expressed by the caller of dpkg-buildpackage.

[1] Build-Depends: locales-all does this, but is rather heavy.
debian/tests/run-with-locales in e.g. src:glib2.0 is another
implementation, but a more centralized version of this would probably
be better.

> Basically, we need to define the severity of locale bugs

More than that, we need to define what is a locale bug and what is a
non-bug - ideally based on what is genuinely useful, rather than on
"this is something that could theoretically work". We should try to
solve bugs, because that benefits our users and Free Software, but we
should put zero effort into solving non-bugs.

What we say is a bug, and what we say is not a bug, is a policy decision
about our scope: we support some things and we do not support others.
There's nothing magical or set-in-stone about the set of things that we
do and don't support, and it can be varied if there is something close to
consensus that it ought to be. When we're deciding what is in-scope and
what is out-of-scope, we should make that decision based on comparing the
costs and benefits of a wider/narrower scope: "this is in-scope because
I say so" or "this is in-scope because we have traditionally said it is"
are considerably weaker arguments than "this is in-scope because otherwise
we can't access this benefit".

As an analogy: we have chosen to define in Policy that /bin/sh is anything
that supports the POSIX shell language, plus a few designated extensions
like `echo -n`. A consequence of that is that "foobar fails to build when
/bin/sh is bash" is considered to be a bug (which, in an ideal world,
we would solve), because bash is a POSIX shell; but "foobar fails to
build when /bin/sh is csh" is a non-bug (which we wouldn't even leave
open as wontfix, we would just close it), because csh isn't a POSIX shell.

In a different parallel universe, we might reasonably have declared
that /bin/sh is required to be bash (like it is in e.g. Fedora), which
would result in some things that are currently bugs becoming non-bugs -
that's a narrower scope than then one that Debian-in-this-universe has,
resulting in it being easier to maintain but less flexible.

Or, conversely, in a different parallel universe, we might have said that
/bin/sh can be literally any POSIX shell, which is a wider scope than
Debian-in-this-universe: "FTBFS when /bin/sh doesn't support echo -n"
is currently a non-bug, but in that hypothetical distribution it would
be a bug, making the distribution harder to maintain but more flexible.

I am, personally, a fan of setting a 

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Simon Richter]

Hi,

On 6/7/24 22:40, Alexandre Detiste wrote:


Maybe a compromise would be to at least mandate some UTF-8 locale.


Having an UTF-8 locale available would be a good thing, but allowing 
packages to rely on the active locale to be UTF-8 based reduces our 
testing scope.


Basically, we need to define the severity of locale bugs -- if we define 
that package builds can expect LC_ALL=C.UTF-8, and reproducible-builds 
testing finds out that passing a non-UTF-8 locale in causes a deviation 
or a build failure, is the maintainer free to close this "wontfix", or 
should this still be addressed?


Besides locales, there are other things that might affect outcomes, and 
we need to find some reasonable position between "packages should be 
reproducible even if built from the maintainer's user account on their 
personal machine" and "anything that is not a sterile systemd-nspawn 
container with exactly the requested Build-Depends and no Recommended 
packages causes undefined behaviour."


Personally my preference would be as close as possible to the former, 
because if I ever need to work on someone else's package, the chance is 
high that I will need incremental builds and a graphical debugger, and 
both of these are a major hassle in containers.


   Simon


OpenPGP_0xEBF67A846AABE354.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Gioele Barabucci]

On 07/06/24 14:32, Guillem Jover wrote:

Related to this, dpkg-buildpackage 1.20.0 gained a --sanitize-env,
which for now on Debian and derivatives sets LC_COLLATE=C.UTF-8 and
umask=0022.


That's great news!


But _iff_ we end up with dpkg-buildpackage being declared the only
supported entry point, [...]
Personally, I really appreciate how dpkg-buildpackage more and more 
provides a standardized API to/for building Debian packages.


However I would prefer to have this API explicitly described in Policy 
rather than hidden and implicitly defined by the code of a specific program.


What I propose is a new section in Policy [1] that explicitly lists all 
these environment requirements (umask, LC_*, SOURCE_DATE_EPOCH, TMPDIR, 
/bin/sh = POSIX shell + -n, etc). Each builder would then be changed to 
be conformant by default, with the option to steer away if desired (for 
example `dpkg-buildpackage --with-env-var LC_ALL=fr_FR.UTF-8`). This 
would create an uniform environment while preserving the ability to run 
d/rules with user-specific settings.


[1] Or any other similarly "binding" document.

Regards,

--
Gioele Barabucci

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Alexandre Detiste]

Maybe a compromise would be to at least mandate some UTF-8 locale.

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Holger Levsen]

On Fri, Jun 07, 2024 at 02:32:14PM +0200, Guillem Jover wrote:
> And I think forcing a locale on buildds makes perfect sense, because
> we want easy access to build logs. But forcing LC_ALL from the build
> tools implies that no tool invoked will get translated messages at
> all, and means that users (not just maintainers) might have a harder
> time understanding what's going on, we make lots of l10n work rather
> pointless, and if no one is running with different locales then l10n
> bugs might easily creep in.

absolutly agreed & thanks for bringing up this aspect!
 
> Related to this, dpkg-buildpackage 1.20.0 gained a --sanitize-env,
> which for now on Debian and derivatives sets LC_COLLATE=C.UTF-8 and
> umask=0022.

that's great news!


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

The past is over.


signature.asc
Description: PGP signature

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Guillem Jover]

Hi!

On Thu, 2024-06-06 at 15:31:55 +0100, Simon McVittie wrote:
> On Thu, 06 Jun 2024 at 13:32:27 +0300, Hakan Bayındır wrote:
> > C, or C.UTF-8 is not a universal locale which works
> > for all.
> 
> Sure, and I don't think anyone is arguing that you or anyone else should
> set the locale for your interactive terminal session, your GUI desktop
> environment, or even your servers to C.UTF-8.
> 
> But, this thread is about build environments for our packages, not about
> runtime environments. We have two-and-a-half possible policies:

> 1. Status quo, in theory:
> 
>Packages cannot make any assumptions about build-time locales.
> 
>The benefits are:
> 
>- Diagnostic messages are in the maintainer's local language, and
>  potentially easier to understand.

I think this is way more important than the relative space used to
mention it though. :) I'm a non-native speaker, who has been involved
in l10n for a long time, while at the same time I've pretty much
always run my systems with either LANG=C.UTF-8 or before that LANG=C,
LC_CTYPE=ca_ES.UTF-8 and LC_COLLATE=ca_ES.UTF-8.

And I think forcing a locale on buildds makes perfect sense, because
we want easy access to build logs. But forcing LC_ALL from the build
tools implies that no tool invoked will get translated messages at
all, and means that users (not just maintainers) might have a harder
time understanding what's going on, we make lots of l10n work rather
pointless, and if no one is running with different locales then l10n
bugs might easily creep in.

>- If a mass-QA effort wants to assess whether the program is broken by
>  a particular locale, they can easily try running its build-time tests
>  in that locale, **if** the tests do not already force a different
>  locale. (But this comes with some serious limitations: it's likely
>  to have a significant number of false-positive situations where the
>  program is actually working perfectly but the **tests** make assumptions
>  that are not true in all locales, and as a result many upstream
>  projects set their build-time tests to force specific locales
>  anyway - often C, en_US.UTF-8 or C.UTF-8 - regardless of what we
>  might prefer in Debian.)

I consider locale sensitive misbehavior as a category of "upstream"
bugs (be that in the package upstream or the native Debian tools), that
deserve to be spotted and fixed. I can understand though the sentiment
of wanting to shrug this problem category off and wanting instead to
sweep it under the carpet, but that has accessibility consequences.

>   The costs are:

>   - […] but if I'm expected to diagnose the
> problem by reading Chinese error messages, as a non-Chinese-speaker I
> am not going to get far.)

Just as an aside, but while getting non-English messages makes for
harder to diagnose bugs, I've never found it a big deal to deal with
that kind of bug reports, as you can grep for (parts of) the
translated message, and then get the original English string from the
.po for example, or can translate the text back to know what it is
talking about, or ask the reported to translate it for you.

> 2½. Unwelcome compromise (increasingly the status quo):
> 
>Whenever a package is non-reproducible, fails to build or fails tests
>in certain locales (for example legacy non-UTF-8 locales like C or
>en_GB.ISO-8859-15), we add `export LC_ALL=C.UTF-8` to debian/rules and
>move on.
> 
>This is just (2.) with extra steps, and has the same benefit and cost
>for the affected packages as (2.) plus an additional cost (someone must
>identify that the package is in this category and copy/paste the extra
>line), and the same benefit and costs for unmodified packages as (1.).

I agree though, that if we end up with every debian/rules
unconditionally exporting LC_ALL, then there's not much point in not
making the build driver do it instead.


Related to this, dpkg-buildpackage 1.20.0 gained a --sanitize-env,
which for now on Debian and derivatives sets LC_COLLATE=C.UTF-8 and
umask=0022.

But _iff_ we end up with dpkg-buildpackage being declared the only
supported entry point, _and_ there is consensus that we'd want to set
some kind of locale variable from the build driver, then I guess this
could be done as a Debian vendor-specific thing, or via the
dpkg-build-api(7) interface.

Thanks,
Guillem

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Holger Levsen]

On Thu, Jun 06, 2024 at 07:11:46PM +0200, Michael Biebl wrote:
> I would prefer that dpkg-buildpackage provides a "sane" build environment by
> default (which I think includes a LC_ setting pointing at a .UTF-8 locale)
> and fewer packages explicitly setting those things via debian/rules.

same here. like the rest of the world does in 2024.
 
> Afaics, this would actually make efforts like reproducible builds *easier*
> as settings provided by reproducible-builds wouldn't be overwritten by
> debian/rules.

it would make a lot of things easier. :)


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

No matter how many mistakes you make or how slow you progress, you are still
way ahead of everyone who isn't trying.


signature.asc
Description: PGP signature

Re: DEP17 /usr-move: debootstrap set uploaded

[Sean Whitton]

Hello,

On Thu 06 Jun 2024 at 11:25am -06, Gunnar Wolf wrote:

> Let me chime in with all of the people that have been amazed at your
> depth of analysis and your commitment to implementing the *properly
> right* situation. It cannot be overstressed that, for this release
> cycle, one of the main reasons Debian will probably keep its honor
> badge of being easy and reliable to upgrade without breakage is the
> work you have taken as yours.

This is a great way to put it.

Helmut's dedication to this was particularly clear from working with him
on it from the TC side.  We have benefitted repeatedly from his careful
analysis.

-- 
Sean Whitton

Re: File descriptor hard limit is now bumped to the kernel max

[Mike Hommey]

On Thu, Jun 06, 2024 at 06:39:15PM +0200, Marco d'Itri wrote:
> On Jun 06, Simon McVittie  wrote:
> 
> > I believe the change Luca describes is increasing rlim_max (hard limit)
> > but not rlim_cur (soft limit), and the code touched by that patch is
> > looking at rlim_cur, so it should be unaffected anyway - unless some larger
> > component is raising rlim_cur.
> Something did, because inn would start reporting ~1G available fds and 
> then explode, and that patch solved the issue. :-)

There are conditions under which the 1024 limit doesn't apply, like
running in a docker container.

Mike

Re: File descriptor hard limit is now bumped to the kernel max

[Russ Allbery]

Simon McVittie  writes:
> On Thu, 06 Jun 2024 at 18:39:15 +0200, Marco d'Itri wrote:

>> Something did, because inn would start reporting ~1G available fds and
>> then explode, and that patch solved the issue. :-)

> It might be worthwhile to try to track down what larger component did
> this, because inheriting a larger rlim_cur without opt-in can also break
> users of select(2) as described in
> .

I took a quick look at the old INN source and didn't see anything obvious.
I was half-expecting it to do something like set the soft limit to the
hard limit (that sounds like a very INN sort of thing to do), but if so, I
couldn't find it in a quick search.

-- 
Russ Allbery (r...@debian.org)  

Re: File descriptor hard limit is now bumped to the kernel max

[Simon McVittie]

On Thu, 06 Jun 2024 at 18:39:15 +0200, Marco d'Itri wrote:
> On Jun 06, Simon McVittie  wrote:
> > I believe the change Luca describes is increasing rlim_max (hard limit)
> > but not rlim_cur (soft limit), and the code touched by that patch is
> > looking at rlim_cur, so it should be unaffected anyway - unless some larger
> > component is raising rlim_cur.
>
> Something did, because inn would start reporting ~1G available fds and 
> then explode, and that patch solved the issue. :-)

It might be worthwhile to try to track down what larger
component did this, because inheriting a larger rlim_cur
without opt-in can also break users of select(2) as described in
.

smcv

Re: DEP17 /usr-move: debootstrap set uploaded

[Étienne Mollier]

Hi Helmut,

Russ Allbery, on 2024-06-06:
> Marc Haber  writes:
> > Helmut Grohne  wrote:
> 
> >> Thanks for bearing with me and also thanks to all the people (release
> >> team and affected package maintainers in particular) who support this
> >> work.
> 
> > Thank you for doing this work. I have rarely seen a change of this
> > magnitude in Debian that was managed on this professional level. I
> > especially praise the way you have communicated the progress.
> 
> 100% agreed.  The care and excellence that you've brought to this work has
> been exceptional.

Very much seconded, you have my thanks added on the stack!  :)

Have a nice day,  :)
-- 
  .''`.  Étienne Mollier 
 : :' :  pgp: 8f91 b227 c7d6 f2b1 948c  8236 793c f67e 8f0d 11da
 `. `'   sent from /dev/pts/1, please excuse my verbosity
   `-


signature.asc
Description: PGP signature

Re: DEP17 /usr-move: debootstrap set uploaded

[Gunnar Wolf]

Helmut Grohne dijo [Thu, Jun 06, 2024 at 09:28:52AM +0200]:
> Hello,
> 
> I have just uploaded
>  * base-files
>  * bash
>  * dash
>  * glibc
>  * util-linux
> to unstable. These were the last remaining packages shipping aliased
> files inside the package set relevant to debootstrap.
> (...)
> Thanks for bearing with me and also thanks to all the people (release
> team and affected package maintainers in particular) who support this
> work.
> 
> Helmut

Let me chime in with all of the people that have been amazed at your
depth of analysis and your commitment to implementing the *properly
right* situation. It cannot be overstressed that, for this release
cycle, one of the main reasons Debian will probably keep its honor
badge of being easy and reliable to upgrade without breakage is the
work you have taken as yours.


signature.asc
Description: PGP signature

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Michael Biebl]

Am 06.06.24 um 18:52 schrieb Andrey Rakhmatullin:

On Thu, Jun 06, 2024 at 12:08:17PM +0200, Johannes Schauer Marin Rodrigues 
wrote:

Or whether we should switch the default and require that d/rules is run in an
environment (for example as set-up by dpkg-buildpackage) where these variables
are set?

(a previous discussion on this:
https://lists.debian.org/debian-devel/2017/10/msg00317.html)





I would prefer that dpkg-buildpackage provides a "sane" build 
environment by default (which I think includes a LC_ setting pointing at 
a .UTF-8 locale) and fewer packages explicitly setting those things via 
debian/rules.


Afaics, this would actually make efforts like reproducible builds 
*easier* as settings provided by reproducible-builds wouldn't be 
overwritten by debian/rules.


Michael




OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: DEP17 /usr-move: debootstrap set uploaded

[Russ Allbery]

Marc Haber  writes:
> Helmut Grohne  wrote:

>> Thanks for bearing with me and also thanks to all the people (release
>> team and affected package maintainers in particular) who support this
>> work.

> Thank you for doing this work. I have rarely seen a change of this
> magnitude in Debian that was managed on this professional level. I
> especially praise the way you have communicated the progress.

100% agreed.  The care and excellence that you've brought to this work has
been exceptional.

-- 
Russ Allbery (r...@debian.org)  

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Andrey Rakhmatullin]

On Thu, Jun 06, 2024 at 12:08:17PM +0200, Johannes Schauer Marin Rodrigues 
wrote:
> Or whether we should switch the default and require that d/rules is run in an
> environment (for example as set-up by dpkg-buildpackage) where these variables
> are set?
(a previous discussion on this:
https://lists.debian.org/debian-devel/2017/10/msg00317.html)



-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Marco d'Itri]

On Jun 06, Johannes Schauer Marin Rodrigues  wrote:

> A question that goes in a similar direction is whether every d/rules that 
> needs
> it should have to do this:
> 
> export DPKG_EXPORT_BUILDFLAGS=y
> include /usr/share/dpkg/buildflags.mk
> 
> Or whether we should switch the default and require that d/rules is run in an
> environment (for example as set-up by dpkg-buildpackage) where these variables
> are set?
Indeed, a few years ago I decided that it does not make any sense, 
removed all these includes and started always using 
dpkg-buildpackage/debuild to call debian/rules.
This is the resilient and future-proof option.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: File descriptor hard limit is now bumped to the kernel max

[Marco d'Itri]

On Jun 06, Simon McVittie  wrote:

> I believe the change Luca describes is increasing rlim_max (hard limit)
> but not rlim_cur (soft limit), and the code touched by that patch is
> looking at rlim_cur, so it should be unaffected anyway - unless some larger
> component is raising rlim_cur.
Something did, because inn would start reporting ~1G available fds and 
then explode, and that patch solved the issue. :-)

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Processed: Re: Bug#1072491: Dead keys stopped working in unspecified environment

[Debian Bug Tracking System]

Processing control commands:

> retitle -1 Dead keys stopped working in unspecified environment
Bug #1072491 [libglib2.0-0t64] libglib2.0-0t64: Dead keys stopped working, 
again!
Changed Bug title to 'Dead keys stopped working in unspecified environment' 
from 'libglib2.0-0t64: Dead keys stopped working, again!'.
> reassign -1 general
Bug #1072491 [libglib2.0-0t64] Dead keys stopped working in unspecified 
environment
Bug reassigned from package 'libglib2.0-0t64' to 'general'.
No longer marked as found in versions glib2.0/2.80.2-2.
Ignoring request to alter fixed versions of bug #1072491 to the same values 
previously set

-- 
1072491: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1072491
Debian Bug Tracking System
Contact ow...@bugs.debian.org with problems

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Simon McVittie]

On Thu, 06 Jun 2024 at 12:56:10 +0200, Johannes Schauer Marin Rodrigues wrote:
> If we imagine a hypothetical switch to LC_ALL=C.UTF-8 for all source
> packages by default, then there will be bugs.

Do you mean: there will be bugs that break the build of certain packages,
which previously built successfully?

Or do you mean: there will be bugs in which a package does not work as
designed at runtime for users of certain locales, and those bugs would
previously have been detected at build-time by showing up as a FTBFS or
non-reproducibility, but are now only detected by users at runtime?

I'm not convinced that either of those is going to be true, and especially
the first one, because at least some (maybe all) of our official buildds
already export LC_ALL=C.UTF-8 for builds:
https://buildd.debian.org/status/fetch.php?pkg=flatpak=amd64=1.14.8-1=1714492944=0

(Search for "Sufficient free space" and read down a few lines further;
and this is not at all specific to Flatpak, that's just an arbitrary
example of a package that I happen to know has a recent buildd log.)

> I like dumping my time into figuring out why my software
> does something different in a very specific environment

That is of course fine, and you're welcome to do that, but the question
is in part about whether the benefit of expecting that every package
maintainer will do this exceeds its cost.

smcv

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Jeremy Bícha]

I believe debhelper already sets LC_ALL=C.UTF-8 for the cmake, meson,
and ninja buildsystems; therefore many but definitely not all packages
are already built with LC_ALL=C.UTF-8.

Thank you,
Jeremy Bícha

Re: File descriptor hard limit is now bumped to the kernel max

[Simon McVittie]

On Thu, 06 Jun 2024 at 15:21:22 +0200, Marco d'Itri wrote:
> On Jun 06, Luca Boccassi  wrote:
> > The last time this was tried some packages were still not ready, so it
> > was patched out to let them be fixed.
>
> I missed the venerable inn 1.x at the time, and I never noticed that it 
> allocates some data structures for all available fds. Apparently this 
> worked well enough for 1M file descriptors, but not for 1G. :-)
> 
> The solution was easy enough:
> https://salsa.debian.org/md/inn/-/blob/master/debian/patches/limit_getfdcount

I believe the change Luca describes is increasing rlim_max (hard limit)
but not rlim_cur (soft limit), and the code touched by that patch is
looking at rlim_cur, so it should be unaffected anyway - unless some larger
component is raising rlim_cur.

Raising rlim_cur is also a problem for anything that relies on select(2),
which can only represent the first 1024 fds (based on FD_SETSIZE).

In (sufficiently) legacy-free code that can promise that it only uses more
scalable mechanisms like poll/epoll, close_range, and dynamically-sized
data structures, getting the benefit of this change requires locally
raising rlim_cur to match rlim_max, like e.g. dbus-daemon does.

However, if programs that raise the soft limit run subprocesses that are
outside their control, then they should also drop rlim_cur back to 1024
for those subprocesses, like dbus-daemon does for activated services.

smcv

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Simon McVittie]

On Thu, 06 Jun 2024 at 13:32:27 +0300, Hakan Bayındır wrote:
> C, or C.UTF-8 is not a universal locale which works
> for all.

Sure, and I don't think anyone is arguing that you or anyone else should
set the locale for your interactive terminal session, your GUI desktop
environment, or even your servers to C.UTF-8.

But, this thread is about build environments for our packages, not about
runtime environments. We have two-and-a-half possible policies:

1. Status quo, in theory:

   Packages cannot make any assumptions about build-time locales.

   The benefits are:

   - Diagnostic messages are in the maintainer's local language, and
 potentially easier to understand.

   - If a mass-QA effort wants to assess whether the program is broken by
 a particular locale, they can easily try running its build-time tests
 in that locale, **if** the tests do not already force a different
 locale. (But this comes with some serious limitations: it's likely
 to have a significant number of false-positive situations where the
 program is actually working perfectly but the **tests** make assumptions
 that are not true in all locales, and as a result many upstream
 projects set their build-time tests to force specific locales
 anyway - often C, en_US.UTF-8 or C.UTF-8 - regardless of what we
 might prefer in Debian.)

   The costs are:

   - Every program that might be run at build-time is expected to continue
 to cope with running in non-UTF-8 locales, even if we strongly deprecate
 non-UTF-8 locales for production use.

   - Diagnostic messages from the reproducible-builds infrastructure are
 in a random language chosen by the infrastructure, which the maintainer
 does not necessarily understand. (If my package fails to build in a
 Chinese locale, that's a valid bug, but if I'm expected to diagnose the
 problem by reading Chinese error messages, as a non-Chinese-speaker I
 am not going to get far.)

   - If a program that is run during build intentionally has locale-specific
 output, and its output ends up in the .deb, then the package maintainer
 must go to additional effort to force that particular program to have
 reproducible output, usually by running it in a specified locale.

2. What's being proposed in this thread:

   Each package can assume that it's built in the C.UTF-8 locale.
   If it needs a different locale during testing, it can set that itself
   (as e.g. glib2.0 does for some tests), but unless it takes explicit
   action, C.UTF-8 will be used.

   The benefit is that packages that require a UTF-8 locale during build
   or during testing (e.g. to process non-English strings in Python)
   can assume that they have one, and an equivalence class of bugs
   (packages where the content of the .deb can vary with the build-time
   locale, or where e.g. build-time tests fail if UTF-8 output is not
   possible) become non-bugs that we do not need to think about.

   The costs are that we don't get the benefits from (1.) any more.

2½. Unwelcome compromise (increasingly the status quo):

   Whenever a package is non-reproducible, fails to build or fails tests
   in certain locales (for example legacy non-UTF-8 locales like C or
   en_GB.ISO-8859-15), we add `export LC_ALL=C.UTF-8` to debian/rules and
   move on.

   This is just (2.) with extra steps, and has the same benefit and cost
   for the affected packages as (2.) plus an additional cost (someone must
   identify that the package is in this category and copy/paste the extra
   line), and the same benefit and costs for unmodified packages as (1.).

2½ seems like the same boil-the-ocean pattern as any number of
manual-work-intensive transitions: Rules-Requires-Root, debhelper compat
levels, compiler hardening flags and so on. In situations where the
desired state is a backwards-compatibility break, the benefit of having
the transition be opt-in can exceed its (considerable!) cost, but we
shouldn't let that trick us into always paying the additional cost of an
opt-in transition, even in situations where it isn't worth it.

> [Turkish dotted/dotless i]
> creates tons of problems with software which are not aware of the
> issue  (Kodi completely breaks for example, and some software needs
> forced/custom environments to run).

I agree that internationalization issues can be a serious problem **at
runtime**, and when our developers and users find such problems, they can
be reported as bugs downstream or upstream, and (hopefully!) fixed. What
I do not agree with is your suggestion that having the package build
occur in an undefined locale will solve this problem.

For example, let's imagine that we decide that perfect support for Turkish
is a release goal. Having reproducible-builds.org build packages in an
arbitrary language (in practice French is often used, I think?) doesn't
prove anything about whether they handle Turkish correctly, whatever
"correctly" might mean.

If someone 

Re: Locale sanitizer (Was: Mandatory LC_ALL=C.UTF-8 during package building)

[Simon McVittie]

On Thu, 06 Jun 2024 at 14:40:23 +0200, Daniel Gröber wrote:
> On Thu, Jun 06, 2024 at 11:32:33AM +0200, Simon Richter wrote:
> > If your package is not reproducible without it, then your package is
> > broken.
> 
> At build-time, if a program doesn't call setlocale before using locale
> dependent standard library functions it's probably a reproducibility
> hazard.

I think that's the wrong way round: if the program *does* call
setlocale(., "") then it's a potential reproducibility hazard, but
until/unless it calls setlocale or equivalent, it's documented in
setlocale(3) that it runs in the portable (but bad[1]) "C" locale.

But if a program that is run during compilation does call setlocale, then
it's most likely doing so for a reason - most commonly so that it can emit
diagnostic messages in the user's locale, rather than in programmer-English
(and advocates of l10n would likely say that it's a bug for a program to
emit diagnostic messages *without* having called setlocale(., "") first).
It's only a reproducibility hazard if locale-dependent functions are
used to parse machine-readable input, or to emit output that ends up in
the .deb. Without further context, we cannot know whether locale-sensitive
functions are being used correctly or incorrectly, in the same way that we
can't tell without context whether a use of strcmp() is correct or
whether a related but different function like strcasecmp() was intended.

If we want programs to be locale-insensitive during build, there is a
well-defined interface for that - namely, setting LC_ALL to (C or) C.UTF-8.
If we don't do that, but instead leave locale environment variables set
to whatever arbitrary value has been inherited from the caller, then we
are effectively saying "we want programs to remain locale-sensitive", and
arguably it would be a (wishlist?) bug for those programs to *not*
respect the locale environment variables (at least for their diagnostic
output). It seems to me that this applies equally to programs that are
or aren't typically used during compilation.

If a program uses locale-sensitive functions to parse its configuration
file or format its output or something like that, then that's often a
bug, but it might equally well be working as designed/documented - again,
we can't tell which without domain-specific knowledge of the program.

smcv

[1] unable to output, or in some cases parse, any character outside the
1-127 ASCII range

Re: File descriptor hard limit is now bumped to the kernel max

[Marco d'Itri]

On Jun 06, Luca Boccassi  wrote:

> The last time this was tried some packages were still not ready, so it
> was patched out to let them be fixed. Enough time has passed now, and
> it's time to let any unknown leftover just break and be fixed. In all
> known cases, the buggy pattern was to manually iterate over the hard
> limit and close every FD one by one, which is completely unnecessary
> since kernel 5.9 (bullseye/oldstable) since the close_range() syscall
> is available, that can do it in one fell swoop. Any packages still
I missed the venerable inn 1.x at the time, and I never noticed that it 
allocates some data structures for all available fds. Apparently this 
worked well enough for 1M file descriptors, but not for 1G. :-)

The solution was easy enough:
https://salsa.debian.org/md/inn/-/blob/master/debian/patches/limit_getfdcount

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: DEP17 /usr-move: debootstrap set uploaded

[Marc Haber]

On Thu, 6 Jun 2024 09:28:52 +0200, Helmut Grohne 
wrote:
>Thanks for bearing with me and also thanks to all the people (release
>team and affected package maintainers in particular) who support this
>work.

Thank you for doing this work. I have rarely seen a change of this
magnitude in Debian that was managed on this professional level. I
especially praise the way you have communicated the progress.

Greetings
Marc
-- 

Marc Haber |   " Questions are the | Mailadresse im Header
Rhein-Neckar, DE   | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Simon Richter]

Hi,

On 6/6/24 19:56, Johannes Schauer Marin Rodrigues wrote:


At the same time though, I also get annoyed of copy-pasting d/rules snippets
from one of my packages to the next instead of making use of a few more
defaults in our package build environment.


Same here -- I just think that such a workaround should be applied only 
when the package fails to build reproducibly, so this is definitely 
something that should not be cargo-culted in.


What we could also do (but that would be a bigger change) would be 
another flag similar to "Rules-Requires-Root" that lists aspects of the 
package that are known to affect reproducibility -- that would be 
declarative, so the reproducible-builds project can disable the test and 
get meaningful results for the remaining typical problems, and could be 
checked and handled by dpkg-buildpackage as well.


   Simon

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Hakan Bayındır (2024-06-06 12:32:27)
> On 6.06.2024 ÖS 1:08, Johannes Schauer Marin Rodrigues wrote:
> > Quoting Simon Richter (2024-06-06 11:32:33)
> >>> Would it be possible to set in stone that packages are supposed to always
> >>> be built in an environment where LC_ALL=C.UTF-8, or, in other words, that
> >>> builders must set LC_ALL=C.UTF-8?
> >>
> >> This would be the opposite of the current rule.
> >>
> >> Setting LC_ALL=C in debian/rules is an one-liner.
> >>
> >> If your package is not reproducible without it, then your package is
> >> broken. It can go in with the workaround, but the underlying problem
> >> should be fixed at some point.
> >>
> >> The reproducible builds checker explicitly tests different locales to
> >> ensure reproducibility. Adding this requirement would require disabling 
> >> this
> >> check, and thus hide an entire class of bugs from detection.
> > 
> > this is one facet of a much bigger discussion (which we've had before). You 
> > can
> > argue both ways, depending on how you look at this problem.
> > 
> > It is the question of whether we want to:
> > 
> >   a) debian/rules is supposed to be runnable in a wide variety of 
> > environments.
> >   If your package FTBFS in a one specific environment, it is the job of 
> > d/rules
> >   to normalize the environment to cater for the specific needs of the 
> > package.
> > 
> >   b) debian/rules is supposed to be run in a well-defined environment. If 
> > your
> >   package FTBFS in this normalized environment, then it is the job of 
> > d/rules to
> >   add the specific needs of the package to d/rules.
> > 
> > So the question is whether you either want to have d/rules normalize
> > heterogeneous environments (a) or whether you want d/rules to make a 
> > normalized
> > environment specific to the build (b). This is of course a spectrum and I 
> > think
> > we currently doing much more of (a).
> 
> I agree with Simon here.

And, if I understand your reply correctly, you do not disagree with me either?

> C, or C.UTF-8 is not a universal locale which > works for all.

Yes. If we imagine a hypothetical switch to LC_ALL=C.UTF-8 for all source
packages by default, then there will be bugs. The question is, which bugs do we
want to fix: Bugs that happen because of a problem that occurs because we did
*not* set LC_ALL=C.UTF-8 (like reproducible builds problems) or problems that
occur because we *did* set LC_ALL=C.UTF-8 as in the example that you are
describing below.

> While C.UTF-8 solves character representation part of 
> "The Turkish Test" [0], it doesn't solve capitalization and sorting  issues.
> 
> In short, Turkish is the reason why some English text has "İ" and "ı" in 
> it, because in Turkish, they're all present (ı, i, I,  İ), and their 
> capitalization rules are different (i becomes İ and ı  becomes I; i.e. 
> no loss/gain of dot during case changes).
> 
> This  creates tons of problems with software which are not aware of the 
> issue  (Kodi completely breaks for example, and some software needs 
> forced/custom environments to run).

As I'm curious: if your software breaks depending on the LC_ALL setting, how do
you make it produce reproducible binaries? If it breaks with a certain LC_ALL,
then during the build you have to set LC_ALL (or one of its friends) to some
specific value, right?

> So, all in all, if your software is expected to run in an international 
> environment, and its build/run behavior breaks in an environment is not 
> to its liking, I also argue that the software is broken to begin with. 
> Because when this problem takes hold in a codebase, it is nigh 
> impossible to fix.
> 
> So, I think it's better to strive to evolve the software to be a better 
> international citizen rather than give all the software we build an 
> artificially sterile environment, which is iteratively harder and harder 
> to build and maintain.

Just to make sure I'm not misunderstood: I also am tending towards *not*
setting LC_ALL=C.UTF-8 (but probably not as strongly as I understood Simon's
mail) just because I like dumping my time into figuring out why my software
does something different in a very specific environment. Figuring this out
does uncover bugs that should be fixed most of the time.

At the same time though, I also get annoyed of copy-pasting d/rules snippets
from one of my packages to the next instead of making use of a few more
defaults in our package build environment.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Hakan Bayındır]

Hi,

On 6.06.2024 ÖS 1:08, Johannes Schauer Marin Rodrigues wrote:

Hi,

Quoting Simon Richter (2024-06-06 11:32:33)

Would it be possible to set in stone that packages are supposed to always
be built in an environment where LC_ALL=C.UTF-8, or, in other words, that
builders must set LC_ALL=C.UTF-8?


This would be the opposite of the current rule.

Setting LC_ALL=C in debian/rules is an one-liner.

If your package is not reproducible without it, then your package is
broken. It can go in with the workaround, but the underlying problem
should be fixed at some point.

The reproducible builds checker explicitly tests different locales to
ensure reproducibility. Adding this requirement would require disabling this
check, and thus hide an entire class of bugs from detection.


this is one facet of a much bigger discussion (which we've had before). You can
argue both ways, depending on how you look at this problem.

It is the question of whether we want to:

  a) debian/rules is supposed to be runnable in a wide variety of environments.
  If your package FTBFS in a one specific environment, it is the job of d/rules
  to normalize the environment to cater for the specific needs of the package.

  b) debian/rules is supposed to be run in a well-defined environment. If your
  package FTBFS in this normalized environment, then it is the job of d/rules to
  add the specific needs of the package to d/rules.

So the question is whether you either want to have d/rules normalize
heterogeneous environments (a) or whether you want d/rules to make a normalized
environment specific to the build (b). This is of course a spectrum and I think
we currently doing much more of (a).


I agree with Simon here. C, or C.UTF-8 is not a universal locale which 
works for all. While C.UTF-8 solves character representation part of 
"The Turkish Test" [0], it doesn't solve capitalization and sorting  issues.


In short, Turkish is the reason why some English text has "İ" and "ı" in 
it, because in Turkish, they're all present (ı, i, I,  İ), and their 
capitalization rules are different (i becomes İ and ı  becomes I; i.e. 
no loss/gain of dot during case changes).


This  creates tons of problems with software which are not aware of the 
issue  (Kodi completely breaks for example, and some software needs 
forced/custom environments to run).


So, all in all, if your software is expected to run in an international 
environment, and its build/run behavior breaks in an environment is not 
to its liking, I also argue that the software is broken to begin with. 
Because when this problem takes hold in a codebase, it is nigh 
impossible to fix.


So, I think it's better to strive to evolve the software to be a better 
international citizen rather than give all the software we build an 
artificially sterile environment, which is iteratively harder and harder 
to build and maintain.



A question that goes in a similar direction is whether every d/rules that needs
it should have to do this:

export DPKG_EXPORT_BUILDFLAGS=y
include /usr/share/dpkg/buildflags.mk

Or whether we should switch the default and require that d/rules is run in an
environment (for example as set-up by dpkg-buildpackage) where these variables
are set?

Going back to the example of LC_ALL=C.UTF-8 and reproducibility: whether or not
this "hides" problem depends on the definition of what things are allowed to
change between two builds and what constitutes these things has changed already
in the past, for example for the build path which is not *not* changed anymore
but instead recorded in the buildinfo. The same could be argued for
LC_ALL=C.UTF-8 and the environment variables already are part of the buildinfo.

So I do not think that there is an easy answer to this question.

Thanks!

cheers, josch


Cheers,

H.

[0]: https://blog.codinghorror.com/whats-wrong-with-turkey/


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Simon Richter (2024-06-06 11:32:33)
> > Would it be possible to set in stone that packages are supposed to always
> > be built in an environment where LC_ALL=C.UTF-8, or, in other words, that
> > builders must set LC_ALL=C.UTF-8?
> 
> This would be the opposite of the current rule.
> 
> Setting LC_ALL=C in debian/rules is an one-liner.
> 
> If your package is not reproducible without it, then your package is
> broken. It can go in with the workaround, but the underlying problem
> should be fixed at some point.
> 
> The reproducible builds checker explicitly tests different locales to
> ensure reproducibility. Adding this requirement would require disabling this
> check, and thus hide an entire class of bugs from detection.

this is one facet of a much bigger discussion (which we've had before). You can
argue both ways, depending on how you look at this problem.

It is the question of whether we want to:

 a) debian/rules is supposed to be runnable in a wide variety of environments.
 If your package FTBFS in a one specific environment, it is the job of d/rules
 to normalize the environment to cater for the specific needs of the package.

 b) debian/rules is supposed to be run in a well-defined environment. If your
 package FTBFS in this normalized environment, then it is the job of d/rules to
 add the specific needs of the package to d/rules.

So the question is whether you either want to have d/rules normalize
heterogeneous environments (a) or whether you want d/rules to make a normalized
environment specific to the build (b). This is of course a spectrum and I think
we currently doing much more of (a).

A question that goes in a similar direction is whether every d/rules that needs
it should have to do this:

export DPKG_EXPORT_BUILDFLAGS=y
include /usr/share/dpkg/buildflags.mk

Or whether we should switch the default and require that d/rules is run in an
environment (for example as set-up by dpkg-buildpackage) where these variables
are set?

Going back to the example of LC_ALL=C.UTF-8 and reproducibility: whether or not
this "hides" problem depends on the definition of what things are allowed to
change between two builds and what constitutes these things has changed already
in the past, for example for the build path which is not *not* changed anymore
but instead recorded in the buildinfo. The same could be argued for
LC_ALL=C.UTF-8 and the environment variables already are part of the buildinfo.

So I do not think that there is an easy answer to this question.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Simon Richter]

Hi,

> Would it be possible to set in stone that packages are supposed to
> always be built in an environment where LC_ALL=C.UTF-8, or, in other
> words, that builders must set LC_ALL=C.UTF-8?

This would be the opposite of the current rule.

Setting LC_ALL=C in debian/rules is an one-liner.

If your package is not reproducible without it, then your package is
broken. It can go in with the workaround, but the underlying problem
should be fixed at some point.

The reproducible builds checker explicitly tests different locales to
ensure reproducibility. Adding this requirement would require disabling
this check, and thus hide an entire class of bugs from detection.

   Simon

Re: Mandatory LC_ALL=C.UTF-8 during package building

[Luca Boccassi]

On Thu, 6 Jun 2024 at 09:07, Gioele Barabucci  wrote:
>
> Hi,
>
> setting LC_ALL=C.UTF-8 in d/rules is a common way to fix many
> reproducibility problems. It is also, in general, a more sane way to
> build packages, in comparison to using whatever locale settings happen
> to be set during a build. However, sprinkling a variant of `export
> LC_ALL=C.UTF-8` in every d/rules is error-prone and a waste of
> maintainers' time.
>
> Would it be possible to set in stone that packages are supposed to
> always be built in an environment where LC_ALL=C.UTF-8, or, in other
> words, that builders must set LC_ALL=C.UTF-8?
>
> In which document should this rule be stated? Policy?

This makes sense to me, seems similar enough to SOURCE_DATE_EPOCH

Re: DEP17 /usr-move: debootstrap set uploaded

[Luca Boccassi]

On Thu, 6 Jun 2024 at 10:02, Johannes Schauer Marin Rodrigues
 wrote:
>
> Hi Helmut,
>
> Quoting Helmut Grohne (2024-06-06 09:28:52)
> > I have just uploaded
> >  * base-files
> >  * bash
> >  * dash
> >  * glibc
> >  * util-linux
> > to unstable. These were the last remaining packages shipping aliased
> > files inside the package set relevant to debootstrap.
>
> thank you (and freexian for funding you) so much for finally reaching this
> milestone!! Thank you also for doing all your work with incredible diligence
> and attention to detail. This is really outstanding and what I consider to be 
> a
> model for how changes in Debian should be performed.
>
> Your upload timing seems to have been great: the buildds seem to already have
> gone through most of what you uploaded.
>
> I cannot wait for the next dinstall in ~5 hours to test what you uploaded with
> the mmdebstrap test suite (which is also testing debootstrap).
>
> Too bad that it is just these days that snapshot.d.o is moving to the new
> infrastructure (a big thanks to the team who is working on that) which means
> that this crucial transition in unstable will not be part of
> snapshot.debian.org. There are no timestamps recorded for June yet.

+1 thank you Helmut, excellent work!

Re: DEP17 /usr-move: debootstrap set uploaded

[Johannes Schauer Marin Rodrigues]

Hi Helmut,

Quoting Helmut Grohne (2024-06-06 09:28:52)
> I have just uploaded
>  * base-files
>  * bash
>  * dash
>  * glibc
>  * util-linux
> to unstable. These were the last remaining packages shipping aliased
> files inside the package set relevant to debootstrap.

thank you (and freexian for funding you) so much for finally reaching this
milestone!! Thank you also for doing all your work with incredible diligence
and attention to detail. This is really outstanding and what I consider to be a
model for how changes in Debian should be performed.

Your upload timing seems to have been great: the buildds seem to already have
gone through most of what you uploaded.

I cannot wait for the next dinstall in ~5 hours to test what you uploaded with
the mmdebstrap test suite (which is also testing debootstrap).

Too bad that it is just these days that snapshot.d.o is moving to the new
infrastructure (a big thanks to the team who is working on that) which means
that this crucial transition in unstable will not be part of
snapshot.debian.org. There are no timestamps recorded for June yet.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Bits from DPL

[Andreas Tille]

Am Mon, Jun 03, 2024 at 04:34:08PM +0200 schrieb gregor herrmann:
> On Mon, 03 Jun 2024 11:00:34 +, Holger Levsen wrote:
> > 
> > however, "costs to attend" are not the same as "costs while attending"...

ACK.
 
> Also & related, https://wiki.debian.org/HostingBSP#Sponsorships says
> (emphasis mine):
> 
>   participants to a BSP can get a reimbursement of up to 100 USD for their
>   *travel fees*.
> 
> whereas the discussions around the MiniDebConf Berlin were about
> sponsored food, IIRC.
> 
> Note that I'm not saying "Debian must pay for food for a week for
> all people at any price, no matter what", just that "100 USD for
> expenses" is not the same as "100 USD for their travel fees".
> 
> No big deal, just maybe a chance for clarification before the next
> Debian event :)

The major friction point for MiniDebConf Berlin, in my opinion, was the
need to raise a large amount of funds at very short notice. This should
be avoided for future Debian events.  My position is clear: I strongly
support in-person meetings and thus I will do my best enabling active
contributors to attend.  If financial limitations are a barrier for
active contributors, we should find ways to help. The amount of
financial support needed can vary greatly depending on the region where
the in-person meeting is happening.  Therefore, please consider this as
a rule of thumb, not a fixed (upper or lower) limit. More importantly,
organizers should strive for realistic cost calculations in advance and
communicate any changes as soon as possible. Finally, securing sponsors
can be very helpful, and the probability of finding them is typically
higher in regions with higher overall costs.

Kind regards
   Andreas.

-- 
https://fam-tille.de

Re: Bits from DPL

[gregor herrmann]

On Mon, 03 Jun 2024 11:00:34 +, Holger Levsen wrote:

> On Sat, Jun 01, 2024 at 09:52:32AM +0200, Andreas Tille wrote:
> > in connection with MiniDebConf Berlin there was some discussion about
> > what expense per attendee of some in person meeting is OK.  Quoting
> > Chris Lamb from his "Bits from the DPL (March 2018)"[meet1]:
> > 
> >   Debian is willing to reimburse up to 100 USD for expenses to attend Bug
> >   Squashing Parties (BSPs). If there are no BSPs near to you, please help
> >   organise one!
> 
> however, "costs to attend" are not the same as "costs while attending"...

Also & related, https://wiki.debian.org/HostingBSP#Sponsorships says
(emphasis mine):

  participants to a BSP can get a reimbursement of up to 100 USD for their
  *travel fees*.

whereas the discussions around the MiniDebConf Berlin were about
sponsored food, IIRC.

Note that I'm not saying "Debian must pay for food for a week for
all people at any price, no matter what", just that "100 USD for
expenses" is not the same as "100 USD for their travel fees".

No big deal, just maybe a chance for clarification before the next
Debian event :)


Cheers,
gregor

-- 
 .''`.  https://info.comodo.priv.at -- Debian Developer https://www.debian.org
 : :' : OpenPGP fingerprint D1E1 316E 93A7 60A8 104D  85FA BB3A 6801 8649 AA06
 `. `'  Member VIBE!AT & SPI Inc. -- Supporter Free Software Foundation Europe
   `-   


signature.asc
Description: Digital Signature

Re: Bug#1072501: ITP: systemd-boot-installer -- Install systemd-boot on a hard disk

[Simon Richter]

Hi,

On 6/3/24 21:05, Colin Watson wrote:


From the d-i side we've generally preferred to have all the UI be part

of the installer (especially for translations etc.).


Makes sense, thanks!

   Simon

Re: Bug#1072501: ITP: systemd-boot-installer -- Install systemd-boot on a hard disk

[Colin Watson]

On Mon, Jun 03, 2024 at 07:51:44PM +0900, Simon Richter wrote:
> On 6/3/24 15:33, Philipp Kern wrote:
> 
> > > > * Package name    : systemd-boot-installer
> > > Can this be merged into the normal systemd source package?
> 
> > I feel like from a d-i perspective that'd be highly unusual? Having the
> > purely d-i-specific components be owned by d-i is the common setup.
> 
> If it doesn't make sense to merge it (i.e. because it is a glue package
> between two unstable interfaces), then that is fine, but I think that the
> interfaces on both the d-i and the systemd side are stable enough.

>From the d-i side we've generally preferred to have all the UI be part
of the installer (especially for translations etc.).

-- 
Colin Watson (he/him)  [cjwat...@debian.org]

Re: Bug#1072501: ITP: systemd-boot-installer -- Install systemd-boot on a hard disk

[Simon Richter]

Hi,

On 6/3/24 15:33, Philipp Kern wrote:


* Package name    : systemd-boot-installer

Can this be merged into the normal systemd source package?


I feel like from a d-i perspective that'd be highly unusual? Having the 
purely d-i-specific components be owned by d-i is the common setup.


If it doesn't make sense to merge it (i.e. because it is a glue package 
between two unstable interfaces), then that is fine, but I think that 
the interfaces on both the d-i and the systemd side are stable enough.


   Simon

Re: Bits from DPL

[Holger Levsen]

On Sat, Jun 01, 2024 at 09:52:32AM +0200, Andreas Tille wrote:
> in connection with MiniDebConf Berlin there was some discussion about
> what expense per attendee of some in person meeting is OK.  Quoting
> Chris Lamb from his "Bits from the DPL (March 2018)"[meet1]:
> 
>   Debian is willing to reimburse up to 100 USD for expenses to attend Bug
>   Squashing Parties (BSPs). If there are no BSPs near to you, please help
>   organise one!

however, "costs to attend" are not the same as "costs while attending"...


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

I too often read "The planet is dying! " It is not. The planet is adapting 
to what we're doing. There will always be a rich ecosystem on our planet.
But that ecosystem will adapt to the point where we no longer have a place in 
it. No, the planet is not dying. We are. (@kleinemaulw...@troet.cafe)


signature.asc
Description: PGP signature

Re: Bug#1072501: ITP: systemd-boot-installer -- Install systemd-boot on a hard disk

[Philipp Kern]

On 03.06.24 05:43, Simon Richter wrote:

On 6/3/24 09:33, Luca Boccassi wrote:


* Package name    : systemd-boot-installer


Can this be merged into the normal systemd source package?


I feel like from a d-i perspective that'd be highly unusual? Having the 
purely d-i-specific components be owned by d-i is the common setup.


Kind regards
Philipp Kern

Re: Bug#1072501: ITP: systemd-boot-installer -- Install systemd-boot on a hard disk

[Simon Richter]

Hi,

On 6/3/24 09:33, Luca Boccassi wrote:


* Package name: systemd-boot-installer


Can this be merged into the normal systemd source package?

   Simon

Re: How long it takes to update metadata.ftp-master.debian.org for packages from stable-security ?

[Leandro Cunha]

https://lists.debian.org/debian-www/2015/01/msg00034.html

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=490848

This problem has existed since 2015 and I will follow to see what the
outcome of this bug will be. Thanks! 樂

Re: How long it takes to update metadata.ftp-master.debian.org for packages from stable-security ?

[Roberto C . Sánchez]

On Sun, Jun 02, 2024 at 04:49:46PM +0200, Computer Enthusiastic wrote:
> Hello everyone,
> 
> The gstreamer1.0-alsa has been updated recently:
> 
> $ apt -a list gstreamer1.0-alsa
> Listing... Done
> gstreamer1.0-alsa/stable-security,now 1.22.0-3+deb12u2 amd64
> [installed,automatic]
> gstreamer1.0-alsa/stable 1.22.0-3+deb12u1 amd64
> 
> When I try to get the Debian changelog from metadata.ftp-master.debian.org,
> it is not available:
> 
> $ apt-get changelog gstreamer1.0-alsa
> Err:1 https://metadata.ftp-master.debian.org gst-plugins-base1.0
> 1.22.0-3+deb12u2 Changelog
>   Changelog unavailable for gst-plugins-base1.0=1.22.0-3+deb12u2 (404 Not

This is #490848

-- 
Roberto C. Sánchez

Re: How long it takes to update metadata.ftp-master.debian.org for packages from stable-security ?

[Leandro Cunha]

On Sun, Jun 2, 2024 at 11:50 AM Computer Enthusiastic
 wrote:
>
> Hello everyone,
>
> The gstreamer1.0-alsa has been updated recently:
>
> $ apt -a list gstreamer1.0-alsa
> Listing... Done
> gstreamer1.0-alsa/stable-security,now 1.22.0-3+deb12u2 amd64
> [installed,automatic]
> gstreamer1.0-alsa/stable 1.22.0-3+deb12u1 amd64
>
> When I try to get the Debian changelog from
> metadata.ftp-master.debian.org, it is not available:
>
> $ apt-get changelog gstreamer1.0-alsa
> Err:1 https://metadata.ftp-master.debian.org gst-plugins-base1.0
> 1.22.0-3+deb12u2 Changelog
>Changelog unavailable for gst-plugins-base1.0=1.22.0-3+deb12u2 (404
> Not Found [IP: 146.75.54.132 443])
> E: Failed to fetch
> https://metadata.ftp-master.debian.org/changelogs/main/g/gst-plugins-base1.0/gst-plugins-base1.0_1.22.0-3%2bdeb12u2_changelog
>   Changelog unavailable for gst-plugins-base1.0=1.22.0-3+deb12u2 (404
> Not Found [IP: 146.75.54.132 443])
>
> I was wondering how long it takes to update
> metadata.ftp-master.debian.org for packages from stable-security repository:
> a) Is there a fixed amount of time or a timeout ?
> b) Is this an automatic process ?
> c) Does it require manual intervention?
>
> I read in the Debian documentation [1][2] that the stable-security.d.o.
> repository runs in parallel with stable.d.o.
>
> Thanks for any answer and/or reference to further documentation.
>
> --
> [1]
> https://www.debian.org/doc/manuals/developers-reference/pkgs.html#bug-security
> [2]
> https://www.debian.org/doc/manuals/developers-reference/pkgs.html#bug-security-building
>

It occurs with other packages like nodejs.

apt changelog nodejs
Err:1 https://metadata.ftp-master.debian.org nodejs
18.19.0+dfsg-6~deb12u1 Changelog
 Changelog unavailable for nodejs=18.19.0+dfsg-6~deb12u1 (404  Not
Found [IP: 2a04:4e42:16::644 443])
E: Falhou ao buscar
https://metadata.ftp-master.debian.org/changelogs/main/n/nodejs/nodejs_18.19.0%2bdfsg-6%7edeb12u1_changelo
g  Changelog unavailable for nodejs=18.19.0+dfsg-6~deb12u1 (404  Not
Found [IP: 2a04:4e42:16::644 443])

-- 
Cheers,
Leandro Cunha

Re: t64 suffix

[Stephen Kitt]

On Tue, 28 May 2024 11:00:34 +0200, Mathieu Malaterre 
wrote:
> On Mon, May 27, 2024 at 10:26 PM Steve Langasek  wrote:
> > On Thu, May 23, 2024 at 08:14:20PM +0200, Mathieu Malaterre wrote:  
> > > 2. Keep the package-name-doesnt-match-sonames lintian warning (for now)
> > > ?  
> >
> > What package are you seeing such a warning on?  The mass-NMUs included a
> > lintian override to suppress this warning.  
> 
> I think I am missing something big here...anyway here it goes:
> 
> * https://udd.debian.org/lintian/?packages=highway
> 
> (I'll fix the symbols-file-contains-current-version-with-debian-revision
> asap).

The output there indicates that the override added in
https://salsa.debian.org/debian-phototools-team/highway/-/commit/9c4e2b47532c2f8aa781cfd0d11764cc54324e81
doesn’t take into account all the libraries shipped in the package; all you
need to do is update the override to

libhwy1t64: package-name-doesnt-match-sonames libhwy-contrib1 libhwy-test1 
libhwy1

That will fix both warnings.

Regards,

Stephen


pgpbcyJw2RYKI.pgp
Description: OpenPGP digital signature

Re: Suggestion

[Marc Haber]

On Fri, 31 May 2024 18:35:53 +, nino He?i 
wrote:
>My suggestion is regarding locales. Currently we get to chose locale based 
>on choices of language and keyboard layout and here in lies the problem. 
>Rather than offer locales based on our selections don't,just list all of 
>them in installer and let us users chose which one we want.

The "expert" install offers just this and it can be preseeded
accordingly. My routine choice is en_US.UTF-8, en_GB.UTF-8¹ and
de_DE.UTF-8 with en_US being the default.

|[2/5131]mh@swivel:~ $ < /etc/locale.gen grep -vE '^(#.*|)$'
|de_DE.UTF-8 UTF-8
|en_GB.UTF-8 UTF-8
|en_US.UTF-8 UTF-8
|[3/5132]mh@swivel:~ $ locale
|LANG=de_DE.UTF-8
|LANGUAGE=en
|LC_CTYPE="de_DE.UTF-8"
|LC_NUMERIC=de_DE.UTF-8
|LC_TIME=de_DE.UTF-8
|LC_COLLATE="de_DE.UTF-8"
|LC_MONETARY=de_DE.UTF-8
|LC_MESSAGES=en_US.UTF-8
|LC_PAPER=de_DE.UTF-8
|LC_NAME=de_DE.UTF-8
|LC_ADDRESS=de_DE.UTF-8
|LC_TELEPHONE=de_DE.UTF-8
|LC_MEASUREMENT=de_DE.UTF-8
|LC_IDENTIFICATION="de_DE.UTF-8"
|LC_ALL=
|[4/5132]mh@swivel:~ $

Greetings
Marc

¹ some software comes with its English translation in en_GB and if
that's not present it falls back to the ugly de_DE translation
-- 

Marc Haber |   " Questions are the | Mailadresse im Header
Rhein-Neckar, DE   | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402

Re: Suggestion

[Andrew M.A. Cater]

On Fri, May 31, 2024 at 06:35:53PM +, nino Heđi wrote:
> My suggestion is regarding locales. Currently we get to chose locale based
> on choices of language and keyboard layout and here in lies the problem.
> Rather than offer locales based on our selections don't,just list all of
> them in installer and let us users chose which one we want.

For most purposes in a single language, a keyboard, language and timezone
can all be set in one locale. If you need to switch between multiple
languages, you are the exceptional case.

> For me english language and croatian layout blocks me from chosing
> croatian locale during  installation so i have to do it after installation
> is complete and I do not  like it and i think i am not the only one.

Set Croatian locale up during installation - hr_HR.UTF-8 UTF-8 - and add 
a locale with an appropriate English keyboard layout in addition - maybe
 en_US.UTF-8 UTF-8

At any point you can run dpkg-reconfigure locales as root/sudo to generate
extra locales as necessary

Then set up your locales file accordingly. You should be able to switch
straightforwardly between keyboard layouts in a graphical desktop environment
/ use a soft keyboard with no problem in any event.

> I am going to switch to linux soon with Debian being my daily driver and all
> the servers I decide run will also be on top of debian. So make small
> change in installer for debian and turn off locales based on match between
> language of a system and keyboard layout because it is problematic for
> someone who doesn't use US layout on keyboard and i am among those since i
> am croatian.
> -- 

There's always more than one way to solve a problem appropriately in
Debian - I'm sure others can think of other solutions.

With every good wish, as ever,

Andrew Cater
(amaca...@debian.org)
> Sent with Vivaldi Mail. Download Vivaldi for free at vivaldi.com
> 

Re: Y2038-safe replacements for utmp/wtmp and lastlog

[Vincent Lefevre]

On 2024-05-31 01:05:51 +0800, Jun MO wrote:
> And if I understood correctly, wtmpdb require program use PAM to update
> wtmpdb, thus program not use PAM will still write /var/log/wtmp. For
> example, tmux write /var/log/wtmp via libutempter0 and I do not see tmux
> depends on pam.

GNU Screen and xterm also write to /var/log/wtmp via libutempter0. So
this is libutempter0 that should actually be fixed. There is no need
to change the interface, as utempter_add_record uses neither the time
nor a specific structure to write to the file:

int
utempter_add_record(int fd, const char *host);

utempter_add_record() function adds a login record  to  the  database
for  the  TTY belonging to the pseudo-terminal master file descriptor
fd, using the username corresponding with the real  user  ID  of  the
calling process and the optional hostname host.  This function spawns
a privileged process to perform the actual logging.

> $ cat /var/log/wtmp.1 /var/log/wtmp >> wtmp-combined
> $ last -f wtmp-combined
> 
> The output will show when a user logout for those previously
> "gone - no logout" lines. I just realize this about a month ago.

Thanks. I didn't know that.

> I know there is a utmpdump(1) in the util-linux package, that is still
> available in 2.40.1-3 version. The command will dump more information
> than last(1); but it is for every single records, one may need to
> manually match login with logout, boot with shutdown, etc. Still it
> seems to me some information, e.g. exit_status, are still missing.

The host names are also dropped.

I've never seen an exit status. But the wtmp(5) man page says
"not used by Linux init(1)".

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: Y2038-safe replacements for utmp/wtmp and lastlog

[Luca Boccassi]

On Fri, 31 May 2024 at 06:07, Jakub Wilk  wrote:
>
> * Jun MO , 2024-05-31 01:05:
> >And something "off topic". I find there is a char __glibc_reserved[20]
> >variable in the struct utmp, which is commented as "Reserved for future
> >use". Just a brainstorm, if this variable is not currently used, maybe
> >it can be used to solve the Y2038 problem for wtmp?
>
> Or, more easily, you could treat the timestamp as unsigned int:
> https://sourceware.org/cgit/glibc/commit/?id=5361ad3910c257bc
>
> "Architectures which use a 32-bit seconds-since-epoch field in struct
> lastlog, struct utmp, struct utmpx (such as i386, powerpc64le, rv32,
> rv64, x86-64) switched from a signed to an unsigned type for that field.
> This allows these fields to store timestamps beyond the year 2038, until
> the year 2106. Please note that applications are still expected to
> migrate off the interfaces declared in  and  (except
> for login_tty) due to locking and session management problems."

Everybody else is taking advantage of this to switch away from these
legacy interfaces - which are really not good and fraught with issues,
if we are being honest about it. I don't think it makes sense to
invent yet more debianisms just to keep them alive. Just take a backup
of existing files and move on.

Re: Bug#1068017: Y2038-safe replacements for utmp/wtmp and lastlog

[Vincent Lefevre]

On 2024-05-30 18:41:51 +0200, Chris Hofstaedtler wrote:
> wtmpdb takes over the "last" name. Unfortunately without support for
> reading the old files. Nobody wrote tooling to import them or so.

In the new versions of the package, "last" could have been installed
under another name (e.g. last-legacy) instead of dropping it entirely,
in particular because some programs still write to the old wtmp
(which has no issues until 2038).

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: pandoc-filter-diagram_0.2.1-1_amd64.changes REJECTED

[Jonas Smedegaard]

Quoting Shengjing Zhu (2024-05-16 12:14:57)
> On Thu, May 16, 2024 at 4:35 PM Jonas Smedegaard  wrote:
> >
> > Hi FTP-masters (cc d-devel list),
> >
> > A package, that I had initially introduced to Debian some months ago and
> > had been pending in NEW queue since, was rejected few days ago, like
> > this:
> >
> > Quoting Debian FTP Masters (2024-05-14 12:00:12)
> > >
> > > An exception was raised while processing the package:
> > > Traceback (most recent call last):
> > >   File "/srv/ftp-master.debian.org/dak/dak/process_policy.py", line 121, 
> > > in wrapper
> > > function(upload, srcqueue, comments, transaction)
> > >   File "/srv/ftp-master.debian.org/dak/dak/process_policy.py", line 248, 
> > > in comment_accept
> > > transaction.copy_binary(
> > >   File "/srv/ftp-master.debian.org/dak/daklib/archive.py", line 390, in 
> > > copy_binary
> > > self._ensure_extra_source_exists(filename, db_source, archive, 
> > > extra_archives=extra_archives)
> > >   File "/srv/ftp-master.debian.org/dak/daklib/archive.py", line 214, in 
> > > _ensure_extra_source_exists
> > > raise ArchiveException('{0}: Built-Using refers to package {1} (= 
> > > {2}) not in target archive {3}.'.format(filename, source.source, 
> > > source.version, archive.archive_name))
> > > daklib.archive.ArchiveException: 
> > > p/pandoc-filter-diagram/pandoc-filter-diagram_0.2.1-1_amd64.deb: 
> > > Built-Using refers to package rust-ahash (= 0.8.9-2) not in target 
> > > archive ftp-master.
> >
> > I it correct to derive from the above, that packages in NEW queue must
> > be freshly built, and that I (and we, generally) therefore should
> > routinely rebuild packages pending in NEW queue to ensure that they are
> > acceptably?
> 
> Not a ftp-master, but if you see such a message, it means that your
> package has already been accepted, and you can continue uploading
> without going through the NEW queue again.

Thanks, Shengjing Zhu - you are right, althought the concrete *release*
of the package was rejected, the package was nonetheless approved.

Confusing.  I learned something new that day.

 - Jonas

-- 
 * Jonas Smedegaard - idealist & Internet-arkitekt
 * Tlf.: +45 40843136  Website: http://dr.jones.dk/
 * Sponsorship: https://ko-fi.com/drjones

 [x] quote me freely  [ ] ask before reusing  [ ] keep private

signature.asc
Description: signature

Re: Y2038-safe replacements for utmp/wtmp and lastlog

[Jakub Wilk]

* Jun MO , 2024-05-31 01:05:
And something "off topic". I find there is a char __glibc_reserved[20] 
variable in the struct utmp, which is commented as "Reserved for future 
use". Just a brainstorm, if this variable is not currently used, maybe 
it can be used to solve the Y2038 problem for wtmp?


Or, more easily, you could treat the timestamp as unsigned int:
https://sourceware.org/cgit/glibc/commit/?id=5361ad3910c257bc

"Architectures which use a 32-bit seconds-since-epoch field in struct 
lastlog, struct utmp, struct utmpx (such as i386, powerpc64le, rv32, 
rv64, x86-64) switched from a signed to an unsigned type for that field. 
This allows these fields to store timestamps beyond the year 2038, until 
the year 2106. Please note that applications are still expected to 
migrate off the interfaces declared in  and  (except 
for login_tty) due to locking and session management problems."


--
Jakub Wilk

Re: Bug#1068017: Y2038-safe replacements for utmp/wtmp and lastlog

[Chris Hofstaedtler]

* Jun MO  [240530 19:09]:
> On Thu, 30 May 2024 13:18:17 +0200, Vincent Lefevre wrote:
> > I agree, this may be useful. Unfortunately, the current status is
> > that one cannot have both: installing wtmpdb forces the upgrade of
> > util-linux to 2.40.1-3 (at least), where "last" is no longer installed.
> 
> Thanks for the change about version 2.40.1-3 of the util-linux package.
> This is indeed mentioned in the NEWS.Debian from the 2.40.1-3 util-linux
> package, and the NEWS.Debian also suggests installing wtmpdb. But
> the last(1) from wtmpdb can not read /var/log/wtmp:
> 
> $ last -f /var/log/wtmp
> wtmpdb_read_all: SQL error: file is not a database
> 
> And if I understood correctly, wtmpdb require program use PAM to update
> wtmpdb, thus program not use PAM will still write /var/log/wtmp.

Yes. It appears it is unclear if programs that are not
login-equivalents should write to wtmp in the first place.

wtmpdb upstream leans - in my vague reading - towards "no".
 
Chris

Re: Y2038-safe replacements for utmp/wtmp and lastlog

[Jun MO]

On Thu, 30 May 2024 13:18:17 +0200, Vincent Lefevre wrote:
> I agree, this may be useful. Unfortunately, the current status is
> that one cannot have both: installing wtmpdb forces the upgrade of
> util-linux to 2.40.1-3 (at least), where "last" is no longer installed.

Thanks for the change about version 2.40.1-3 of the util-linux package.
This is indeed mentioned in the NEWS.Debian from the 2.40.1-3 util-linux
package, and the NEWS.Debian also suggests installing wtmpdb. But
the last(1) from wtmpdb can not read /var/log/wtmp:

$ last -f /var/log/wtmp
wtmpdb_read_all: SQL error: file is not a database

And if I understood correctly, wtmpdb require program use PAM to update
wtmpdb, thus program not use PAM will still write /var/log/wtmp. For
example, tmux write /var/log/wtmp via libutempter0 and I do not see tmux
depends on pam. But now one can not read /var/log/wtmp neither from
util-linux or wtmpdb. Fortunately, last(1) only links to linux-vdso.so.1,
libc.so.6 and ld-linux-x86-64.so.2. One can extract the /usr/bin/last
binary from old util-linux .deb which can be downloaded from
snapshot.debian.org.

> However, I think that it is better to archive human-readable text files
> (generated by "last" in the past) rather than the wtmp files.
>
> I've used the attached script to get both the IP addresses and the
> host names with "last" (I don't know whether there's a better way to
> get full information).

I agree that human-readable text files are better than the wtmp binary
format files, if the text files provide all information or at least
information one wanted to keep. I find that last(1) may not print all
information, and you need some option to let it print something fully;
so I personally still prefer to keep those wtmp files. For example,
I have noted that the IPv6 address in the output of `last' is truncated
long time ago, and find just a couple of months ago that the -a option
will put the full address in the last column(I see you use that option
in your script). And the output from rotated files(e.g wtmp.1) may have
something like "gone - no logout". Provided the wtmp files are just
many "records" of raw data of C struct of "utmp"(defined in utmp.h, or
see `man 5 utmp'), one record for login, one record for logout,
one record for boot, etc, one can do something like:

$ cat /var/log/wtmp.1 /var/log/wtmp >> wtmp-combined
$ last -f wtmp-combined

The output will show when a user logout for those previously
"gone - no logout" lines. I just realize this about a month ago.

I know there is a utmpdump(1) in the util-linux package, that is still
available in 2.40.1-3 version. The command will dump more information
than last(1); but it is for every single records, one may need to
manually match login with logout, boot with shutdown, etc. Still it
seems to me some information, e.g. exit_status, are still missing.
For archive season, one may write a program that read the wtmp files
and dump all variables of the struct utmp.

And something "off topic". I find there is a char __glibc_reserved[20]
variable in the struct utmp, which is commented as "Reserved for
future use". Just a brainstorm, if this variable is not currently used,
maybe it can be used to solve the Y2038 problem for wtmp?

Regards,
Jun MO

Re: Bug#1068017: Y2038-safe replacements for utmp/wtmp and lastlog

[Chris Hofstaedtler]

* Vincent Lefevre  [240530 13:21]:
> On 2024-05-08 16:53:53 +0800, Jun MO wrote:
> > For last(1) my concern is that it will be helped to keep the original
> > last(1) for back-compatibility to read old wtmp files.
> 
> I agree, this may be useful. Unfortunately, the current status is
> that one cannot have both: installing wtmpdb forces the upgrade of
> util-linux to 2.40.1-3 (at least), where "last" is no longer installed.

wtmpdb takes over the "last" name. Unfortunately without support for
reading the old files. Nobody wrote tooling to import them or so.

Chris

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon McVittie]

On Thu, 30 May 2024 at 15:41:50 +0200, Johannes Schauer Marin Rodrigues wrote:
> I also found another issue with this change in systemd. After the upload to
> unstable, 76 out of 264 mmdebstrap tests on jenkins.debian.net started to 
> fail:
> 
> https://jenkins.debian.net/job/mmdebstrap-jenkins-worker/692/consoleText
> 
> The problem is, that debootstrap wants to mknod which will not work on a tmpfs
> mounted with nodev:
> 
> + debootstrap --no-merged-usr --variant=buildd oldstable /tmp/tmp.nWmx8YeAh3 
> http://127.0.0.1/debian
> /usr/sbin/debootstrap: 1840: cannot create /tmp/tmp.nWmx8YeAh3/test-dev-null: 
> Permission denied
> E: Cannot install into target '/tmp/tmp.nWmx8YeAh3' mounted with noexec or 
> nodev
> 
> Maybe this affects more CI scripts and test setups which attempt to create a
> temporary chroot with debootstrap in /tmp.

I believe this arrangement would also fail if a separate on-disk /tmp
was mounted nodev (which is somewhat common security hardening advice,
although I don't know whether d-i sets this up if asked for a separate
/tmp).

In principle, even the root filesystem could probably be mounted nodev
these days, since /dev is typically a devtmpfs; but I've never tried it,
and I don't know whether anyone really does that.

> The fix which is documented in systemd NEWS makes everything work again:
> 
> --customize-hook='touch "$1/etc/systemd/system/tmp.mount"'

Alternatively, you could consider using somewhere like /var/tmp or
/var/cache/mmdebstrap that is less likely to be mounted nodev?

(As a bonus, those locations are normally on-disk and therefore less
likely to run out of space for chroots/filesystem images/etc. than /tmp.)

smcv

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Peter Pentchev (2024-05-30 10:33:08)
> > thank you for having discussed this change on d-devel and for adding
> > documentation to NEWS and release notes to announce this change. I also
> > think it is sensible to roll this only out on new installations and to keep
> > the behaviour on existing setups. Thank you for implementing that as well.
> > 
> > That being said, maybe some Perl wizard knows how to do a flock on a 
> > directory
> > in Perl?
> 
> I wouldn't call myself a Perl wizard by a long stretch, but I can give it a 
> try :)
> 
> >  I tried this:
> > 
> > use Fcntl qw(LOCK_EX);
> > opendir(my $handle, ".") or die "opendir: $!";
> [snip]
> 
> Here lies your problem. The flock(2) syscall works on file descriptors,
> the things returned by open(2), not on the dirent structures returned by
> opendir(3). So you need something like this:
> 
> use v5.10;  # I really should switch to at least 5.16 if not 5.24
> use strict;
> use warnings;
> 
> use Fcntl qw(O_RDONLY O_DIRECTORY LOCK_EX);
> 
> my $dirname = "/tmp/to-be-locked";
> sysopen(my $handle, "/tmp/to-be-locked", O_RDONLY | O_DIRECTORY) or
> die "Could not open $dirname: $!\n";
> flock($handle, LOCK_EX) or
> die "Could not lock $dirname: $!\n";
> 
> say "locked, it seems";
> sleep(3600);'
> 
> I only put the sleep() part so I could check using lsof that
> the directory was indeed locked. And yeah, the v5.10 part is a leftover
> from the days (...until a month or two ago...) when I still had to
> support stock CentOS 7 systems; I really should train my fingers to
> put 5.24 there.
> 
> Hope that helps!

it absolutely does! Thank you! I was misled by `perldoc -f flock` which states
that it works on filehandles. I'll add your name to the git commit message
unless you object.  :)

I also found another issue with this change in systemd. After the upload to
unstable, 76 out of 264 mmdebstrap tests on jenkins.debian.net started to fail:

https://jenkins.debian.net/job/mmdebstrap-jenkins-worker/692/consoleText

The problem is, that debootstrap wants to mknod which will not work on a tmpfs
mounted with nodev:

+ debootstrap --no-merged-usr --variant=buildd oldstable /tmp/tmp.nWmx8YeAh3 
http://127.0.0.1/debian
/usr/sbin/debootstrap: 1840: cannot create /tmp/tmp.nWmx8YeAh3/test-dev-null: 
Permission denied
E: Cannot install into target '/tmp/tmp.nWmx8YeAh3' mounted with noexec or nodev

Maybe this affects more CI scripts and test setups which attempt to create a
temporary chroot with debootstrap in /tmp.

The fix which is documented in systemd NEWS makes everything work again:

--customize-hook='touch "$1/etc/systemd/system/tmp.mount"'

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Y2038-safe replacements for utmp/wtmp and lastlog

[Vincent Lefevre]

On 2024-05-08 16:53:53 +0800, Jun MO wrote:
> For last(1) my concern is that it will be helped to keep the original
> last(1) for back-compatibility to read old wtmp files.

I agree, this may be useful. Unfortunately, the current status is
that one cannot have both: installing wtmpdb forces the upgrade of
util-linux to 2.40.1-3 (at least), where "last" is no longer installed.

However, I think that it is better to archive human-readable text files
(generated by "last" in the past) rather than the wtmp files.

I've used the attached script to get both the IP addresses and the
host names with "last" (I don't know whether there's a better way to
get full information).

-- 
Vincent Lefèvre  - Web: 
100% accessible validated (X)HTML - Blog: 
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)
#!/usr/bin/env perl

# Output wtmp logs, based on command 'last' from util-linux.
# This command is called twice:
#   1. without the -d option;
#   2. with the -d option to get the host names.

use strict;

my ($proc) = '$Id: lastf 103313 2017-11-06 01:47:30Z vinc17/joooj $'
  =~ /^.Id: (\S+) \d+ / or die;

@ARGV == 1 or $! = 1, die "Usage: $proc \n";
-f $ARGV[0] or $! = 1, die "$proc: $ARGV[0] is not a regular file\n";
-r $ARGV[0] or $! = 1, die "$proc: $ARGV[0] is not a readable file\n";

sub rdlog ($@)
  {
my $file = shift;
open LAST, '-|', 'last', @_, '-axf', $file
  or die "$proc: can't exec 'last': $!";
my @t = ;
close LAST, or die "$proc: 'last' failed: $!";
$t[-1] =~ / begins / && $t[-2] eq "\n"
  or $! = 2, die "$proc: bad format";
return @t;
  }

my @t1 = rdlog $ARGV[0];
my @t2 = rdlog $ARGV[0], '-d';
$#t1 == $#t2 or $! = 1, die "$proc: bad length ($#t1 vs $#t2)\n";
$t1[-1] eq $t2[-1] or die $! = 1, die "$proc: inconsistent last line\n";

foreach my $i (0..$#t1-2)
  {
$_ = $t1[$i];
chomp;
my ($u1) = /^(.){60}/
  or $! = 2, die "$proc: bad format";
my ($u2,$v2) = $t2[$i] =~ /^(.){60}(.*)$/
  or $! = 2, die "$proc: bad format";
$u1 eq $u2 or $! = 2, die "$proc: bad format";
$v2 eq '0.0.0.0' or $_ .= " ($v2)";
print "$_\n";
  }

print "\n$t1[-1]";

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Hakan Bayındır]

> On 30 May 2024, at 09:15, Marc Haber  wrote:
> 
> On Wed, 29 May 2024 14:35:27 +0300, Hakan Bay?nd?r
>  wrote:
>> I'll kindly disagree here. I'd rather not have to go back to every 
>> system and make sure that they all behave the way I want after doing a 
>> periodic, completely boring "apt-get upgrade".
> 
> This change is likely to come to the majority of our installations
> ("stable") with a release upgrade, which is never boring, but one of
> the most exciting things that can happen to a Debian stable system.

You’re right, new Debian releases always brings me joy too. I used “completely 
boring” in a positive way, to underline how uneventful a Debian release upgrade 
is in 99.999% of the time. In other words, I wanted to underline that I prefer 
the next release would be what I expect from Debian. Upgrade, reboot, continue 
drinking coffee and do productive things.

If a new installation comes with different defaults, as I said before, I’m OK 
with that. Software evolves, and should evolve. What I don’t prefer is forcing 
on this evolution on me, on an established system. I work with a lot of 
servers. I don’t want to worry about uncontrolled configuration changes 
happening on updates.

> 
> People doing this responsibly read the release notes before beginning,
> and those release notes have in the past contained things that needed
> doing manually in the process such as the well-known "please upgrade
> kernel first and reboot" during one udev/systemd upgrade.

I also read the release notes and any caveats and warnings before starting, 
however what I don’t expect is to reconfigure a fleet of servers to restore 
some settings which I customized for the particular role and software on that 
server. I know configuration changes are negotiated during upgrades, but when 
partition changes and automated deletion schedules are “inflicted”, that’s 
something bigger than “we upgraded this daemon/tool and brought better 
defaults, wanna look?” Which happens during upgrades.

> Ubuntu seems to have put the release notes in an automatism disguise
> called do-release-upgrade which probably changes from release to
> release regarding what specialty is in the store for this update. We
> don't have that, we ask our users to read the release notes.

I’m not a big fan of Ubuntu and how they do things, and I don’t use it. I 
prefer Debian to be Debian and to need some RTFM process to administer 
correctly. I’m an old school person and sysadmin. I prefer a more direct, “this 
machine has no brain, use yours” approach. :)

> Greetings
> Marc

Hope I managed to express myself in an understandable way this time. :)

Cheers,

H.

> -- 
> 
> Marc Haber |   " Questions are the | Mailadresse im Header
> Rhein-Neckar, DE   | Beginning of Wisdom " | 
> Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Peter Pentchev]

On Thu, May 30, 2024 at 08:35:47AM +0200, Johannes Schauer Marin Rodrigues 
wrote:
> Hi,
> 
> Quoting Luca Boccassi (2024-05-28 01:54:08)
> > Thanks for the useful input, the following has been done:
> > 
> > - existing installations pre-trixie will get an orphaned tmpfiles.d in
> > /etc/ that keeps the existing behaviour unchanged (no cleanup of
> > /var/tmp)
> > - openssh and tmux have been fixed to provide a tmpfiles.d exception
> > to retain their respective files
> > - the /tmp/ description in debian-installer has been updated to note
> > it is a tmpfs by default (via a commit in partman-basicfilesystems,
> > will upload if nobody gets around to it before Trixie's freeze)
> > - two paragraphs have been provided for the release notes ticket
> > - the changes are also noted in NEWS, with instructions on how to
> > override locally
> > - as mentioned, the latest upload to unstable makes /tmp/ a tmpfs by
> > default and for new installations 10+ days old files in /tmp/ and 30+ days
> > old files in /var/tmp/ are cleaned up daily
> 
> thank you for having discussed this change on d-devel and for adding
> documentation to NEWS and release notes to announce this change. I also think
> it is sensible to roll this only out on new installations and to keep the
> behaviour on existing setups. Thank you for implementing that as well.
> 
> That being said, maybe some Perl wizard knows how to do a flock on a directory
> in Perl?

I wouldn't call myself a Perl wizard by a long stretch, but I can give it a try 
:)

>  I tried this:
> 
> use Fcntl qw(LOCK_EX);
> opendir(my $handle, ".") or die "opendir: $!";
[snip]

Here lies your problem. The flock(2) syscall works on file descriptors,
the things returned by open(2), not on the dirent structures returned by
opendir(3). So you need something like this:

use v5.10;  # I really should switch to at least 5.16 if not 5.24
use strict;
use warnings;

use Fcntl qw(O_RDONLY O_DIRECTORY LOCK_EX);

my $dirname = "/tmp/to-be-locked";
sysopen(my $handle, "/tmp/to-be-locked", O_RDONLY | O_DIRECTORY) or
die "Could not open $dirname: $!\n";
flock($handle, LOCK_EX) or
die "Could not lock $dirname: $!\n";

say "locked, it seems";
sleep(3600);'

I only put the sleep() part so I could check using lsof that
the directory was indeed locked. And yeah, the v5.10 part is a leftover
from the days (...until a month or two ago...) when I still had to
support stock CentOS 7 systems; I really should train my fingers to
put 5.24 there.

Hope that helps!

G'luck,
Peter

-- 
Peter Pentchev  r...@ringlet.net r...@debian.org pe...@morpheusly.com
PGP key:https://www.ringlet.net/roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13


signature.asc
Description: PGP signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Luca Boccassi (2024-05-28 01:54:08)
> Thanks for the useful input, the following has been done:
> 
> - existing installations pre-trixie will get an orphaned tmpfiles.d in
> /etc/ that keeps the existing behaviour unchanged (no cleanup of
> /var/tmp)
> - openssh and tmux have been fixed to provide a tmpfiles.d exception
> to retain their respective files
> - the /tmp/ description in debian-installer has been updated to note
> it is a tmpfs by default (via a commit in partman-basicfilesystems,
> will upload if nobody gets around to it before Trixie's freeze)
> - two paragraphs have been provided for the release notes ticket
> - the changes are also noted in NEWS, with instructions on how to
> override locally
> - as mentioned, the latest upload to unstable makes /tmp/ a tmpfs by
> default and for new installations 10+ days old files in /tmp/ and 30+ days
> old files in /var/tmp/ are cleaned up daily

thank you for having discussed this change on d-devel and for adding
documentation to NEWS and release notes to announce this change. I also think
it is sensible to roll this only out on new installations and to keep the
behaviour on existing setups. Thank you for implementing that as well.

That being said, maybe some Perl wizard knows how to do a flock on a directory
in Perl? I tried this:

use Fcntl qw(LOCK_EX);
opendir(my $handle, ".") or die "opendir: $!";
flock($handle, LOCK_EX) or die "flock: $!";

And got this:

flock() on unopened filehandle $handle at test.pl line 8.
(Are you trying to call flock() on dirhandle $handle?)
flock: Bad file descriptor at test.pl line 8.

Wrapping $handle in fileno() doesn't make a difference either. Perl sources
seem to indicate that directory handles are "evil_fh"?

https://sources.debian.org/src/perl/5.38.2-4/util.c/?hl=3783#L3783

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Marc Haber]

On Wed, 29 May 2024 14:35:27 +0300, Hakan Bay?nd?r
 wrote:
>I'll kindly disagree here. I'd rather not have to go back to every 
>system and make sure that they all behave the way I want after doing a 
>periodic, completely boring "apt-get upgrade".

This change is likely to come to the majority of our installations
("stable") with a release upgrade, which is never boring, but one of
the most exciting things that can happen to a Debian stable system.

People doing this responsibly read the release notes before beginning,
and those release notes have in the past contained things that needed
doing manually in the process such as the well-known "please upgrade
kernel first and reboot" during one udev/systemd upgrade.

Ubuntu seems to have put the release notes in an automatism disguise
called do-release-upgrade which probably changes from release to
release regarding what specialty is in the store for this update. We
don't have that, we ask our users to read the release notes.

Greetings
Marc
-- 

Marc Haber |   " Questions are the | Mailadresse im Header
Rhein-Neckar, DE   | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Hakan Bayındır]

> On 29 May 2024, at 17:33, Marvin Renich  wrote:
> 
> * Hakan Bayındır  [240529 07:51]:
>> On 28.05.2024 ÖS 8:16, Andreas Metzler wrote:
>>> On 2024-05-28 Luca Boccassi  
>>> wrote:
>>> [...]
 - existing installations pre-trixie will get an orphaned tmpfiles.d in
 /etc/ that keeps the existing behaviour unchanged (no cleanup of
 /var/tmp)
>>> [...]
>>> 
>>> Hello,
>>> 
>>> I think it is bad choice to deliberately have different behavior for
>>> freshly installed and upgraded systems. Offering upgrades has always
>>> been one of the major selling points of Debian, and imho this
>>> implicitely includes that you do not get a worse or second class Debian
>>> installation when you upgrade it than if you installed from scratch.
>>> 
>>> cu Andreas
>> 
>> I'll kindly disagree here.
> 
> While I agree with Andreas that having the same behavior for upgraded
> systems and new installations is generally better, I agree with you that
> in this specific case it is not the better choice.
> 
>> I'd rather not have to go back to every system
>> and make sure that they all behave the way I want after doing a periodic,
>  ^
>> completely boring "apt-get upgrade".
> 
> You haven't specified what behavior you want.  Is it the existing
> behavior or the new behavior?  This thread is exactly about choosing
> between the two, and possibly between different behavior for new and
> existing systems.
Sorry, I thought that the sentence was self explanatory. Using English as a 
second language has its peculiarities. Let me explain.

1. For existing systems, I don’t want anything modified. Let it be Debian’s old 
defaults, or a custom config I made for that system. IOW, I want my old systems 
to have /tmp as a proper disk partition, and nothing to be cleaned 
periodically, or whatever I set to these systems.

2. For new installation, I’m fine with what Debian proposes. For clarity, I’m 
still against automated cleaning of /tmp and /var/tmp of the reasons I outlined 
before (tl;dr: Long running systems with seldom accessed but required files), 
but I’m fine with whatever Debian decides and ships. At most, I’ll configure 
the behavior I need, but I don’t expect it to be changed down the line by 
package updates.

Hope this clears this part.
> 
> There are four combinations of old/new behavior and upgrading/new
> installation.  Eliminating the obviously bad combination, we are left
> with three:
> 
> A.  Keep old behavior for both upgrading and new installations.
> B.  Keep old behavior for upgrading, use new behavior for new installations.
> C.  Apply new behavior for both.

As I aforementioned, I’m a proponent of “A”, but it’s not favored it seems. So, 
I want to drive the line at “B”. “C” can cause problems because a seasoned 
Debian install is probably old enough to attend school (i.e. 7+ years), and a 
ton of custom configuration is accumulated in /etc, ~/.local, ~/.config and 
elsewhere. Touching working systems and periodically deleting files out of 
nowhere can cause big headaches and worse. 

> The new behavior is preferable for many use cases, but the old behavior
> is not a "BUG" that must be fixed.  Debian has had the old behavior for
> a very long time.
> 
> A number of people have spoken up on this list saying that they are
> relying on the old behavior, and that changing to the new behavior could
> potentially cause serious data loss.

I personally don’t rely on the old behavior, but I work with clusters, and as I 
detailed, some files can linger for a very long time before deleted. These are 
not bugs of these systems, it’s just a side effect of how clusters and long 
running jobs work. Also, RedHat and their derivatives behaves exactly the same 
as how current Debian behaves. /tmp is a partition (which we sometimes mount to 
a high speed NVMe RAID on multi-GPU systems), which is used for data exchange 
between processors, processes, etc., and these files live a pretty long life 
for longer jobs.

> 
> Some people have stated an opinion that keeping upgraded systems in sync
> with the behavior of new installations is desirable.
> 
> So to choose between A, B, and C, we must rank the following:
> 
> X.  desirability of new behavior
> Y.  preventing data loss for an unspecified, but non-zero, number of
>existing systems
> Z.  desirability of having upgraded systems match new installations.
> 
> Both X and Z are primarily opinions with some (non-overwhelming)
> technical merit to each.
> 
> Sufficient technical arguments have been provided on this meta-thread to
> support that Y is highly important and also more important than both X
> and Z.  This means that choice C fails.

So yes, Y is very important for some (small in number, big in footprint) 
installations.

> If Z were more important than X, then the order of importance would
> become Y, then Z, then X, which would make choice A the winner.
> 
> However, there have been no technical arguments