Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[rhys]

You're now at the stage where you're not just MISSING the point of what people 
are trying to tell you, you're actively IGNORING it. 

Automatically deleting files is a bad idea. Those files aren't yours. You don't 
know why they are there. Leave them alone. 

--J

Sent from my mobile device.


From: Luca Boccassi 
Sent: Monday, May 6, 2024 08:20
To: Barak A. Pearlmutter
Cc: debian-devel@lists.debian.org
Subject: Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default 
[was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

On Mon, 6 May 2024 at 13:42, Barak A. Pearlmutter  wrote: 
> 
> > Then upon reading the release notes, on such a machine, one can simply do: 
> > 
> > touch /etc/tmpfiles.d/tmp.conf 
> > 
> > And they get no automated cleanups. 
> 
> This also disables on-boot cleaning of /tmp/. 

Yes, as it's going to be a tmpfs, so it is no longer needed. Trivial 
to maintain though if one wants to do so, though. 

> The root issue here is that deleting not-read-in-a-while 
> but-maybe-stat'ed-recently-by-make-that-doesn't-count files from 
> /var/tmp/ by default, particularly when the system didn't used to, 
> violates the principle of least surprise. 

Which is what release notes are for, if everything was always the same 
we wouldn't spend time to put those together

> There's an old debugging story 

While personal anecdotes and stories can be interesting and amusing in 
many circumstances, I am not really looking for those at this very 
moment. What I am looking for right now is packages or internal 
infrastructure that need 
an update to cope with these two changes before I upload them, so if 
you know of any please do let me know and I'll happily look into it 
and at least file a bug, if not a MR. Thanks. 

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Luca Boccassi  writes:
> Richard Lewis  wrote:

>> - tmux stores sockets in /tmp/tmux-$UID
>> - I think screen might use /tmp/screens

>> I suppose if you detached for a long time you might find yourself
>> unable to reattach.

>> I think you can change the location of these.

> And those are useful only as long as screen/tmux are still running,
> right (I don't really use either that much)? If so, a flock is the right
> solution for these

Also, using /tmp as a path for those sockets was always a questionable
decision.  I believe current versions of screen use /run/screen, which is
a more reasonable location.  Using a per-user directory would be even
better, although I think screen intentionally supports shared screens
between users (which is a somewhat terrifying feature from a security
standpoint, but that's a different argument).

-- 
Russ Allbery (r...@debian.org)  

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Luca Boccassi (2024-05-07 00:09:51)
> To be more specific, as per documentation:
> 
> https://www.freedesktop.org/software/systemd/man/latest/tmpfiles.d.html
> 
> 'x' lines can be used to override cleanup rules, and support globbing,
> so something like:
> 
> x /tmp/mmdebstrap.*

thank you for being patient with me. I saw the man page but I also tried using
codesearch to look for other packages doing the same thing already and was
unable to find one. This made me doubt whether I had understood this correctly.
For example, tmpfiles.d(5) makes it look like there are several minuses
required after the path but apparently those are optional. I've never written
such files before, so your input is useful for me, thank you!

> > And just to confirm (I read this elsewhere in this thread): if my
> > /etc/fstab has an entry for /tmp (with a tmpfs) does this automatically
> > mean that no cleanup will happen or do i still have to put something into
> > /etc to disable the periodic cleanup?
> That's something different, fstab is about whether /tmp is a tmpfs or not,
> cleanups still happen regardless of the filesystem type.

I shall put the following into my local /etc/tmpfiles.d/josch.conf to disable
the cleanup of /tmp completely, then:

x /tmp

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 23:03, Richard Lewis
 wrote:
>
> Luca Boccassi  writes:
>
> > Hence, I am not really looking for philosophical discussions or lists
> > of personal preferences or hypotheticals, but for facts: what would
> > break where, and how to fix it?
>
> - tmux stores sockets in /tmp/tmux-$UID
> - I think screen might use /tmp/screens
>
> I suppose if you detached for a long time you might find yourself unable
> to reattach.
>
> I think you can change the location of these.

And those are useful only as long as screen/tmux are still running,
right (I don't really use either that much)? If so, a flock is the
right solution for these

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 23:00, Johannes Schauer Marin Rodrigues
 wrote:
>
> Quoting Luca Boccassi (2024-05-06 23:28:59)
> > On Mon, 6 May 2024 at 22:27, Simon McVittie  wrote:
> > >
> > > On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues 
> > > wrote:
> > > > If [files can be deleted automatically while mmdebstrap is using them],
> > > > how should applications guard against that from
> > > > happening?
> > >
> > > As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
> > > flock(2) lock on its chroot's root directory, systemd-tmpfiles should
> > > fail to take out its own lock on the directory during cleanup, and
> > > respond to that by treating the directory as "in use" and skipping it.
> >
> > That also works, but only as long as mmdebootstrap is actually
> > running, and as far as I understand it is not a long-running service,
> > not sure if it works for this use case
>
> I guess those users who want to place their chroots in /tmp would have to
> disable the cleanup timer on their systems.
>
> For the mmdebstrap user who just wants to create some tarballs, I think the
> flock solution would be sufficient, thank you.
>
> > More specific settings win, so your application can ship its own tmpfiles.d
> > snippet that defines a longer (or infinite) cleanup time for directories
> > under /tmp or /var/tmp.
>
> How would that look like? Say, my application creates temporary directories
> with the pattern /tmp/mmdebstrap. -- how would the recommended line
> which disables cleanup in /usr/lib/tmpfiles.d/mmdebstrap.conf look like?

To be more specific, as per documentation:

https://www.freedesktop.org/software/systemd/man/latest/tmpfiles.d.html

'x' lines can be used to override cleanup rules, and support globbing,
so something like:

x /tmp/mmdebstrap.*

> And just to confirm (I read this elsewhere in this thread): if my /etc/fstab
> has an entry for /tmp (with a tmpfs) does this automatically mean that no
> cleanup will happen or do i still have to put something into /etc to disable
> the periodic cleanup?

That's something different, fstab is about whether /tmp is a tmpfs or
not, cleanups still happen regardless of the filesystem type.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Quoting Luca Boccassi (2024-05-06 23:28:59)
> On Mon, 6 May 2024 at 22:27, Simon McVittie  wrote:
> >
> > On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues 
> > wrote:
> > > If [files can be deleted automatically while mmdebstrap is using them],
> > > how should applications guard against that from
> > > happening?
> >
> > As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
> > flock(2) lock on its chroot's root directory, systemd-tmpfiles should
> > fail to take out its own lock on the directory during cleanup, and
> > respond to that by treating the directory as "in use" and skipping it.
> 
> That also works, but only as long as mmdebootstrap is actually
> running, and as far as I understand it is not a long-running service,
> not sure if it works for this use case

I guess those users who want to place their chroots in /tmp would have to
disable the cleanup timer on their systems.

For the mmdebstrap user who just wants to create some tarballs, I think the
flock solution would be sufficient, thank you.

> More specific settings win, so your application can ship its own tmpfiles.d
> snippet that defines a longer (or infinite) cleanup time for directories
> under /tmp or /var/tmp.

How would that look like? Say, my application creates temporary directories
with the pattern /tmp/mmdebstrap. -- how would the recommended line
which disables cleanup in /usr/lib/tmpfiles.d/mmdebstrap.conf look like?

And just to confirm (I read this elsewhere in this thread): if my /etc/fstab
has an entry for /tmp (with a tmpfs) does this automatically mean that no
cleanup will happen or do i still have to put something into /etc to disable
the periodic cleanup?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Richard Lewis]

Luca Boccassi  writes:

> Hence, I am not really looking for philosophical discussions or lists
> of personal preferences or hypotheticals, but for facts: what would
> break where, and how to fix it?

- tmux stores sockets in /tmp/tmux-$UID
- I think screen might use /tmp/screens

I suppose if you detached for a long time you might find yourself unable
to reattach.

I think you can change the location of these.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 22:27, Simon McVittie  wrote:
>
> On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues wrote:
> > If [files can be deleted automatically while mmdebstrap is using them],
> > how should applications guard against that from
> > happening?
>
> As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
> flock(2) lock on its chroot's root directory, systemd-tmpfiles should
> fail to take out its own lock on the directory during cleanup, and
> respond to that by treating the directory as "in use" and skipping it.

That also works, but only as long as mmdebootstrap is actually
running, and as far as I understand it is not a long-running service,
not sure if it works for this use case

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 21:08, Johannes Schauer Marin Rodrigues
 wrote:
>
> Hi,
>
> Quoting Luca Boccassi (2024-05-06 15:20:08)
> > While personal anecdotes and stories can be interesting and amusing in many
> > circumstances, I am not really looking for those at this very moment. What I
> > am looking for right now is packages or internal infrastructure that need an
> > update to cope with these two changes before I upload them, so if you know 
> > of
> > any please do let me know and I'll happily look into it and at least file a
> > bug, if not a MR. Thanks.
>
> mmdebstrap uses $TMPDIR (because that's what Perl's File::Temp uses) and falls
> back to /tmp for storing the chroot directories it creates. The modification
> times on files in that chroot directory are potentially much older than 10 
> days
> as they match the timestamps on the packages installed in the chroot. Do I
> understand it correctly that with the proposed change and an unfortunate 
> timing
> it can happen that files older than 10 days get removed from a temporary 
> chroot
> directory in /tmp? If yes, how should applications guard against that from
> happening?

More specific settings win, so your application can ship its own
tmpfiles.d snippet that defines a longer (or infinite) cleanup time
for directories under /tmp or /var/tmp.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon McVittie]

On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues wrote:
> If [files can be deleted automatically while mmdebstrap is using them],
> how should applications guard against that from
> happening?

As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
flock(2) lock on its chroot's root directory, systemd-tmpfiles should
fail to take out its own lock on the directory during cleanup, and
respond to that by treating the directory as "in use" and skipping it.

smcv

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 21:30, Salvo Tomaselli  wrote:
>
> On a fresh installed fedora system I downloaded a .iso in /tmp, then the
> OOMkiller killed wayland, so everything died.
>
> If you know you won't ever fill it up, I guess it's fine. But I'd go for the
> safer (and sadly slower) option.

You can just configure it (or disable it) on older devices or where
you don't want it, it's very easy to do as already mentioned

Solving a file conflict between package "nq" / "fq"

[Preuße , Hilmar]

Hi all,

during the preparation of a new version of package "nq" (via NMU) it was 
found that there exists a file conflict with package "fq" (#1005961), 
which was incorrectly solved in the past. For now I unarchived and 
reopened the old issue. According to the policy:


"Two different packages must not install programs with different 
functionality but with the same filenames. (...) If this case happens, 
one of the programs must be renamed. The maintainers should report this 
to the debian-devel mailing list and try to find a consensus about which 
program will have to be renamed. (...)"


Hence I contact this list. Is there a formal process to generate 
decisions / consensus? Please note that I'm not the maintainer of "nq" 
and I'm not in the position to rename binaries to solve file conflicts.


Hilmar
--
sigfault



OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Richard Lewis]

Luca Boccassi  writes:

> On Mon, 6 May 2024 at 15:42, Richard Lewis
>  wrote:
>>
>> Luca Boccassi  writes:
>>
>> > Hence, I am not really looking for philosophical discussions or lists
>> > of personal preferences or hypotheticals, but for facts: what would
>> > break where, and how to fix it?
>>
>> cleaning /tmp or /var/tmp: users may lose files if they dont realise a
>>   directory tmp can be cleaned without a reboot. something in /var/tmp
>>   that's been in there for 35 days before an upgrade might be deleted
>>   before the user reads the NEWS.Debian email, meaning they have no
>>   chance to react). Maybe you could postpone the very first deletion
>>   until after the next reboot?
>>
>> using a tmpfs: is there a risk of losing unrelated data due to more
>>   frequent OOM killing random other programmes due to /tmp using all the
>>   memory?  is there a case to only use a tmpfs if the system has
>>   "enough" memory?
>
> Again, those are all hypotheticals, and one can construct similarly
> contrived thought exercises for most possible permutations of most
> configurations, and the answer is always the same: customize the
> configuration accordingly. Hence, not relevant right now.

> What is relevant is: which packages, if any, or which DSA-owned
> systems, if any, are actually affected and how?

Why do you think that the impact on users is less important than the
impact on debian packages?

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Luca Boccassi (2024-05-06 15:20:08)
> While personal anecdotes and stories can be interesting and amusing in many
> circumstances, I am not really looking for those at this very moment. What I
> am looking for right now is packages or internal infrastructure that need an
> update to cope with these two changes before I upload them, so if you know of
> any please do let me know and I'll happily look into it and at least file a
> bug, if not a MR. Thanks.

mmdebstrap uses $TMPDIR (because that's what Perl's File::Temp uses) and falls
back to /tmp for storing the chroot directories it creates. The modification
times on files in that chroot directory are potentially much older than 10 days
as they match the timestamps on the packages installed in the chroot. Do I
understand it correctly that with the proposed change and an unfortunate timing
it can happen that files older than 10 days get removed from a temporary chroot
directory in /tmp? If yes, how should applications guard against that from
happening?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Andrey Rakhmatullin]

On Mon, May 06, 2024 at 04:50:50PM +0100, Barak A. Pearlmutter wrote:
> > tmpfiles.d snippets can be defined to cleanup on a timer _anything_,
> 
> It's a question of what the *default* behaviour should be.
> 
> For whatever reason, a lot of people who process large data use
> /var/tmp/FOO/ as a place to store information that should not be
> backed up, but also should not just disappear.
To be honest I'm greatly surprised by this idea, and by the suggestion
that a lot of people do this, to me this is very similar by that half-joke
about people storing useful files in the Recycle Bin.
But I also thought that "/tmp is on tmpfs, /var/tmp isn't, and that's
their main difference" are our defaults for years, when they apparently
aren't (now I understand why people sometimes suggest using /dev/shm or
/run for this or that to increase performance).

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 16:51, Barak A. Pearlmutter  wrote:
>
> > tmpfiles.d snippets can be defined to cleanup on a timer _anything_,
>
> It's a question of what the *default* behaviour should be.

No, it is not, at least not for the strawman you conjured. So I gather
that git doesn't warn when cloning on other directories that might be
cleaned up - great, that's a non-issue too.
If you want it to do so, then the appropriate way is for you to file
an RFE upstream and ask them to parse the installed tmpfiles.d on a
system before cloning. This is not that hard to do, and the format is
stable, and if they ask about that I'll happily back you up on this
stability guarantee.
But what git does or does not allow is then completely unrelated to
anything happening here.

> For whatever reason, a lot of people who process large data use /var/tmp/FOO/ 
> as a place to store information that should not be backed up, but also should 
> not just disappear.

Then such people, assuming they actually exist, can configure their
custom systems accordingly upon reading the release notes before
upgrading, as they would do anyway if installing on CentOS or any
other major OS. Hence, not an issue either.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

> tmpfiles.d snippets can be defined to cleanup on a timer _anything_,

It's a question of what the *default* behaviour should be.

For whatever reason, a lot of people who process large data use
/var/tmp/FOO/ as a place to store information that should not be
backed up, but also should not just disappear.

Maybe they should use ~/FOO/ but put a CACHEDIR.TAG file with the
right magic contents there. Or touch ~/FOO/.nobackup and use duplicity
--exclude-if-present .nobackup. Or ~/.cache/FOO/, although that's a
bit counterintuitive because it's "hidden" and not really meant for
this purpose.

There's a reason people don't want these files below ~/, which is that
some backup mechanisms, like btrfs snapshots or LVMs snapshots, may
snapshot files there regardless; and often /home/ is on more
stable/expensive storage (raid k for k≥1) while /var/tmp/ is on a
cheaper substrate.

Given the era of large data we're living in, this is a ubiquitous
problem. Since we don't give users a standard solution, they've rolled
their own. If we think that solution is wrong, we should (a) make an
actual official suggested location for this sort of material,
/var/scratch/ or /var/scratch/${USER}/ with a link there from
~/Scratch or something, and (b) transition them to it, ideally without
unexpectedly deleting their files and messing up their work in the
process.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 16:42, Simon Richter  wrote:
>
> Hi,
>
> On 5/6/24 19:57, Michael Biebl wrote:
>
> > Afaik, /var/tmp has never been cleaned up on /boot.
> > So I'm not sure what you mean with "no longer"?
>
> Oof, you're right, it was /tmp, /var/run, /var/lock:
>
>  [ "$VERBOSE" != no ] && echo -n "Cleaning"
>  [ -d /tmp ] && cleantmp
>  [ -d /var/run ] && cleanrun
>  [ -d /var/lock ] && cleanlock
>  [ "$VERBOSE" != no ] && echo "."
>
> Would it make sense to make it a bug for a package to use /var/tmp (on
> my system, I can see files from audacity and reportbug there) and
> declare that this directory is for the use of the sysadmin only?

In general it should be the opposite, it's fine for packages to use
/tmp or /var/tmp (using appropriate measures such as random file/dir
names to avoid clashes and attacks), as long as it's really for
temporary stuff, as in, if they are lost it's not a problem and they
are recreated. If you are storing your precious irreplaceable state on
/tmp or /var/tmp, then you are doing it wrong (TM).
In general it's users who should be careful and know what they are
doing, especially due to the world-writable nature of both
directories.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 16:30, Simon Richter  wrote:
>
> Hi,
>
> On 5/6/24 20:19, Luca Boccassi wrote:
>
> > Is that the default layout, or a selectable option?
>
> When you create a partition manually, it asks for the mount point, and
> makes a number of suggestions in a dropdown, and /tmp is one of these.
>
> There is also a "enter manually" option.
>
> > If the latter, to me it makes sense to keep it, maybe with updated
> > wording. Making things customizable is not really an issue, as long as
> > it's clear what happens where.
>
> The only wording there is "/tmp - Temporary files" or something like
> that (has been a while since I saw that particular menu) -- it's an
> option in a menu, so space is limited.
>
> FWIW, I don't see much of a use case for leaving /tmp on the root
> partition, so if no /tmp mount point is configured, mounting a tmpfs is
> a good choice. Basically anything that can boot the installer has enough
> RAM.

Found it:

https://salsa.debian.org/installer-team/partman-basicfilesystems/-/blob/master/debian/partman-basicfilesystems.templates?ref_type=heads#L97

So  unless someone has a better suggestion, I can change it from:

"/tmp - temporary files"

To something like:

"/tmp - temporary files (default: tmpfs)"

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 15:42, Richard Lewis
 wrote:
>
> Luca Boccassi  writes:
>
> > Hence, I am not really looking for philosophical discussions or lists
> > of personal preferences or hypotheticals, but for facts: what would
> > break where, and how to fix it?
>
> cleaning /tmp or /var/tmp: users may lose files if they dont realise a
>   directory tmp can be cleaned without a reboot. something in /var/tmp
>   that's been in there for 35 days before an upgrade might be deleted
>   before the user reads the NEWS.Debian email, meaning they have no
>   chance to react). Maybe you could postpone the very first deletion
>   until after the next reboot?
>
> using a tmpfs: is there a risk of losing unrelated data due to more
>   frequent OOM killing random other programmes due to /tmp using all the
>   memory?  is there a case to only use a tmpfs if the system has
>   "enough" memory?

Again, those are all hypotheticals, and one can construct similarly
contrived thought exercises for most possible permutations of most
configurations, and the answer is always the same: customize the
configuration accordingly. Hence, not relevant right now.
What is relevant is: which packages, if any, or which DSA-owned
systems, if any, are actually affected and how?

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon Richter]

Hi,

On 5/6/24 19:57, Michael Biebl wrote:


Afaik, /var/tmp has never been cleaned up on /boot.
So I'm not sure what you mean with "no longer"?


Oof, you're right, it was /tmp, /var/run, /var/lock:

[ "$VERBOSE" != no ] && echo -n "Cleaning"
[ -d /tmp ] && cleantmp
[ -d /var/run ] && cleanrun
[ -d /var/lock ] && cleanlock
[ "$VERBOSE" != no ] && echo "."

Would it make sense to make it a bug for a package to use /var/tmp (on 
my system, I can see files from audacity and reportbug there) and 
declare that this directory is for the use of the sysadmin only?


   Simon

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon Richter]

Hi,

On 5/6/24 20:19, Luca Boccassi wrote:


Is that the default layout, or a selectable option?


When you create a partition manually, it asks for the mount point, and 
makes a number of suggestions in a dropdown, and /tmp is one of these.


There is also a "enter manually" option.


If the latter, to me it makes sense to keep it, maybe with updated
wording. Making things customizable is not really an issue, as long as
it's clear what happens where.


The only wording there is "/tmp - Temporary files" or something like 
that (has been a while since I saw that particular menu) -- it's an 
option in a menu, so space is limited.


FWIW, I don't see much of a use case for leaving /tmp on the root 
partition, so if no /tmp mount point is configured, mounting a tmpfs is 
a good choice. Basically anything that can boot the installer has enough 
RAM.


   Simon

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 16:03, Barak A. Pearlmutter  wrote:
>
> If it clones into /tmp the *entire* tree will either be reaped (upon
> reboot) or not.
>
> But having just some files deleted from a git dir or git working dir
> is much more dangerous, because various git commands can treat files
> deleted from the working tree as deliberate changes to be committed,
> and files deleted from the git dir can render it unusable.

It's not really that different, it's a variation of the same problem.
I take it that git doesn't warn when cloning on tmpfs, then, so that's
not an issue.
tmpfiles.d snippets can be defined to cleanup on a timer _anything_,
and have done so for ~15 years. Does git currently parse all installed
tmpfiles.d snippets and warn if cloning on an arbitrary directory
covered by one of them?

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Andrey Rakhmatullin]

On Mon, May 06, 2024 at 07:42:11AM -0700, Russ Allbery wrote:
> >> I'm not sure if we have software on long running servers which place
> >> files in /tmp and /var/tmp and expect files to not be deleted during
> >> runtime, even if not accessed for a long time. This is certainly an
> >> issue to be aware of and keep an eye on.
> 
> > Note that FHS mandates it for /var/tmp: "Files and directories located
> > in /var/tmp must not be deleted when the system is booted. Although data
> > stored in /var/tmp is typically deleted in a site-specific manner, it is
> > recommended that deletions occur at a less frequent interval than /tmp."
> 
> It mandates that it not be cleaned on *boot*.  Not that it never be
> cleaned during runtime.  
Ah, I've read that as "while the system is booted".


-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

If it clones into /tmp the *entire* tree will either be reaped (upon
reboot) or not.

But having just some files deleted from a git dir or git working dir
is much more dangerous, because various git commands can treat files
deleted from the working tree as deliberate changes to be committed,
and files deleted from the git dir can render it unusable.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Andrey Rakhmatullin  writes:
> On Mon, May 06, 2024 at 10:40:00AM +0200, Michael Biebl wrote:

>> I'm not sure if we have software on long running servers which place
>> files in /tmp and /var/tmp and expect files to not be deleted during
>> runtime, even if not accessed for a long time. This is certainly an
>> issue to be aware of and keep an eye on.

> Note that FHS mandates it for /var/tmp: "Files and directories located
> in /var/tmp must not be deleted when the system is booted. Although data
> stored in /var/tmp is typically deleted in a site-specific manner, it is
> recommended that deletions occur at a less frequent interval than /tmp."

It mandates that it not be cleaned on *boot*.  Not that it never be
cleaned during runtime.  It anticipates that it be cleaned periodically,
just less frequently than /tmp.

There is a specific prohibition against clearing /var/tmp on reboot
because /var/tmp historically has been used to store temporary files whose
whole reason for existence is that they need to survive a reboot, such as
vi recover files, but are still safe to delete periodically.

Historically, deleting anything in /var/tmp that hadn't been accessed in
over seven days was a perfectly reasonable and typical configuration.
These days, we have the complication that it's fairly common to turn off
atime updates for performance reasons, which makes it a bit harder to
implement that policy when /var/tmp isn't its own partition and thus
inherits that setting from the rest of the system.

-- 
Russ Allbery (r...@debian.org)  

Bug#1070643: ITP: libhdrhistogram -- High Dynamic Range (HDR) Histogram library

[Micke Nordin]

Package: wnpp
Severity: wishlist
Owner: Micke Nordin 
X-Debbugs-Cc: debian-devel@lists.debian.org, k...@sunet.se

* Package name: libhdrhistogram
  Version : 0.11.8
* URL : https://github.com/HdrHistogram/HdrHistogram_c
* License : CC0-1.0 or BSD-2-clause
  Programming Lang: C
  Description : High Dynamic Range (HDR) Histogram library

 This C-based implementation of a High Dynamic Range (HDR) Histogram
 contains a subset of the functionality supported by the Java implementation.
 The current supported features are:
Standard histogram with 64 bit counts (32/16 bit counts not supported)
All iterator types (all values, recorded, percentiles, linear, logarithmic)
Histogram serialisation (encoding version 1.2, decoding 1.0-1.2)
Reader/writer phaser and interval recorder
  Features not supported, but planned
Auto-resizing of histograms
  Features unlikely to be implemented
Double histograms
Atomic/Concurrent histograms
16/32 bit histograms

  This package is needed by Debian Redict Maintainers
   so that we can devendor it from the
  redict package

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 15:31, Barak A. Pearlmutter  wrote:
>
> > What I am looking for right now is packages or internal
> > infrastructure that need
> > an update to cope with these two changes before I upload them, so if
> > you know of any please do let me know and I'll happily look into it
> > and at least file a bug, if not a MR. Thanks.
>
> Okay.
>
> git and other version control systems should be modified to warn if
> the work tree or git dir are in /var/tmp and therefore exposed to
> unexpected deletion of files, and possibly modified to refuse to clone
> there without an explicit --force-unstable-storage option.

Does it do something similar when cloning into /tmp/ or any other
tmpfs right now?

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

> What I am looking for right now is packages or internal
> infrastructure that need
> an update to cope with these two changes before I upload them, so if
> you know of any please do let me know and I'll happily look into it
> and at least file a bug, if not a MR. Thanks.

Okay.

git and other version control systems should be modified to warn if
the work tree or git dir are in /var/tmp and therefore exposed to
unexpected deletion of files, and possibly modified to refuse to clone
there without an explicit --force-unstable-storage option.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon McVittie]

On Mon, 06 May 2024 at 13:41:58 +0100, Barak A. Pearlmutter wrote:
> As someone who regularly deals with large datasets, and keeps them in
> the "approved" don't-back-these-up location /var/tmp

Independent of whether we make the change Luca is suggesting or not,
I don't think /var/tmp is a good recommendation for users storing
temporary data manually, because safe use of "obvious" filenames in a
world-writeable location like /var/tmp requires you to be aware of the
possibility of symlink attacks, and either ensure that your kernel has a
mitigation for them (as Debian's does with default runtime configuration),
or be careful to avoid them, or be OK with any human user account and
any system uid on the system being able to achieve at least denial of
service and maybe privilege escalation to your account.

/tmp and /var/tmp are fine places for *programs* to store temporary data -
for example Flatpak uses a subdirectory in /var/tmp for half-downloaded
apps and runtimes - but only if they're using mkstemp(), mkdtemp()
or equivalent to generate an unpredictable location and allocate it
without race conditions, which is the only way these directories are
safe to use in portable software. This is not particularly user-friendly
to do manually.

For files you don't want backed up, I'd personally recommend storing
them somewhere in your home directory, perhaps ~/tmp, with a CACHEDIR.TAG
(http://justsolve.archiveteam.org/wiki/Cache_Directory_Tagging_Standard),
and using backup software that understands that convention (for example
borg, restic, tar).

smcv

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Richard Lewis]

Luca Boccassi  writes:

> Hence, I am not really looking for philosophical discussions or lists
> of personal preferences or hypotheticals, but for facts: what would
> break where, and how to fix it?

cleaning /tmp or /var/tmp: users may lose files if they dont realise a
  directory tmp can be cleaned without a reboot. something in /var/tmp
  that's been in there for 35 days before an upgrade might be deleted
  before the user reads the NEWS.Debian email, meaning they have no
  chance to react). Maybe you could postpone the very first deletion
  until after the next reboot?

using a tmpfs: is there a risk of losing unrelated data due to more
  frequent OOM killing random other programmes due to /tmp using all the
  memory?  is there a case to only use a tmpfs if the system has
  "enough" memory?

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 13:42, Barak A. Pearlmutter  wrote:
>
> > Then upon reading the release notes, on such a machine, one can simply do:
> >
> > touch /etc/tmpfiles.d/tmp.conf
> >
> > And they get no automated cleanups.
>
> This also disables on-boot cleaning of /tmp/.

Yes, as it's going to be a tmpfs, so it is no longer needed. Trivial
to maintain though if one wants to do so, though.

> The root issue here is that deleting not-read-in-a-while
> but-maybe-stat'ed-recently-by-make-that-doesn't-count files from
> /var/tmp/ by default, particularly when the system didn't used to,
> violates the principle of least surprise.

Which is what release notes are for, if everything was always the same
we wouldn't spend time to put those together.

> There's an old debugging story

While personal anecdotes and stories can be interesting and amusing in
many circumstances, I am not really looking for those at this very
moment. What I am looking for right now is packages or internal
infrastructure that need
an update to cope with these two changes before I upload them, so if
you know of any please do let me know and I'll happily look into it
and at least file a bug, if not a MR. Thanks.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

> Then upon reading the release notes, on such a machine, one can simply do:
>
> touch /etc/tmpfiles.d/tmp.conf
>
> And they get no automated cleanups.

This also disables on-boot cleaning of /tmp/.

The root issue here is that deleting not-read-in-a-while
but-maybe-stat'ed-recently-by-make-that-doesn't-count files from
/var/tmp/ by default, particularly when the system didn't used to,
violates the principle of least surprise. It will cause problems for
end users. And disabling it *properly* requires serious sysadmin
knowledge and is easy to get wrong.

There's an old debugging story: Maclisp on a DEC-10 was glitchy some
days and not others. Someone marked the crashes on a calendar, and
they correlated with the phase of the moon! Turns out the program
printed the lunar phase ("crescent", "waning gibbous", etc) in its
header, just for fun, and when the string was long it wrote past the
end of a buffer, etc.

The story is still told fifty years later because it's so funny, it's
a total violation of levels. People debugging in a windowless basement
should not need to know the phase of the moon!

Users want their systems to run reliably and reproducibly. Getting
different behaviour depending on how long it's been since some of the
files you're using have been *accessed* adds oodles of extra state
people have to keep track of in their mental model of what some
program needs to run. Sure, it can be administratively disabled on one
particular system; but then you run the same program or perform the
same work-flow on another system, and it fails. Something fails
reliably after running for over 30 days ... unless you keep an eye on
it.

As someone who regularly deals with large datasets, and keeps them in
the "approved" don't-back-these-up location /var/tmp/, this just seems
crazy to me. When I tell a grad student to install Debian on their
laptop, do I really need to spend time explaining to them to be sure
to turn off the "delete your old files for no reason" option which is
enabled by default? When they're using someone else's cluster, should
I remind them to run a cron job "find /var/tmp/foo -exec touch ..." so
the 25 Tb of data we pushed onto scratch storage on that cluster
doesn't partly disappear? Our priority should be our users. Our users
are not well served by having their files deleted. They put them there
for a reason!

--Barak.

Bug#1070494: ITP: linux-livepatching -- linux livepatching module for Debian

[Santiago Ruano Rincón]

Package: wnpp
Severity: wishlist
Owner: Emmanuel Arias , Santiago Ruano Rincón 

X-Debbugs-Cc: debian-devel@lists.debian.org, t...@security.debian.org, 
debian-ker...@lists.debian.org, debian-...@lists.debian.org, eam...@debian.org

* Package name: linux-livepatching   
  Version : 0.0.1 
  Upstream Contact: Emmanuel Arias , Santiago Ruano Rincón 

* URL : https://salsa.debian.org/debian/linux-livepatching
* License : GPL-2+
  Programming Lang: C, Shell scripting, Makefile
  Description : linux livepatching module for Debian

Livepatch modules from the Linux Kernel gives the possibility to apply
security fixes to the Kernel while minimizing the need of rebooting the
machine. We can list a lot of cases where users cannot or do not want to
reboot the system. For instance, while running complex scientific
computations, or systems that need to keeping services up and running as much
as possible without interruption. But in those cases, the system needs to be
stable and secure. Livepatching gives that possibility. For now, this package
is a prototype to do a first step to integrate linux livepatching into
Debian


More than an ITP, this is an Intent to Design an Implement.

(CCing debian-lts, since the subject was brought up there some time ago last
year, and there may be people interested. However, this is something that
should be discussed also with the kernel and the security teams. And as Ben
said during an LTS Team meeting, if this idea is implemented in Debian, it
must go through unstable first.)

Other than having serious fun, the goal of this is ITP is to bring livepatching
for security fixes for the kernel that have been available in the Debian
releases. For the moment, we are looking to design and implement a first
approach, that will live in experimental, while we solve the known and unknown
challenges.

# State of the art, what others do

As readers most be aware, some commercial distributions propose linux live
patching.  However, none of their services is freely (as in beer or as in
speech) available.

* RedHat:
Introduced Kpatch in 2014. Kpatch is packaged in debian, but its current status
is not good. It was removed from bullseye (currently only in buster, sid and
experimental)
https://www.redhat.com/en/blog/introducing-kpatch-dynamic-kernel-patching

* Suse
Released kGraft also in 2014, under GPL 2 (and 3 for userspace tools).
According to the wikipedia, it aims at being merged into the kernel, and a
minimalistic design became part of linux since 4.0.

* Ubuntu
Livepatching is included in Ubuntu Pro. No public details about the
implementation. This is offered as a service where it seems the modules are
downloaded directly from a Ubuntu server (not as a package).

# Mainline kernel

Documentation about livepatching support in the mainline kernel can be found
at:
https://www.kernel.org/doc/html/latest/livepatch/livepatch.html
We aim at building on top of it.


# Limitations and known questions 

As discussed with Salvatore some time ago, there are quite some things to 
consider:
* Triaging issues: who and how issues would be triaged?
* Preparing patches: how patches will be selected, backported and etcetera, and
  maintained as a patch stack for specific kernel versions, during the whole 
life
  of a Debian release.
  We don't intend to add any extra load to the already busy Kernel or Security
  teams. We aim at maintaining this as a team though.
* Testing: what are the requirements for the testing infrastructure? What kind
  of machines are needed for testing the patches before publishing them?

Also:
* Patches should be cumulative.
  How long a specific linux version/package would be supported? The goal of
  this project is to make it possible to apply a limited set security issues
  without rebooting. Until when, for how long?
* We limit the initial prototype to non-signed images. Secure Boot does not
  allow to install non-signed modules.
* We limit the initial prototype to amd64.

The initial prototype is based on binary module packaging, contrary to what
other vendors do. We will see how this scale.

Comments and questions are welcome in the ITP bug.

Cheers,

Emmanuel and Santiago

P.S. This is the outcome of one of our conversations during the MiniDebConf in
Santa Fe, Argentina. Thanks to the people that made it possible :-)


signature.asc
Description: PGP signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Marvin Renich]

* Michael Biebl  [240506 07:15]:
> Am 06.05.24 um 12:35 schrieb Simon Richter:
> > Hi,
> > 
> > On 5/6/24 17:40, Michael Biebl wrote:
> > 
> > > If we go with a/, then I think d-i should be updated to no longer
> > > create /tmp as a separate partition.
> > 
> > I think if the admin explicitly configures tmpfs as a separate file
> > system, then that should be honored -- if there is memory pressure,
> > tmpfs+swap is pretty much the worst file system there is.
> > 
> 
> If you explicitly configure /tmp via /etc/fstab it will override the default
> tmp.mount that is shipped by systemd.
> 
> My point is a different one: If we enable /tmp-on-tmpfs by default, then
> offering a partition layout in d-i that places /tmp as a physical partition
> is confusing/inconsistent imho.

I don't think so at all, especially if the description of this option
explains that if you don't choose this option /tmp will be a tmpfs.  But
either way, this is an option that is different from the default, and it
doesn't seem either confusing or inconsistent to me.

...Marvin

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 12:15, Michael Biebl  wrote:
>
> Am 06.05.24 um 12:35 schrieb Simon Richter:
> > Hi,
> >
> > On 5/6/24 17:40, Michael Biebl wrote:
> >
> >> If we go with a/, then I think d-i should be updated to no longer
> >> create /tmp as a separate partition.
> >
> > I think if the admin explicitly configures tmpfs as a separate file
> > system, then that should be honored -- if there is memory pressure,
> > tmpfs+swap is pretty much the worst file system there is.
> >
>
> If you explicitly configure /tmp via /etc/fstab it will override the
> default tmp.mount that is shipped by systemd.
>
> My point is a different one: If we enable /tmp-on-tmpfs by default, then
> offering a partition layout in d-i that places /tmp as a physical
> partition is confusing/inconsistent imho.

Is that the default layout, or a selectable option?

If the latter, to me it makes sense to keep it, maybe with updated
wording. Making things customizable is not really an issue, as long as
it's clear what happens where.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Marvin Renich]

* Simon Richter  [240506 06:51]:
> Hi,
> 
> On 5/6/24 17:40, Michael Biebl wrote:
> 
> > If we go with a/, then I think d-i should be updated to no longer create
> > /tmp as a separate partition.
> 
> I think if the admin explicitly configures tmpfs as a separate file system,
> then that should be honored -- if there is memory pressure, tmpfs+swap is
> pretty much the worst file system there is.

I agree.  It has already been said that /tmp on tmpfs is not right for
all situations (I agree it is a good default for most systems), and
since d-i doesn't do this by default, what benefit is there to removing
that option from the installer?  There is clearly benefit to leaving it
in.

...Marvin

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 11:33, Barak A. Pearlmutter  wrote:
>
> > We have two separate issues here:
>
> > a/ /tmp-on-tmpfs
> > b/ time based clean-up of /tmp and /var/tmp
>
> > I think it makes sense to discuss/handle those separately.
>
> Agreed.
>
> I also don't see any issue with a/, at worst people will be annoyed
> with it for some reason and can then change it back.
>
> > Regarding b/: ...
>
> > The tmpfiles rule tmp.conf as shipped by systemd upstream ...
> > Files that are older then 10 days or 30 days are automatically cleaned up.
>
> This seems like a rather dangerous thing to spring on people.
>
> First of all, time can be pretty fluid on user machines.

Then upon reading the release notes, on such a machine, one can simply do:

touch /etc/tmpfiles.d/tmp.conf

And they get no automated cleanups. This stuff is designed to be
trivially overridable, both by end-users and image builders. What I am
looking for is, what packages need bugs/MRs filed to deal with this
change, if any.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 11:48, Michael Biebl  wrote:
>
> Am 06.05.24 um 12:18 schrieb Luca Boccassi:
> > Defaults are defaults, they are trivially and fully overridable where
> > needed if needed. Especially container and VM managers these days can
> > super trivially override them via SMBIOS Type11 strings or
> > Credentials, ephemerally and without changing the guest image at all.
>
>
> Aligning defaults across distros does have value.
> That said, a distro like Debian has a larger scope than say a desktop
> oriented one like Fedora.
> Debian is used on a broad spectrum of systems: from embedded to server
> to cloud to desktop.
> So I think it is valuable to gather feedback from all affected parties
> to make an informed decision.
>
> What upstream is doing should not be the only driving factor.

It's impossible to have defaults that make everyone happy, there will
always be someone who doesn't like any choice one might pick (there
are people unhappy with the current ones too).

Hence, I am not really looking for philosophical discussions or lists
of personal preferences or hypotheticals, but for facts: what would
break where, and how to fix it? I only found autopkgtest so far, which
uses /tmp/ in the guest and expects it to survive across reboots, and
I have a MR up already for that. Anything else?

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Andrey Rakhmatullin]

On Mon, May 06, 2024 at 10:40:00AM +0200, Michael Biebl wrote:
> I'm not sure if we have software on long running servers which place files
> in /tmp and /var/tmp and expect files to not be deleted during runtime, even
> if not accessed for a long time. This is certainly an issue to be aware of
> and keep an eye on.
Note that FHS mandates it for /var/tmp: "Files and directories located in
/var/tmp must not be deleted when the system is booted. Although data
stored in /var/tmp is typically deleted in a site-specific manner, it is
recommended that deletions occur at a less frequent interval than /tmp."




-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Michael Biebl]

Am 06.05.24 um 12:35 schrieb Simon Richter:

Hi,

On 5/6/24 17:40, Michael Biebl wrote:

If we go with a/, then I think d-i should be updated to no longer 
create /tmp as a separate partition.


I think if the admin explicitly configures tmpfs as a separate file 
system, then that should be honored -- if there is memory pressure, 
tmpfs+swap is pretty much the worst file system there is.




If you explicitly configure /tmp via /etc/fstab it will override the 
default tmp.mount that is shipped by systemd.


My point is a different one: If we enable /tmp-on-tmpfs by default, then 
offering a partition layout in d-i that places /tmp as a physical 
partition is confusing/inconsistent imho.





Regarding b/:
The current setup as used in Debian is to only clean /tmp on boot 
(which is pointless with /tmp-on-tmpfs) and never clean up /var/tmp


There is also the tmpreaper package, which does periodic cleanup based 
on atime. I'd expect that to be installed on a lot of long-running 
machines, it is certainly installed on mine.


Also if /var/tmp is no longer cleaned on boot, that is a bug.


Afaik, /var/tmp has never been cleaned up on /boot.
So I'm not sure what you mean with "no longer"?

Michael


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Epoch bump for bcachefs-tools

[Emilio Pozuelo Monfort]

On 26/04/2024 15:57, Jonathan Carter wrote:

Hi Debian Developers

In January, bcachefs[1] finally made it into the mainline Linux kernel as an 
experimental filesystem in to Linux 6.7


In 2022 something odd happened and the versions releases were 23 and 24 
(previously we had alpha versions in Debian that were just git snapshots) before 
reverting to 1.x.x release tags.


We released those higher numbers in Debian, which means that the current version 
in Debian, bcachefs-tools 1.7.0, has the upstream version number of 24~really1.7.0.


Since it's a filesystem (and experimental one at that), where the version number 
is quite important for both tools and humans alike (who might not understand the 
version number currently in use), I think it would be justified to bump the 
epoch of this package.


I'm mailing debian-devel before commiting such an epoch bump, as per our 
versioning policy[2].


Yesterday I made an upload with the 1.7.0 version to experimental, this version 
uses significantly more rust code compared to the old versions. If it doesn't 
manifest any major issues, I'd like to upload it to unstable along with the 
epoch bump.


That sounds fine. If this was a temporary thing (oops, I uploaded 1.8.0 but I 
have to revert to 1.7.0 since the new version requires kernel x.y) then +really 
is fine. But in this situation, let's just add an epoch and avoid the +really 
for the foreseeable future.


Cheers,
Emilio

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Michael Biebl]

Am 06.05.24 um 12:18 schrieb Luca Boccassi:

Defaults are defaults, they are trivially and fully overridable where
needed if needed. Especially container and VM managers these days can
super trivially override them via SMBIOS Type11 strings or
Credentials, ephemerally and without changing the guest image at all.



Aligning defaults across distros does have value.
That said, a distro like Debian has a larger scope than say a desktop 
oriented one like Fedora.
Debian is used on a broad spectrum of systems: from embedded to server 
to cloud to desktop.
So I think it is valuable to gather feedback from all affected parties 
to make an informed decision.


What upstream is doing should not be the only driving factor.


Michael



OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Samuel Thibault]

Barak A. Pearlmutter, le lun. 06 mai 2024 11:15:35 +0100, a ecrit:
> To me, the purpose of /var/tmp/ when I have my "user" hat on is: a
> place to put files I don't want backed up, particularly large ones,
> and which if I run out of disk space is a place to look for stuff to
> delete. it's not "a place to put files that the system might
> capriciously delete just because."

Definite +1 on this.

My /var/tmp has various "I want to get rid of this at some point"
temporary stuff, and I would be quite annoyed if it would disappear
unexpectedly.

> One compromise would be to enable the /var/tmp/ reaper by default only
> on new installations.

At the very least, yes.

Samuel

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Michael Biebl]

Am 06.05.24 um 12:15 schrieb Barak A. Pearlmutter:

We have two separate issues here:



a/ /tmp-on-tmpfs
b/ time based clean-up of /tmp and /var/tmp



I think it makes sense to discuss/handle those separately.


Agreed.

I also don't see any issue with a/, at worst people will be annoyed
with it for some reason and can then change it back.


I forgot to mention, that while I was sceptical of /tmp-on-tmpfs in the 
past, I nowadays basically enable it everywhere, even on smaller systems 
like PIs.

So it's a change I support.

I did run into odd issues like autopkgtest filling up /tmp and then 
various programs like Firefox behaving oddly and worked around this by 
using TMPDIR=/var/tmp for stuff that needed larger tmp files.








OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon Richter]

Hi,

On 5/6/24 17:40, Michael Biebl wrote:

If we go with a/, then I think d-i should be updated to no longer create 
/tmp as a separate partition.


I think if the admin explicitly configures tmpfs as a separate file 
system, then that should be honored -- if there is memory pressure, 
tmpfs+swap is pretty much the worst file system there is.


For example, I have a VM with 32 threads but only 16 GiB of memory 
allocated to it, which is normally sufficient for compiling large 
projects, but occasionally starts thrashing really hard, and /tmp on 
ext4 with very relaxed ordering rules is a good way to make it behave.


(What we're really want is a file system that behaves like 
FILE_ATTRIBUTE_TEMPORARY on Windows, where backing store is allocated, 
and pages are evicted first on memory pressure, but are not flushed 
automatically)



Regarding b/:
The current setup as used in Debian is to only clean /tmp on boot (which 
is pointless with /tmp-on-tmpfs) and never clean up /var/tmp


There is also the tmpreaper package, which does periodic cleanup based 
on atime. I'd expect that to be installed on a lot of long-running 
machines, it is certainly installed on mine.


Also if /var/tmp is no longer cleaned on boot, that is a bug.

I'm not sure if we have software on long running servers which place 
files in /tmp and /var/tmp and expect files to not be deleted during 
runtime, even if not accessed for a long time. This is certainly an 
issue to be aware of and keep an eye on.


IIRC it was an explicit decision not to install tmpreaper by default and 
leave cleanup policies to the local admin, for precisely this reason.


It makes sense to revisit this decision 20 years later, but a lot of the 
original rationale remains valid: it affects only a handful of machines 
(because everyone else reboots often anyway), and we kind of expect 
those machines to be operated by skilled people.


   Simon

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 09:40, Michael Biebl  wrote:
>
> We have two separate issues here:
>
> a/ /tmp-on-tmpfs
> b/ time based clean-up of /tmp and /var/tmp
>
> I think it makes sense to discuss/handle those separately.
>
> Regarding a/:
> tmp.mount as shipped by systemd uses the following mount options:
> "mode=1777,strictatime,nosuid,nodev,size=50%"
>
> In the past there were concerns that those 50% of available RAM wasn't a
> one-size-fits-all solution, especially for (LXC) containers and VMs
>
> One also needs to keep in mind that debian-installer still offers a
> partitioning setup with /tmp on a separate partition. This will be
> created via an entry in /etc/fstab. Such a /tmp entry in /etc/fstab will
> override tmp.mount.
>
> If we go with a/, then I think d-i should be updated to no longer create
> /tmp as a separate partition.
>
>
> Regarding b/:
> The current setup as used in Debian is to only clean /tmp on boot (which
> is pointless with /tmp-on-tmpfs) and never clean up /var/tmp
>
> The tmpfiles rule tmp.conf as shipped by systemd upstream contains:
>
> q /tmp 1777 root root 10d
> q /var/tmp 1777 root root 30d
>
> Files that are older then 10 days or 30 days are automatically cleaned
> up. The age of the files are determined as such:
>
> "The age of a file system entry is determined from its last modification
> timestamp (mtime), its last access timestamp (atime), and (except for
> directories) its last status change timestamp (ctime). By default, any
> of these three (or two) values will prevent cleanup if it is more recent
> than the current time minus the age field."
>
> I'm not sure if we have software on long running servers which place
> files in /tmp and /var/tmp and expect files to not be deleted during
> runtime, even if not accessed for a long time. This is certainly an
> issue to be aware of and keep an eye on.

Defaults are defaults, they are trivially and fully overridable where
needed if needed. Especially container and VM managers these days can
super trivially override them via SMBIOS Type11 strings or
Credentials, ephemerally and without changing the guest image at all.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

> We have two separate issues here:

> a/ /tmp-on-tmpfs
> b/ time based clean-up of /tmp and /var/tmp

> I think it makes sense to discuss/handle those separately.

Agreed.

I also don't see any issue with a/, at worst people will be annoyed
with it for some reason and can then change it back.

> Regarding b/: ...

> The tmpfiles rule tmp.conf as shipped by systemd upstream ...
> Files that are older then 10 days or 30 days are automatically cleaned up.

This seems like a rather dangerous thing to spring on people.

First of all, time can be pretty fluid on user machines.

Someone has a laptop, they do some work and generate big files in
/var/tmp/, maybe they're editing videos or processing medical images
or downloading and pre-processing large datasets for machine learning.
Now they close their laptop and go on vacation for six weeks. When
they come back they open it and ... shazam all the files they were
working on are deleted.

Another scenario. I download a big source tarball, firefox say. I want
to make a few small changes and compile a custom version. I keep it in
/var/tmp/ for obvious reasons. (The patches are stored elsewhere of
course.) Now I download a new patch and apply it and recompile. And
get an error, because *some* of the files in the unpacked tree were
too old and got auto-deleted.

Imagine suspending a computer in the middle of a big build, and when I
un-suspend it a few weeks later the build fails because some files are
now too old and get deleted. Is this really desirable?

To me, the purpose of /var/tmp/ when I have my "user" hat on is: a
place to put files I don't want backed up, particularly large ones,
and which if I run out of disk space is a place to look for stuff to
delete. it's not "a place to put files that the system might
capriciously delete just because."

I do understand the imperative to keep the filesystem clean and to do
routine maintenance and upkeep without user intervention. But changing
something that used to be "stable storage unless your disk crashes"
into "files placed here can mysteriously disappear" seems like a major
potentially-breaking change that can disrupt common work flows.

I can see popping up a warning: "disk space on the filesystem
containing /var/tmp is getting low and /var/tmp contains xxxGb of
files, hit OKAY to delete the xxxGb of oldest files there" or
something like that. But doing it silently is scary.

One compromise would be to enable the /var/tmp/ reaper by default only
on new installations. People can always turn it on if they want. Also,
when files are deleted, I think the user should get warnings all over
their desktop. I'd also suggest that the reaper by default never
delete files from /var/tmp if there is a lot of space available, or if
/var/tmp isn't consuming much space.

--Barak Pearlmutter

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 06:36, Paul Gevers  wrote:
>
> Hi Luca,
>
> On 05-05-2024 10:04 p.m., Luca Boccassi wrote:
>
>  > Hence, I intend to apply these changes in the next src:systemd upload
>  > to unstable, probably next week.
>
> > In case anybody is aware of packages/programs needing an update to cope
> > with these changes, or any other issue, please let me know and I will
> > file bugs.
>
> You filed MR341 against autopkgtest, thanks for that. Can you please
> hold off a bit longer than only one week to get the infrastructure
> updated? I'm confident that every test that has the needs-reboot
> restriction will start failing (which are only 21 tests according to
> codesearch). However, I fear that every test is at risk under common
> circumstances.
>
> I don't want to be rushed into an autopkgtest update and an
> infrastructure update for something that we can plan (there's always
> risk involved and I want to have the time and peace to cope with the
> process). Having said that, maybe we will be ready next week.

Sure, no problem. That MR fixes autopkgtest for my local runs, let me
know if something else is needed.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Michael Biebl]

Am 05.05.24 um 22:04 schrieb Luca Boccassi:

This will be mentioned in NEWS (and I guess in the release notes when
the time comes), together with the instructions to override for anybody
wanting to keep the old behaviour, which is as trivial as:



..


touch /etc/tmpfiles.d/tmp.conf


This doesn't restore the old/current behaviour, which is to cleanup /tmp 
on boot. For that you would need something like


echo "D /tmp 1777 root root -" > /etc/tmpfiles.d/tmp.conf


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Holger Levsen]

clone 966621 -1
reassign -1 release-notes
thanks

On Mon, May 06, 2024 at 10:40:00AM +0200, Michael Biebl wrote:
> We have two separate issues here:
> 
> a/ /tmp-on-tmpfs
> b/ time based clean-up of /tmp and /var/tmp
> 
> I think it makes sense to discuss/handle those separately.

very much agreed. a.) is mostly "just" a /tmp size issue but b.) can introduce
interesting unforseen breakages for long running stuff.



-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

“I'll tell you what freedom is to me No fear.” (Nina Simone)


signature.asc
Description: PGP signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Michael Biebl]

We have two separate issues here:

a/ /tmp-on-tmpfs
b/ time based clean-up of /tmp and /var/tmp

I think it makes sense to discuss/handle those separately.

Regarding a/:
tmp.mount as shipped by systemd uses the following mount options:
"mode=1777,strictatime,nosuid,nodev,size=50%"

In the past there were concerns that those 50% of available RAM wasn't a 
one-size-fits-all solution, especially for (LXC) containers and VMs


One also needs to keep in mind that debian-installer still offers a 
partitioning setup with /tmp on a separate partition. This will be 
created via an entry in /etc/fstab. Such a /tmp entry in /etc/fstab will 
override tmp.mount.


If we go with a/, then I think d-i should be updated to no longer create 
/tmp as a separate partition.



Regarding b/:
The current setup as used in Debian is to only clean /tmp on boot (which 
is pointless with /tmp-on-tmpfs) and never clean up /var/tmp


The tmpfiles rule tmp.conf as shipped by systemd upstream contains:

q /tmp 1777 root root 10d
q /var/tmp 1777 root root 30d

Files that are older then 10 days or 30 days are automatically cleaned 
up. The age of the files are determined as such:


"The age of a file system entry is determined from its last modification 
timestamp (mtime), its last access timestamp (atime), and (except for 
directories) its last status change timestamp (ctime). By default, any 
of these three (or two) values will prevent cleanup if it is more recent 
than the current time minus the age field."


I'm not sure if we have software on long running servers which place 
files in /tmp and /var/tmp and expect files to not be deleted during 
runtime, even if not accessed for a long time. This is certainly an 
issue to be aware of and keep an eye on.



Michael


OpenPGP_signature.asc
Description: OpenPGP digital signature