Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Luca Boccassi (2024-05-28 01:54:08)
> Thanks for the useful input, the following has been done:
> 
> - existing installations pre-trixie will get an orphaned tmpfiles.d in
> /etc/ that keeps the existing behaviour unchanged (no cleanup of
> /var/tmp)
> - openssh and tmux have been fixed to provide a tmpfiles.d exception
> to retain their respective files
> - the /tmp/ description in debian-installer has been updated to note
> it is a tmpfs by default (via a commit in partman-basicfilesystems,
> will upload if nobody gets around to it before Trixie's freeze)
> - two paragraphs have been provided for the release notes ticket
> - the changes are also noted in NEWS, with instructions on how to
> override locally
> - as mentioned, the latest upload to unstable makes /tmp/ a tmpfs by
> default and for new installations 10+ days old files in /tmp/ and 30+ days
> old files in /var/tmp/ are cleaned up daily

thank you for having discussed this change on d-devel and for adding
documentation to NEWS and release notes to announce this change. I also think
it is sensible to roll this only out on new installations and to keep the
behaviour on existing setups. Thank you for implementing that as well.

That being said, maybe some Perl wizard knows how to do a flock on a directory
in Perl? I tried this:

use Fcntl qw(LOCK_EX);
opendir(my $handle, ".") or die "opendir: $!";
flock($handle, LOCK_EX) or die "flock: $!";

And got this:

flock() on unopened filehandle $handle at test.pl line 8.
(Are you trying to call flock() on dirhandle $handle?)
flock: Bad file descriptor at test.pl line 8.

Wrapping $handle in fileno() doesn't make a difference either. Perl sources
seem to indicate that directory handles are "evil_fh"?

https://sources.debian.org/src/perl/5.38.2-4/util.c/?hl=3783#L3783

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Marc Haber]

On Wed, 29 May 2024 14:35:27 +0300, Hakan Bay?nd?r
 wrote:
>I'll kindly disagree here. I'd rather not have to go back to every 
>system and make sure that they all behave the way I want after doing a 
>periodic, completely boring "apt-get upgrade".

This change is likely to come to the majority of our installations
("stable") with a release upgrade, which is never boring, but one of
the most exciting things that can happen to a Debian stable system.

People doing this responsibly read the release notes before beginning,
and those release notes have in the past contained things that needed
doing manually in the process such as the well-known "please upgrade
kernel first and reboot" during one udev/systemd upgrade.

Ubuntu seems to have put the release notes in an automatism disguise
called do-release-upgrade which probably changes from release to
release regarding what specialty is in the store for this update. We
don't have that, we ask our users to read the release notes.

Greetings
Marc
-- 

Marc Haber |   " Questions are the | Mailadresse im Header
Rhein-Neckar, DE   | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Hakan Bayındır]

> On 29 May 2024, at 17:33, Marvin Renich  wrote:
> 
> * Hakan Bayındır  [240529 07:51]:
>> On 28.05.2024 ÖS 8:16, Andreas Metzler wrote:
>>> On 2024-05-28 Luca Boccassi  
>>> wrote:
>>> [...]
 - existing installations pre-trixie will get an orphaned tmpfiles.d in
 /etc/ that keeps the existing behaviour unchanged (no cleanup of
 /var/tmp)
>>> [...]
>>> 
>>> Hello,
>>> 
>>> I think it is bad choice to deliberately have different behavior for
>>> freshly installed and upgraded systems. Offering upgrades has always
>>> been one of the major selling points of Debian, and imho this
>>> implicitely includes that you do not get a worse or second class Debian
>>> installation when you upgrade it than if you installed from scratch.
>>> 
>>> cu Andreas
>> 
>> I'll kindly disagree here.
> 
> While I agree with Andreas that having the same behavior for upgraded
> systems and new installations is generally better, I agree with you that
> in this specific case it is not the better choice.
> 
>> I'd rather not have to go back to every system
>> and make sure that they all behave the way I want after doing a periodic,
>  ^
>> completely boring "apt-get upgrade".
> 
> You haven't specified what behavior you want.  Is it the existing
> behavior or the new behavior?  This thread is exactly about choosing
> between the two, and possibly between different behavior for new and
> existing systems.
Sorry, I thought that the sentence was self explanatory. Using English as a 
second language has its peculiarities. Let me explain.

1. For existing systems, I don’t want anything modified. Let it be Debian’s old 
defaults, or a custom config I made for that system. IOW, I want my old systems 
to have /tmp as a proper disk partition, and nothing to be cleaned 
periodically, or whatever I set to these systems.

2. For new installation, I’m fine with what Debian proposes. For clarity, I’m 
still against automated cleaning of /tmp and /var/tmp of the reasons I outlined 
before (tl;dr: Long running systems with seldom accessed but required files), 
but I’m fine with whatever Debian decides and ships. At most, I’ll configure 
the behavior I need, but I don’t expect it to be changed down the line by 
package updates.

Hope this clears this part.
> 
> There are four combinations of old/new behavior and upgrading/new
> installation.  Eliminating the obviously bad combination, we are left
> with three:
> 
> A.  Keep old behavior for both upgrading and new installations.
> B.  Keep old behavior for upgrading, use new behavior for new installations.
> C.  Apply new behavior for both.

As I aforementioned, I’m a proponent of “A”, but it’s not favored it seems. So, 
I want to drive the line at “B”. “C” can cause problems because a seasoned 
Debian install is probably old enough to attend school (i.e. 7+ years), and a 
ton of custom configuration is accumulated in /etc, ~/.local, ~/.config and 
elsewhere. Touching working systems and periodically deleting files out of 
nowhere can cause big headaches and worse. 

> The new behavior is preferable for many use cases, but the old behavior
> is not a "BUG" that must be fixed.  Debian has had the old behavior for
> a very long time.
> 
> A number of people have spoken up on this list saying that they are
> relying on the old behavior, and that changing to the new behavior could
> potentially cause serious data loss.

I personally don’t rely on the old behavior, but I work with clusters, and as I 
detailed, some files can linger for a very long time before deleted. These are 
not bugs of these systems, it’s just a side effect of how clusters and long 
running jobs work. Also, RedHat and their derivatives behaves exactly the same 
as how current Debian behaves. /tmp is a partition (which we sometimes mount to 
a high speed NVMe RAID on multi-GPU systems), which is used for data exchange 
between processors, processes, etc., and these files live a pretty long life 
for longer jobs.

> 
> Some people have stated an opinion that keeping upgraded systems in sync
> with the behavior of new installations is desirable.
> 
> So to choose between A, B, and C, we must rank the following:
> 
> X.  desirability of new behavior
> Y.  preventing data loss for an unspecified, but non-zero, number of
>existing systems
> Z.  desirability of having upgraded systems match new installations.
> 
> Both X and Z are primarily opinions with some (non-overwhelming)
> technical merit to each.
> 
> Sufficient technical arguments have been provided on this meta-thread to
> support that Y is highly important and also more important than both X
> and Z.  This means that choice C fails.

So yes, Y is very important for some (small in number, big in footprint) 
installations.

> If Z were more important than X, then the order of importance would
> become Y, then Z, then X, which would make choice A the winner.
> 
> However, there have been no technical arguments 

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Lyndon Brown]

On Wed, 2024-05-29 at 18:58 +0200, Andreas Metzler wrote:
> Hello,
> 
> That is false dichotomy. data-loss will occur when people use /tmp or
> /var/tmp for persistent data-storage because "This has (for a couple
> of
> years) worked on Debian systems" not because "This has (for a couple
> of
> years) worked on *this* *specific* Debian system.".
> 
> cu Andreas

I think you might be missing the point that this is the *DEFAULT*
behaviour. You're free to override it. If you personally have no
precious data in tmp storage to worry about then after upgrade you can
simply tweak the config to enable the new behaviour, as I have done.

I'm using Sid and after doing a little reading up on the topic
yesterday after the upgrade I believe all that needed doing was to
delete the /etc/tmpfiles.d/tmp.conf file, which as I understand it
contains a simple override of the new behaviour defined in
/usr/lib/tmpfiles.d/tmp.conf.

FWIW I think Luca made a perfectly sensible choice here.

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Noah Meyerhans]

On Wed, May 29, 2024 at 06:58:32PM +0200, Andreas Metzler wrote:
> >> I think it is bad choice to deliberately have different behavior for
> >> freshly installed and upgraded systems. Offering upgrades has always
> >> been one of the major selling points of Debian, and imho this
> >> implicitely includes that you do not get a worse or second class Debian
> >> installation when you upgrade it than if you installed from scratch.
> 
> > I strongly disagree: it is a bad choice to change on upgrades a default 
> > which may cause data loss.
> 
> That is false dichotomy. data-loss will occur when people use /tmp or
> /var/tmp for persistent data-storage because "This has (for a couple of
> years) worked on Debian systems" not because "This has (for a couple of
> years) worked on *this* *specific* Debian system.".

Both perspectives are valid, and that is part of why this change is
concerning to me.  Transitioning the filesystem configuration of an
existing system is inherently dangerous and can lead to data loss, as
Marco points out.  But leaving a system to diverge from the default
Debian base configuration leads to configurtion drift that may trigger
obscure bugs, untested configuration, difficult upgrades, etc.  I'm not
convinced the change is worth the risk.

noah

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Andreas Metzler]

On 2024-05-29 Marco d'Itri  wrote:
> On May 28, Andreas Metzler  wrote:

>> I think it is bad choice to deliberately have different behavior for
>> freshly installed and upgraded systems. Offering upgrades has always
>> been one of the major selling points of Debian, and imho this
>> implicitely includes that you do not get a worse or second class Debian
>> installation when you upgrade it than if you installed from scratch.

> I strongly disagree: it is a bad choice to change on upgrades a default 
> which may cause data loss.

Hello,

That is false dichotomy. data-loss will occur when people use /tmp or
/var/tmp for persistent data-storage because "This has (for a couple of
years) worked on Debian systems" not because "This has (for a couple of
years) worked on *this* *specific* Debian system.".

cu Andreas

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Marvin Renich]

* Hakan Bayındır  [240529 07:51]:
> On 28.05.2024 ÖS 8:16, Andreas Metzler wrote:
> > On 2024-05-28 Luca Boccassi  
> > wrote:
> > [...]
> > > - existing installations pre-trixie will get an orphaned tmpfiles.d in
> > > /etc/ that keeps the existing behaviour unchanged (no cleanup of
> > > /var/tmp)
> > [...]
> > 
> > Hello,
> > 
> > I think it is bad choice to deliberately have different behavior for
> > freshly installed and upgraded systems. Offering upgrades has always
> > been one of the major selling points of Debian, and imho this
> > implicitely includes that you do not get a worse or second class Debian
> > installation when you upgrade it than if you installed from scratch.
> > 
> > cu Andreas
> 
> I'll kindly disagree here.

While I agree with Andreas that having the same behavior for upgraded
systems and new installations is generally better, I agree with you that
in this specific case it is not the better choice.

> I'd rather not have to go back to every system
> and make sure that they all behave the way I want after doing a periodic,
  ^
> completely boring "apt-get upgrade".

You haven't specified what behavior you want.  Is it the existing
behavior or the new behavior?  This thread is exactly about choosing
between the two, and possibly between different behavior for new and
existing systems.

There are four combinations of old/new behavior and upgrading/new
installation.  Eliminating the obviously bad combination, we are left
with three:

A.  Keep old behavior for both upgrading and new installations.
B.  Keep old behavior for upgrading, use new behavior for new installations.
C.  Apply new behavior for both.

The new behavior is preferable for many use cases, but the old behavior
is not a "BUG" that must be fixed.  Debian has had the old behavior for
a very long time.

A number of people have spoken up on this list saying that they are
relying on the old behavior, and that changing to the new behavior could
potentially cause serious data loss.

Some people have stated an opinion that keeping upgraded systems in sync
with the behavior of new installations is desirable.

So to choose between A, B, and C, we must rank the following:

X.  desirability of new behavior
Y.  preventing data loss for an unspecified, but non-zero, number of
existing systems
Z.  desirability of having upgraded systems match new installations.

Both X and Z are primarily opinions with some (non-overwhelming)
technical merit to each.

Sufficient technical arguments have been provided on this meta-thread to
support that Y is highly important and also more important than both X
and Z.  This means that choice C fails.

If Z were more important than X, then the order of importance would
become Y, then Z, then X, which would make choice A the winner.

However, there have been no technical arguments whatsoever that Z is
more important than either of X or Y, only a few personal opinions.
And, from the discussion, the consensus appears to be that X is more
important than Z, so the order is Y, then X, then Z.

This gives us choice B as the winner.

It also looks like Luca Boccassi has already made the changes to effect
choice B.  Thank you, Luca!

,,,Marvin

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Hakan Bayındır]

On 28.05.2024 ÖS 8:16, Andreas Metzler wrote:

On 2024-05-28 Luca Boccassi  
wrote:
[...]

- existing installations pre-trixie will get an orphaned tmpfiles.d in
/etc/ that keeps the existing behaviour unchanged (no cleanup of
/var/tmp)

[...]

Hello,

I think it is bad choice to deliberately have different behavior for
freshly installed and upgraded systems. Offering upgrades has always
been one of the major selling points of Debian, and imho this
implicitely includes that you do not get a worse or second class Debian
installation when you upgrade it than if you installed from scratch.

cu Andreas


I'll kindly disagree here. I'd rather not have to go back to every 
system and make sure that they all behave the way I want after doing a 
periodic, completely boring "apt-get upgrade".


Cheers,

H.


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Wed, 29 May 2024 at 08:18, Marc Haber  wrote:
>
> On Wed, 29 May 2024 00:44:29 +0200, Marco d'Itri  wrote:
> >On May 28, Andreas Metzler  wrote:
> >> I think it is bad choice to deliberately have different behavior for
> >> freshly installed and upgraded systems. Offering upgrades has always
> >> been one of the major selling points of Debian, and imho this
> >> implicitely includes that you do not get a worse or second class Debian
> >> installation when you upgrade it than if you installed from scratch.
> >I strongly disagree: it is a bad choice to change on upgrades a default
> >which may cause data loss.
>
> I also think that a change of this kind is release notes material. A
> system having been updated for two decades is bound to carry some
> cruft.


> - two paragraphs have been provided for the release notes ticket

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Marc Haber]

On Wed, 29 May 2024 00:44:29 +0200, Marco d'Itri  wrote:
>On May 28, Andreas Metzler  wrote:
>> I think it is bad choice to deliberately have different behavior for
>> freshly installed and upgraded systems. Offering upgrades has always
>> been one of the major selling points of Debian, and imho this
>> implicitely includes that you do not get a worse or second class Debian
>> installation when you upgrade it than if you installed from scratch.
>I strongly disagree: it is a bad choice to change on upgrades a default 
>which may cause data loss.

I also think that a change of this kind is release notes material. A
system having been updated for two decades is bound to carry some
cruft.

Greetings
Marc
-- 

Marc Haber |   " Questions are the | Mailadresse im Header
Rhein-Neckar, DE   | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Marco d'Itri]

On May 28, Andreas Metzler  wrote:

> I think it is bad choice to deliberately have different behavior for
> freshly installed and upgraded systems. Offering upgrades has always
> been one of the major selling points of Debian, and imho this
> implicitely includes that you do not get a worse or second class Debian
> installation when you upgrade it than if you installed from scratch.
I strongly disagree: it is a bad choice to change on upgrades a default 
which may cause data loss.

-- 
ciao,
Marco


signature.asc
Description: PGP signature

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Andreas Metzler]

On 2024-05-28 Luca Boccassi  
wrote:
[...]
> - existing installations pre-trixie will get an orphaned tmpfiles.d in
> /etc/ that keeps the existing behaviour unchanged (no cleanup of
> /var/tmp)
[...]

Hello,

I think it is bad choice to deliberately have different behavior for
freshly installed and upgraded systems. Offering upgrades has always
been one of the major selling points of Debian, and imho this
implicitely includes that you do not get a worse or second class Debian
installation when you upgrade it than if you installed from scratch.

cu Andreas
-- 
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Matthew Garrett  writes:
> On Mon, May 06, 2024 at 07:42:11AM -0700, Russ Allbery wrote:

>> Historically, deleting anything in /var/tmp that hadn't been accessed
>> in over seven days was a perfectly reasonable and typical
>> configuration.  These days, we have the complication that it's fairly
>> common to turn off atime updates for performance reasons, which makes
>> it a bit harder to implement that policy when /var/tmp isn't its own
>> partition and thus inherits that setting from the rest of the system.

> Apologies for being a bit late to this, but is this true? relatime-type 
> setups will still update atime if the time between the previous update 
> and the access is larger than some threshold, so you lose some degree of 
> granularity but the rough policy should still apply.

You are correct and I completely forgot about that.

-- 
Russ Allbery (r...@debian.org)  

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Matthew Garrett]

On Mon, May 06, 2024 at 07:42:11AM -0700, Russ Allbery wrote:
> Historically, deleting anything in /var/tmp that hadn't been accessed in
> over seven days was a perfectly reasonable and typical configuration.
> These days, we have the complication that it's fairly common to turn off
> atime updates for performance reasons, which makes it a bit harder to
> implement that policy when /var/tmp isn't its own partition and thus
> inherits that setting from the rest of the system.

Apologies for being a bit late to this, but is this true? relatime-type 
setups will still update atime if the time between the previous update 
and the access is larger than some threshold, so you lose some degree of 
granularity but the rough policy should still apply.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Sun, 5 May 2024 at 21:04, Luca Boccassi  wrote:
>
> On Tue, 5 Jul 2022 19:42:37 +0200 Michael Biebl 
> wrote:
> >
> > Hi Eric
> >
> > On Fri, 31 Jul 2020 15:12:48 + Eric Desrochers
> >  wrote:
> > > Package: systemd
> > > Version: 245.7-1
> > > Severity: normal
> > >
> > > Dear Maintainer,
> > >
> > > Debian systemd implementation does not clean
> > > /var/tmp by default.
> > >
> > > * quilt patch:
> > > d/p/debian/Bring-tmpfiles.d-tmp.conf-in-line-with-Debian-
> defaul.patch
> > >
> > > * systemd-245.7/tmpfiles.d/tmp.conf:
> > > #q /var/tmp 1777 root root 30d
> > >
> > > The patch exist in Debian since 2012.
> > >
> > > The topic has been discussed and a few suggestion has been put on
> the
> > > table in the following Ubuntu bug:
> https://launchpad.net/bugs/1870585
> > >
> > > I fill this bug today to start a conversation.
> >
> > I haven't received any further input from your side.
> > Are you still interested in this issue or not?
> > I wonder where to go from here and what to do about this bug report.
>
> I think it's been long enough, and for Trixie we should bring the
> defaults in line with upstream and other distributions, which means:
>
> - /tmp/ is a tmpfs
> - /var/tmp/ is cleaned up on a timer
>
> Hence, I intend to apply these changes in the next src:systemd upload
> to unstable, probably next week.
>
> This will be mentioned in NEWS (and I guess in the release notes when
> the time comes), together with the instructions to override for anybody
> wanting to keep the old behaviour, which is as trivial as:
>
> systemctl mask tmp.mount (or touch /etc/systemd/system/tmp.mount)
> touch /etc/tmpfiles.d/tmp.conf
>
> for the former and the latter respectively.
>
> In case anybody is aware of packages/programs needing an update to cope
> with these changes, or any other issue, please let me know and I will
> file bugs.

Thanks for the useful input, the following has been done:

- existing installations pre-trixie will get an orphaned tmpfiles.d in
/etc/ that keeps the existing behaviour unchanged (no cleanup of
/var/tmp)
- openssh and tmux have been fixed to provide a tmpfiles.d exception
to retain their respective files
- the /tmp/ description in debian-installer has been updated to note
it is a tmpfs by default (via a commit in partman-basicfilesystems,
will upload if nobody gets around to it before Trixie's freeze)
- two paragraphs have been provided for the release notes ticket
- the changes are also noted in NEWS, with instructions on how to
override locally
- as mentioned, the latest upload to unstable makes /tmp/ a tmpfs by
default and for new installations 10+ days old files in /tmp/ and 30+
days old files in /var/tmp/ are cleaned up daily

If anybody wants to spend time to provide a MR to query in
debian-installer whether to optionally customize these changes locally
on installation, I will happily review and merge it.

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

> > what would break where, and how to fix it? I only found autopkgtest
> so
> > far, which uses /tmp/ in the guest and expects it to survive across
> > reboots, and I have a MR up already for that. Anything else?
> 
> Perhaps whatever makes these files in /tmp? i think something to do
> with
> X/gdm/gnome?
> 
> /tmp/.X0-lock
> /tmp/.X1024-lock
> /tmp/.X1025-lock
> 
> /tmp/.X11-unix
> /tmp/.X1-lock
> 
> /tmp/.XIM-unix
> 
> /tmp/.font-unix
> 
> /tmp/.ICE-unix

These are all already covered by /usr/lib/tmpfiles.d/x11.conf

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: systemd-dev package in bookworm?

[Sirius]

In days of yore (Wed, 15 May 2024), Sirius thus quoth: 
> Thank you. I will update later with results for kernel 6.9.0 and Xen
> 4.18.2, how they work together.

Quick feedback: it works, although I am seeing some weird log-spewing when
I run things like aptitude and apt-get search. I will persist, but it is a
bit temperamental.

Quick feedback II: I am getting bitten by a kernel-bug akin to some of
these: [1][2] - and from what I can tell, it is not resolved yet. I need
to look into where and how those locks get triggered and if there is
something I can switch off in the kernel to work around it until the issue
is fixed.

[1]: 
https://bugzilla.kernel.org/buglist.cgi?cmdtype=runnamed_id=1140725=folio_wait_bit_common
[2]: 
https://patchwork.kernel.org/project/fstests/patch/20240418001356.95857-1-mcg...@kernel.org/

-- 
Kind regards,

/S

Re: systemd-dev package in bookworm?

[Sirius]

In days of yore (Wed, 15 May 2024), Simon Richter thus quoth: 
> Hi,

Hello Simon,

> On 5/15/24 10:31, Sirius wrote:
> 
> >Where is the systemd-dev package for regular Bookworm? The only package
> >that show up is systemd-dev/stable-backports 254.5-1~bpo12+3 all and if
> >I try and install that, it seems like it wants to uninstall most of my
> >system in the process.
> 
> The version shipped in bookworm did not yet build a separate -dev package.

Understood. After I cloned the Xen git repo and started looking in more
detail, I did find that Bookworm has a libsystemd-dev package and that
seems to satisfy the Xen build dependencies. So I have a compile running
now of 4.18.2.

> This package was introduced in
> 
> systemd (253-2) experimental; urgency=medium
> 
>   * Add systemd-dev package for pkg-config files. Move systemd.pc and
> udev.pc to systemd-dev, so that packages can build-depend on a small
> package instead of the whole set of binaries. (Closes: #945922,
> #917549)
> 
>  -- Luca Boccassi   Mon, 12 Jun 2023 00:22:52 +0100

Okay, so for the next Stable, there will be a systemd-dev package.

> For older versions, you can find systemd.pc and udev.pc in the main
> systemd package; the other files like interface definitions are not
> shipped at all in the packages in bookworm.
> 
> If you just need the .pc files, just add the old systemd package as an
> alternative to the build dependencies.
> 
> Build-Depends: systemd-dev | systemd (<< 253-2)

Thank you. I will update later with results for kernel 6.9.0 and Xen
4.18.2, how they work together.

-- 
Kind regards,

/S

Re: systemd-dev package in bookworm?

[Simon Richter]

Hi,

On 5/15/24 10:31, Sirius wrote:


   Where is the systemd-dev package for regular Bookworm? The only package
   that show up is systemd-dev/stable-backports 254.5-1~bpo12+3 all and if
   I try and install that, it seems like it wants to uninstall most of my
   system in the process.


The version shipped in bookworm did not yet build a separate -dev package.

This package was introduced in

systemd (253-2) experimental; urgency=medium

  * Add systemd-dev package for pkg-config files. Move systemd.pc and
udev.pc to systemd-dev, so that packages can build-depend on a small
package instead of the whole set of binaries. (Closes: #945922,
#917549)

 -- Luca Boccassi   Mon, 12 Jun 2023 00:22:52 +0100

For older versions, you can find systemd.pc and udev.pc in the main 
systemd package; the other files like interface definitions are not 
shipped at all in the packages in bookworm.


If you just need the .pc files, just add the old systemd package as an 
alternative to the build dependencies.


Build-Depends: systemd-dev | systemd (<< 253-2)

   Simon


OpenPGP_0xEBF67A846AABE354.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Michael Biebl]

Am 13.05.24 um 11:42 schrieb Johannes Schauer Marin Rodrigues:

If we want to try and weigh cost against benefit, do the benefits really
outweigh the cost? How costly is it to carry a patch in Debian and deviate from
upstream versus all the problems that participants of this thread now listed?
My gut feeling is, that the cost of these hard to debug problems is far greater
than continuing to deviate from upstream and carry a Debian-specific patch, no?


Concerning the /var/tmp (and /tmp) cleaning, the Debian specific patch 
is 
https://salsa.debian.org/systemd-team/systemd/-/blob/debian/master/debian/patches/debian/Bring-tmpfiles.d-tmp.conf-in-line-with-Debian-defaul.patch?ref_type=heads


This patch/change was added in 2012 and has required very little to no 
maintenance since that time. So the cost of keeping it is very low 
maintenance wise.


Michael



OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

Unless somebody's already put it there, I'm going to move these
suggestions to a wishlist bug against systemd. Not sure if it should
be one bug or a few, one for each suggestion.

Currently discussion about reaping /var/tmp/ is in
https://bugs.debian.org/966621 and
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1870585 but
these are discussing "should we turn on /var/tmp/ reaping" rather than
"if we do turn it on, should we take measures to make it safer".

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Barak A. Pearlmutter (2024-05-13 10:47:43)
> > I'd like to hear some arguments *in favour* of making this change.
> > Alignment with systemd-upstream, reduced package maintenance burden
> > are two that I can think of, but perhaps I've missed more. These two,
> > IMHO, are significantly outweighed by the risks.
> Let me see if I understand the arguments being made in favor.

thank you, I'd also like to understand them.

> 1. Compatibility with upstream. This means all the upstream logic is sort of
>imported by reference, so the below is mainly the upstream logic, as I
>understand it.

Yes, I also think that there is value in doing the same thing that upstream or
other distros are doing. There is a cost if Debian decides to deviate from what
others decided to be the default.

> 2. Defend the system against buggy programs that leave debris in /var/tmp/,
>and against debris left there when programs are terminated prematurely.
>These are programs which use /var/tmp/ internally, but not as part of
>their API, so the user would have no particular way of knowing that they
>are leaving things there, would have no particular reason to check for and
>delete such files, and might not be able to easily recognize which files
>should be removed.

But I do not understand this as an advantage. In my mind it is quite the
opposite. The buggy programs which leave files in /tmp will now have their bugs
not noticed anymore because the files get cleaned up by systemd. On the other
hand, we are now introducing new bugs in programs which should do an flock on
the temporary directory but do not do so yet. Imagine I would not've read
debian-devel. How would I as the mmdebstrap author have noticed that my tool as
a user of /tmp should set up this flock? I imagine the bug report of somebody
who has a weird problem with the chroot but somehow they are unable to
reproduce it because it depends on the cleanup timing. Are the bugs we are
introducing by regularly cleaning up /tmp not potentially super hard to
diagnose and might thus just not get fixed? Is there an effort to go around and
identify programs we ship with long lasting use of /tmp and is filing bugs so
that they are performing an flock? And at the same time we are now ignoring the
bugs in programs who leave files in /tmp and forget to clean them up. Is this
not a disadvantage of this change instead of an advantage?

> I looked at the upstream bug report
> https://github.com/systemd/systemd/issues/32674 which suggests that deletion
> of directory trees in /var/tmp/ be atomic, and trigger only when everything
> in the tree meets criteria for deletion. I added a comment suggesting that
> the policy be tweaked in two ways. (a) Use system-up time rather than
> wall-clock time for measuring file age, to address the "suspending or
> shutting down for over 30 days breaks running data processing scripts that
> uses /var/tmp/ for intermediate files" issue. I sort of have an invariant in
> my head, which is that suspending the computer doesn't break things, and also
> the whole point of /var/tmp/ is that files there are preserved across boots.
> And (b) check if a file is open by some process, the same way fuser(1) does,
> and if so, don't delete it. Could also preserve directories which are the
> current directory of some process, if you want to be even more user friendly.
> 
> The only response I got was "don't use temporary directories for things that
> you cannot afford to lose and recreate", which I don't really understand.
> It's like saying "you should have good backups, so it's not a problem to just
> delete anything in /home older than two days". Bottom line, it's not clear to
> me that upstream has really thought this through with users in mind. I'd
> suggest that Debian may wish to tweak the defaults on this stuff pretty hard
> to be more user friendly.

Thank you, the suspend issue is another issue that is created by this change.
If we want to try and weigh cost against benefit, do the benefits really
outweigh the cost? How costly is it to carry a patch in Debian and deviate from
upstream versus all the problems that participants of this thread now listed?
My gut feeling is, that the cost of these hard to debug problems is far greater
than continuing to deviate from upstream and carry a Debian-specific patch, no?

> - as discussed earlier, add /tmp/00-README and /var/tmp/00-README to explain
> this old-file-deletion policy

I think this is a really good idea.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

> I'd like to hear some arguments *in favour* of making this change.
> Alignment with systemd-upstream, reduced package maintenance burden
> are two that I can think of, but perhaps I've missed more. These two,
> IMHO, are significantly outweighed by the risks.

Let me see if I understand the arguments being made in favor.

1. Compatibility with upstream. This means all the upstream logic is
sort of imported by reference, so the below is mainly the upstream
logic, as I understand it.

2. Defend the system against buggy programs that leave debris in
/var/tmp/, and against debris left there when programs are terminated
prematurely. These are programs which use /var/tmp/ internally, but
not as part of their API, so the user would have no particular way of
knowing that they are leaving things there, would have no particular
reason to check for and delete such files, and might not be able to
easily recognize which files should be removed.

3. Defend the system against forgetful users, who create files in
/var/tmp/ but neglect to delete them when they're no longer needed.

Unfortunately 2 & 3 cannot be mechanically distinguished, so even if
you wanted to have separate policies for these two classes of files,
it's not really technically possible. So upstream is optimizing for
case (2), and suggests that /var/tmp/ be sort of reserved for programs
and scripts that are aware of this policy, and users not manually
create files there.

I hope that's an accurate characterization.

I looked at the upstream bug report
https://github.com/systemd/systemd/issues/32674 which suggests that
deletion of directory trees in /var/tmp/ be atomic, and trigger only
when everything in the tree meets criteria for deletion. I added a
comment suggesting that the policy be tweaked in two ways. (a) Use
system-up time rather than wall-clock time for measuring file age, to
address the "suspending or shutting down for over 30 days breaks
running data processing scripts that uses /var/tmp/ for intermediate
files" issue. I sort of have an invariant in my head, which is that
suspending the computer doesn't break things, and also the whole point
of /var/tmp/ is that files there are preserved across boots. And (b)
check if a file is open by some process, the same way fuser(1) does,
and if so, don't delete it. Could also preserve directories which are
the current directory of some process, if you want to be even more
user friendly.

The only response I got was "don't use temporary directories for
things that you cannot afford to lose and recreate", which I don't
really understand. It's like saying "you should have good backups, so
it's not a problem to just delete anything in /home older than two
days". Bottom line, it's not clear to me that upstream has really
thought this through with users in mind. I'd suggest that Debian may
wish to tweak the defaults on this stuff pretty hard to be more user
friendly.

Here are a couple cheap tweaks I'd suggest. My hope is that these
would avoid some of the worst case scenarios discussed while still
satisfying the goals 1/2/3 above, and being super easy to implement.

- lengthen the reap time for /var/tmp/ to eight weeks, since Europeans
often take six-week vacations.

- make a "tempdir" command, philosophically similar to tempfile in
debianutils, which creates a fresh directory in /var/tmp/ and drops
the user into a shell with that as the current directory, directory
flocked until that subshell terminates. Could give an optional
directory argument to lock so you can get back into a directory, and
maybe have options to run a program there before or instead of the
subshell.

- following a resume-from-suspend or boot, shut off the
delete-old-files-in-var-tmp mechanism for a while, maybe eight hours
or something like that. Maybe shorten the delay if it doesn't get to
run across multiple resume-or-boot cycles for a week or two.

- as discussed earlier, add /tmp/00-README and /var/tmp/00-README to
explain this old-file-deletion policy

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Bill Allombert]

Le Mon, May 06, 2024 at 11:15:35AM +0100, Barak A. Pearlmutter a écrit :
> > We have two separate issues here:
> 
> > a/ /tmp-on-tmpfs
 
Note that /tmp-on-tmpfs and cleanup-tmp-at-boot are not equivalent.

With cleanup-tmp-at-boot, if your system crashes, you can still backup
/tmp before rebooting.

Cheers,
-- 
Bill. 

Imagine a large red swirl here.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Stéphane Blondon]

Le mar. 7 mai 2024, 20:18,  a écrit :

> Even after a reboot, I would be upset to lose the debug files that I've
> been accumulating for several days while trying to track down an
> intermittent problem with this stupid VPN...
>


At reboot, /tmp isautomatically flushed. It's the default behaviour since
years (at least on physical machines).

-- 
Stephane

Avoiding /var/tmp for long-running compute (was: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default])

[Russ Allbery]

"Jonathan Dowland"  writes:

> Else-thread, Russ begs people to stop doing this. I agree people
> shouldn't! We should also work on education and promotion of the
> alternatives.

Also, helping people use better tools for managing workloads like this
that make their lives easier and have better semantics, thus improving
life for everyone.

I'm suggesting solutions that I don't have time to help implement, and of
course it will take a long time for better tools to filter into all those
clusters, so this doesn't address the immediate problem of this thread
(hence the subject change).  But based on my past experience with these
types of systems, I bet a lot of the patterns captured in software are
older ones.  Linux has a *lot* of facilities today that it didn't have, or
at least weren't widely used, five years ago.  It would be great to help
some of those improvements filter down, because they can make a lot of
these problems go away.

For example, take the case of scratch space for batch computing.  The
logical lifespan for temporary files for a batch computing job is the
lifetime of the job, whatever that may be.  (I know there are exceptions,
but here I'm just talking about defaults.)  Previously one would have to
build support into the batch job management system for creating and
managing those per-job temporary directories, and ensure the jobs support
TMPDIR or other environment variables to control where they store data,
and everyone was doing this independently.  (I've done a *lot* of this
kind of thing, once upon a time.)

But now we have mount namespaces, and systemd has PrivateTmp that builds
on top of that.  So if the job is managed by an execution manager, it can
create per-job temporary directories and it may already support (as
systemd does) the semantics of deleting the contents of those directories
on job exit, and it bind-mounts those into the process space and the
process is none the wiser.  I think all of the desirable glue may not
fully be there (controlling what underlying file system is used for
PrivateTmp, ensuring they're also excluded from normal cleanup, etc.), but
this is very close to a much better way of handling this problem that
still exposes /tmp and /var/tmp to the job so that none of the
often-crufty scientific computing software has to change.

The new capabilities that Linux now has due to namespaces are marvellous
and solve a whole lot of problems that I didn't realize were even
solvable, and right now I suspect there are huge opportunities for
substantial improvements without a whole lot of effort by just plumbing
those facilities through to higher-level layers like batch systems.  Whole
classes of long-standing problems would just disappear, or at least be
far, far easier to manage.

Substantial, substantial caveat: I have been out of this world for a
while, and maybe most of this work has already been done?  That would be
amazing.  The best possible response to this post would be for someone to
tell me I'm five years behind and the batch systems have already picked up
this work and we can just point people at them.

-- 
Russ Allbery (r...@debian.org)  

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Jonathan Dowland]

On Mon May 6, 2024 at 5:01 PM BST, Luca Boccassi wrote:
> On Mon, 6 May 2024 at 16:51, Barak A. Pearlmutter 
> wrote:
> > For whatever reason, a lot of people who process large data use
> > /var/tmp/FOO/ as a place to store information that should not be
> > backed up, but also should not just disappear.
>
> Then such people, assuming they actually exist, can configure their
> custom systems accordingly upon reading the release notes before
> upgrading, as they would do anyway if installing on CentOS or any
> other major OS. Hence, not an issue either.

They actually exist. I'm been one of them, I've worked with many of
them, it's an incredibly common pattern in academic computing at least,
and changing it in Debian should be a very carefully explored decision.

You've pointed out that changing the behaviour from the default, in
either direction, is trivial. The issue is not one of individual
preference but of what is default. The long-established status quo
is not to clean /var/tmp. There is serious risk here: to users data
and correspondingly to Debian's reputation for stability, which
many of us have worked hard to maintain for a very long time.

If you think we need hard data to quantify this practice, then let's
work on a plan for how to gather that going forward, rather than
dismiss this outright because we haven't collected it.

Else-thread, Russ begs people to stop doing this. I agree people
shouldn't! We should also work on education and promotion of the
alternatives.

I'd like to hear some arguments *in favour* of making this change.
Alignment with systemd-upstream, reduced package maintenance burden
are two that I can think of, but perhaps I've missed more. These two,
IMHO, are significantly outweighed by the risks.

Please hold off making this change now and let this discussion continue.


-- 
Please do not CC me for listmail.

  Jonathan Dowland
✎j...@debian.org
   https://jmtd.net

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Emanuele Rocca]

Hi,

On 2024-05-07 09:43, Russ Allbery wrote:
> I understand your point, which is that this pattern is out there in the
> wild and Debian is in danger of breaking existing usage patterns by
> matching the defaults of other distributions.  This is a valid point, and
> I appreciate you making it.

The more general point being that if systems have certain properties,
whether by design or by accident, people tend to rely on them if these
properties are useful for whatever reasons.

In the specific case of /var/tmp in Debian, for a very long time now the
properties have been: (1) persistent, world-writable storage (2) outside
of /home (3) available by default on all systems without any
configuration. To many, these properties make for a good place where
transient-ish work can be done without the risk of losing it upon reboot
(or power loss, or similar). Not being in /home is an important one,
because for instance /home may be regularly backed up, or it may be on a
NFS share, or who knows what else, and you may not want that who knows
why.

All of that being said, I do see the value in uniformity with other
distros, also because it surely makes things easier for maintainers.
And yes, https://xkcd.com/1172/.

It's just that changes are usually a costs/benefits tradeoff -- in the
xkcd a CPU is overheating, whereas in this case the problem to fix seems
a bit less obvious.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Marc Haber]

On Tue, 07 May 2024 22:29:30 +0100, Richard Lewis
 wrote:
>Holger Levsen  writes:
>> I'm a bit surprised how many people seem to really rely on data in /tmp
>> to survive for weeks or even months. I wonder if they backup /tmp?
>
>I use /tmp for things that fall somewhere between "needs a backup" and
>"unimportant, can be deleted whenever".

For me there is a difference between /tmp and /var/tmp. On all my
systems, /tmp has been a tmpfs for decades, /var/tmp being used for
data that is too large for tmpfs.

Even losing /tmp would probably affect some of my running programs
including the X11 session.

Greetings
Marc
-- 

Marc Haber |   " Questions are the | Mailadresse im Header
Rhein-Neckar, DE   | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Simon Richter  writes:
> On 5/8/24 07:05, Russ Allbery wrote:

>> It sounds like that is what kicked off this discussion, but moving /tmp
>> to tmpfs also usually makes programs that use /tmp run faster.  I
>> believe that was the original motivation for tmpfs back in the day.

> IIRC it started out as an implementation of POSIX SHM, and was later
> generalized.

I believe you're correct for Linux specifically but not in general for
UNIX.  For example, I'm fairly sure this is not the case on Solaris, which
was the first place I encountered tmpfs and where tmpfs /tmp was the
default starting in Solaris 2.1 in 1992.  tmpfs was present in SunOS in
1987, so I'm pretty sure it predates POSIX shared memory.

Linux was very, very late to the tmpfs world.

> When /var runs full, the problem is probably initrd building.

I'm not quite sure what to make of this statement.  On my systems, /var
contains all sorts of rather large things, such as PostgreSQL databases,
INN spool files, and mail spools.  I have filled up /var on many systems
over the years, and it's never been by building initrd images.

> Taking a quick look around all my machines, the accumulated cruft in
> /var/tmp is on the order of kilobytes -- mostly reportbug files, and a
> few from audacity -- and these machines have not been reinstalled in the
> last ten years.

Yes, I don't think many programs use it.  I think that's a good thing; the
specific semantics of /var/tmp are only useful in fairly narrow
situations, and overfilling it is fairly dangerous.

Back in the day, /var/tmp was the thing that you used if /tmp was too
small (because it was usually tmpfs).  For example, using sort -T /var/tmp
to sort large files is an old UNIX rune.  And, of course, students would
use it because they ran out of quota in their home directories and then
get upset when their files got deleted automatically, back in the days of
shared UNIX login clusters.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Tue, 7 May 2024 at 17:33, Sam Hartman  wrote:
>
> > "Luca" == Luca Boccassi  writes:
>
> Luca> On Mon, 6 May 2024 at 15:42, Richard Lewis
> Luca>  wrote:
> >>
> >> Luca Boccassi  writes:
> >>
> >> > Hence, I am not really looking for philosophical discussions or
> >> lists > of personal preferences or hypotheticals, but for facts:
> >> what would > break where, and how to fix it?
>
> ssh-agent appears to default to creating a socket under /tmp.
> I think respecting $XDG_RUNTIME_DIR would be better.
>
> /etc/X11/Xsession.d/90x11-common_ssh-agent also doesn't override where
> the socket ends up.
> I definitely think for session scripts like that $XDG_RUNTIME_DIR would
> be better.
>
>
> gnome-keyring's ssh-agent handles this better, although last time I
> checked, it did not support pkcs11, so I could not use it with PIV
> cards.
> (Other parts of gnome-keyring do support pkcs11).

The ssh agent provided by gnupg also behaves correctly and creates the
socket in XDG_RUNTIME_DIR. I have filed a bug for openssh-client's
ssh-agent.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon Richter]

Hi,

On 5/8/24 07:05, Russ Allbery wrote:


It sounds like that is what kicked off this discussion, but moving /tmp to
tmpfs also usually makes programs that use /tmp run faster.  I believe
that was the original motivation for tmpfs back in the day.


IIRC it started out as an implementation of POSIX SHM, and was later 
generalized. The connection to SHM is still there -- POSIX SHM only 
works if a tmpfs is mounted anywhere in the system. Canonically, a tmpfs 
is mounted on /dev/shm for that purpose, but if /tmp is a tmpfs, then 
/dev/shm doesn't need to exist.


I agree that it makes a lot of things run faster (especially compiling, 
which creates temporary files in /tmp), but it has also caused 
situations that required pressing SysRq to resolve (also during compiling).



For /var/tmp, I think the primary motivation to garbage-collect those
files is that filling up /var/tmp is often quite bad for the system.  It's
frequently not on its own partition, but is shared with at least /var, and
filling up /var can be very bad.  It can result in bounced mail, unstable
services, and other serious problems.


When /var runs full, the problem is probably initrd building.

Taking a quick look around all my machines, the accumulated cruft in 
/var/tmp is on the order of kilobytes -- mostly reportbug files, and a 
few from audacity -- and these machines have not been reinstalled in the 
last ten years.


   Simon

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Tue, 7 May 2024 at 22:57, Russ Allbery  wrote:
>
> Richard Lewis  writes:
> > Luca Boccassi  writes:
>
> >> what would break where, and how to fix it?
>
> > Another one for you to investigate: I believe apt source and 'apt-get
> > source' download and extract things into /tmp, as in the mmdebootstap
> > example mentioned by someone else, this will create "old" files that
> > could immediately be flagged for deletion causing surprises.
>
> > (People restoring from backups might also find this an issue)
>
> systemd-tmpfiles respects atime and ctime by default, not just mtime, so I
> think this would only be a problem on file systems that didn't support
> those attributes.  atime is often turned off, but I believe support for
> ctime is fairly universal among the likely file systems for /var/tmp, and
> I believe tmpfs supports all three.  (I'm not 100% sure, though, so please
> correct me if I'm wrong.)

Yes atime/ctime are used too, so things that are really in the process
of being used are not really an issue.

I checked screen and even in bookworm it uses /run/screen/ as you
said, so it's fine.

I checked tmux and indeed it uses /tmp/tmux-UID/ - which is a terrible
choice given it's predictable so if something manages to run first it
can hijack it, but that's really a pre-existing issue. I've filed a
bug to notify that it needs to start flocking the file in /tmp/ while
running to avoid them being deleted while in use.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Richard Lewis  writes:

> btw, i'm not trying to argue against the change, but i dont yet
> understand the rationale (which id like to be put into the
> release-notes): is there perhaps something more compelling than "other
> distributions and upstream already do this"?

It sounds like that is what kicked off this discussion, but moving /tmp to
tmpfs also usually makes programs that use /tmp run faster.  I believe
that was the original motivation for tmpfs back in the day.

For /var/tmp, I think the primary motivation to garbage-collect those
files is that filling up /var/tmp is often quite bad for the system.  It's
frequently not on its own partition, but is shared with at least /var, and
filling up /var can be very bad.  It can result in bounced mail, unstable
services, and other serious problems.

Most modern desktop systems now have large enough drives that this isn't
as much of a concern as it used to be, but VMs often still have quite
small / partitions and put /var/tmp on that partition.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Sven Mueller]

Am 07.05.2024 22:56 schrieb Richard Lewis :Luca Boccassi  writes:



> qwhat would

> break where, and how to fix it?



Another one for you to investigate: I believe apt source and 'apt-get

source' download and extract things into /tmp, as in the mmdebootstap

example mentioned by someone else, this will create "old" files that

could immediately be flagged for deletion causing surprises.`apt download` and `apt source` download to your current working directory. Same for apt-get.
(People restoring from backups might also find this an issue)I would not expect people to restore to /tmp and expecting that restore to work across a reboot. And to be honest, I find the expectation to have any guarantee on files in /tmp or /var/tmp across a reboot quite surprising. The directories are named "tmp" because they are meant as temporary storage, which implies automatic deleting at some point IMHO.Now, bad choices by various tools have been mentioned, so a cleaner for these directories that runs outside a reboot has to be careful anyhow. But during a reboot? I don't think that should be too much of a problem. Cheers, Sven

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Richard Lewis  writes:
> Luca Boccassi  writes:

>> what would break where, and how to fix it?

> Another one for you to investigate: I believe apt source and 'apt-get
> source' download and extract things into /tmp, as in the mmdebootstap
> example mentioned by someone else, this will create "old" files that
> could immediately be flagged for deletion causing surprises.

> (People restoring from backups might also find this an issue)

systemd-tmpfiles respects atime and ctime by default, not just mtime, so I
think this would only be a problem on file systems that didn't support
those attributes.  atime is often turned off, but I believe support for
ctime is fairly universal among the likely file systems for /var/tmp, and
I believe tmpfs supports all three.  (I'm not 100% sure, though, so please
correct me if I'm wrong.)

-- 
Russ Allbery (r...@debian.org)  

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Richard Lewis]

Holger Levsen  writes:

> I'm a bit surprised how many people seem to really rely on data in /tmp
> to survive for weeks or even months. I wonder if they backup /tmp?

I use /tmp for things that fall somewhere between "needs a backup" and
"unimportant, can be deleted whenever". I think all of the issues raised
(disappearing files from git checkouts, old files in unpacked
tarballs/debootstraps/downloads, autopkgtests, 3rd-party-software, bad
choices in tmux/ssh-agent/etc) fall into that spectrum: It's not that
the data is critical, but losing it creates more work -- what is the
reason for accepting that 'risk'?

btw, i'm not trying to argue against the change, but i dont yet
understand the rationale (which id like to be put into the
release-notes): is there perhaps something more compelling than "other
distributions and upstream already do this"?

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Richard Lewis]

Luca Boccassi  writes:

> qwhat would
> break where, and how to fix it?

Another one for you to investigate: I believe apt source and 'apt-get
source' download and extract things into /tmp, as in the mmdebootstap
example mentioned by someone else, this will create "old" files that
could immediately be flagged for deletion causing surprises.

(People restoring from backups might also find this an issue)

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Andrey Rakhmatullin]

On Tue, May 07, 2024 at 09:49:17PM +0200, Johannes Schauer Marin Rodrigues 
wrote:
> Quoting Andrey Rakhmatullin (2024-05-06 19:14:40)
> > On Mon, May 06, 2024 at 04:50:50PM +0100, Barak A. Pearlmutter wrote:
> > > > tmpfiles.d snippets can be defined to cleanup on a timer _anything_,
> > > 
> > > It's a question of what the *default* behaviour should be.
> > > 
> > > For whatever reason, a lot of people who process large data use
> > > /var/tmp/FOO/ as a place to store information that should not be
> > > backed up, but also should not just disappear.
> > To be honest I'm greatly surprised by this idea, and by the suggestion
> > that a lot of people do this, to me this is very similar by that half-joke
> > about people storing useful files in the Recycle Bin.
> 
> I'm doing exactly that. I use paper I already printed stuff on but which were
> misprints or which are no longer useful to me and are just destined for the
> recycling bin to write stuff on that is important to me. After I'm done with 
> my
> work on this scrap paper I decide what I want to keep and copy to permanent
> storage and what I want to really throw away.
I actually meant the Windows feature :)


-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Quoting Andrey Rakhmatullin (2024-05-06 19:14:40)
> On Mon, May 06, 2024 at 04:50:50PM +0100, Barak A. Pearlmutter wrote:
> > > tmpfiles.d snippets can be defined to cleanup on a timer _anything_,
> > 
> > It's a question of what the *default* behaviour should be.
> > 
> > For whatever reason, a lot of people who process large data use
> > /var/tmp/FOO/ as a place to store information that should not be
> > backed up, but also should not just disappear.
> To be honest I'm greatly surprised by this idea, and by the suggestion
> that a lot of people do this, to me this is very similar by that half-joke
> about people storing useful files in the Recycle Bin.

I'm doing exactly that. I use paper I already printed stuff on but which were
misprints or which are no longer useful to me and are just destined for the
recycling bin to write stuff on that is important to me. After I'm done with my
work on this scrap paper I decide what I want to keep and copy to permanent
storage and what I want to really throw away.

I would not like if (suppose I had a person cleaning up my stuff) a cleaning
person came to my desk, saw that there was obvious scrap paper ultimately
destined for the bin and threw that away every once in a while. I might be
taking my notes on what is ultimately for the trash but I want to be the one to
decide when to bring the trash out.

Thank you for this great example! :)

cheers, josch

signature.asc
Description: signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Holger Levsen (2024-05-07 17:22:48)
> On Tue, May 07, 2024 at 04:24:06PM +0300, Hakan Bayındır wrote:
> > Consider a long running task, which will take days or weeks (which is the
> > norm in simulation and science domains in general). System emitted a warning
> > after three days, that it'll delete my files in three days. My job won't be
> > finished, and I'll be losing three days of work unless I catch that warning.
> Then it will be high time you learn not to abuse /tmp that way and work in
> your (or your services) home/data directory.
> 
> Problem easily avoided. plus you don't need to make /tmp 20 TB because you
> have lots of data. ;)
> 
> I'm a bit surprised how many people seem to really rely on data in /tmp to
> survive for weeks or even months. I wonder if they backup /tmp?

I like using /tmp because it's a tmpfs which makes some things faster. For
quite a few things I do not want them to be stored long-term on my SSD so I
resort to using /tmp and not the directory I called ~/tmp inside my $HOME.

This is also not only about data surviving for weeks and months. Elsewhere in
this thread i mentioned mmdebstrap as an application which creates files in
/tmp which have a modification time far in the past. The same happens when
using other tools, for example lets say I want to have a small scratch space
into which I wget some files:

$ wget https://www.debian.org/Pics/debian-logo-1024x576.png
$ stat -c %y debian-logo-1024x576.png
2020-12-17 10:59:08.0 +0100

Will this mean that debian-logo-1024x576.png might accidentally get cleaned up
unless I disable that mechanism? The problem is not limited to people with a
crazy large /tmp either. My system has 3.7 GB of ram and having /tmp be a tmpfs
(even though it's very small) is still beneficial for me because my maximum
read speed from my SSD is 140 MB/s. My small RAM is much faster than that.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

I guess sometimes when people discuss technical matters, good ideas pop up.

(Although I still think that its problematic interactions with lengthy
suspends makes the whole idea of auto-deletion based purely on
timestamps problematic. I can imagine more coherent mechanisms, which
doesn't count time the machine was suspended against the removal
clock, which check if any files are open, check if any process has its
current directory in a tree, which treats trees as a whole instead of
deleting leaves piecemeal only deleting the tree if all its contents
are ripe, etc. That would introduce considerable complexity though.
However, it is the sort of thing a good sysadmin would do before
manually removing stuff in /var/tmp/, so ...)

Re: Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Josh Triplett]

Barak A. Pearlmutter wrote:
> You know, that's a pretty good idea!
>
> Put a 00README-TMP.txt in /tmp/ and /var/tmp/ which briefly states the
> default deletion policy, the policy in place if it's not the default,
> and a pointer to info about altering it. "/tmp's contents are deleted
> at boot while /var/tmp is preserved across rebooting." Maybe in
> /var/tmp suggest /var/scratch/ or /var/cache/tmp or such as a place
> sysadmins might want to set up for not-backed-up but not-auto-deleted
> material.
>
> If the contents aren't dynamic, maybe they could be links to files in
> /usr/share/doc/systemd/.

This seems like a *great* idea. systemd-tmpfiles configuration can
easily create such a file, either with contents or as a symlink to a
documentation file in /usr/share/doc.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[rhys]

It is not "abuse of /tmp" to put files there, even if they need to be there for 
a long time. That is an unnecessary characterization. 

Yes, /tmp gets backed up along with the rest of the system on every VM in my 
environment. 

Sometimes "temporary" CAN mean "weeks or even months." That's not something 
that needs to be determined in advance by someone else. 

This is keeping in mind that I, myself, said that I would be fine with /tmp as 
a tmpfs. I wouldn't cry at all if /tmp were cleaned out at boot time (which 
could be weeks or even months).

But a) at boot time, all processes have been restarted. It would be an 
exceptional process that needs to resume after a reboot using data in /tmp 
(though not impossible) and...

...perhaps more importantly, b) I wouldn't want to declare on behalf of all 
Debian users that doing so Is Wrong And You Should Not Do It. That strikes me 
as unnecessarily presumptuous. 

It's one thing to support a change that MY systems can cope with. It's quite 
another to declare that EVERYONE will now use these rules, particularly when it 
means deleting their files. 

Even after a reboot, I would be upset to lose the debug files that I've been 
accumulating for several days while trying to track down an intermittent 
problem with this stupid VPN...

Sent from my mobile device.


From: Holger Levsen 
Sent: Tuesday, May 7, 2024 10:22
To: debian-devel@lists.debian.org
Subject: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default 
[was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

Then it will be high time you learn not to abuse /tmp that way 

I'm a bit surprised how many people seem to really rely on data in /tmp
to survive for weeks or even months. I wonder if they backup /tmp?


--
cheers,
Holger

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

"When one man dies it's a tragedy. When thousands die it's statistics."
(Stalin commenting the worlds reaction on Covid 19.)

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Marvin Renich]

Early in this meta-thread it was suggested to separate /tmp-is-tmpfs
from cleanup-of-{,/var}/tmp.  I am really surprised that nobody has
suggested the obvious separation of new installs from upgrades.

Changing the local configuration for either feature is trivial either
way.  I think the proposed changes are reasonable for _new_
_installations_.  However, making configuration changes on upgrade that
may potentially cause significant problems, even if only for a small
number of users, and even if they are currently abusing tmp directories,
seems unnecessary to me.

The release notes can say "The defaults for new installations have
changed.  For upgrades, if it will not cause a disruption on your
system, you can get the new behavior by changing these configuration
files"

Personally, I believe (for upgrades) changing tmp-is-tmpfs will have
much less disruptive effect overall than the cleanup of /var/tmp, but I
don't see any reason to force either change on upgrade.

...Marvin

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Hakan Bayındır  writes:

> The applications users use create these temporary files without users'
> knowledge. They work in their own directories, but applications create
> another job dependent state files in both /tmp and /var/tmp. These are
> different programs and I assure you they’re not created there because
> user (or we) configured something. These files live there during the
> lifetime of the job, and cleaned afterwards by the application.

Then someone should fix those applications, because that behavior will
result in user data loss if they're not fixed.  However, first one should
check whether the applications are just honoring TMPDIR or equivalent
variables, in which case TMPDIR on batch systems often should be set to a
user-specific or job-specific persistent directory for exactly this
reason.  That way you can use a user-specific cleanup strategy, such as
purging that directory when all of the user's jobs have finished.

I understand your point, which is that this pattern is out there in the
wild and Debian is in danger of breaking existing usage patterns by
matching the defaults of other distributions.  This is a valid point, and
I appreciate you making it.

My replies are not intended to dispute that point, but to say that the
burden of addressing this buggy behavior should not rest entirely on
Debian.  What the combination of batch system and application is doing is
semantically incorrect and is dangerous, and it really should be fixed.
Even if Debian changes nothing, at some point someone will deploy workers
with a different base operating system and be very surprised when these
files are automatically deleted.

We were automatically cleaning /tmp and /var/tmp on commercial UNIX
systems in 1995 and fixing broken applications that didn't honor TMPDIR.
This is not a new problem.  Nor is having /var/tmp fill up and cause all
sorts of system problems because someone turned off /var/tmp cleaning
while trying to work around broken applications.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Sam Hartman]

> "Luca" == Luca Boccassi  writes:

Luca> On Mon, 6 May 2024 at 15:42, Richard Lewis
Luca>  wrote:
>> 
>> Luca Boccassi  writes:
>> 
>> > Hence, I am not really looking for philosophical discussions or
>> lists > of personal preferences or hypotheticals, but for facts:
>> what would > break where, and how to fix it?

ssh-agent appears to default to creating a socket under /tmp.
I think respecting $XDG_RUNTIME_DIR would be better.

/etc/X11/Xsession.d/90x11-common_ssh-agent also doesn't override where
the socket ends up.
I definitely think for session scripts like that $XDG_RUNTIME_DIR would
be better.


gnome-keyring's ssh-agent handles this better, although last time I
checked, it did not support pkcs11, so I could not use it with PIV
cards.
(Other parts of gnome-keyring do support pkcs11).

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Hakan Bayındır]

Sent from my iPhone

> On 7 May 2024, at 18:39, Holger Levsen  wrote:
> 
> On Tue, May 07, 2024 at 04:24:06PM +0300, Hakan Bayındır wrote:
>> Consider a long running task, which will take days or weeks (which is the
>> norm in simulation and science domains in general). System emitted a warning
>> after three days, that it'll delete my files in three days. My job won't be
>> finished, and I'll be losing three days of work unless I catch that warning.
> 
> Then it will be high time you learn not to abuse /tmp that way and
> work in your (or your services) home/data directory.
> 
> Problem easily avoided. plus you don't need to make /tmp 20 TB because you
> have lots of data. ;)
> 
> I'm a bit surprised how many people seem to really rely on data in /tmp
> to survive for weeks or even months. I wonder if they backup /tmp?
Me is figurative here. Neither me, nor my code nor our users abuse these 
folders. The applications they use create these files without users’ knowledge. 

And yes, these applications rely on the data they saved on /tmp during the job. 
Again, let me repeat. These are not users' files, but applications internal 
data which they automatically create.

And sometimes these /tmp folders are put on high speed internal NVMe RAIDs to 
allow multiple GPUs work together with lower latency, for weeks. 
> -- 
> cheers,
>Holger
> 
> ⢀⣴⠾⠻⢶⣦⠀
> ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
> ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
> ⠈⠳⣄
> 
> "When one man dies it's a tragedy. When thousands die it's statistics." 
> (Stalin commenting the worlds reaction on Covid 19.)

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Hakan Bayındır]

> On 7 May 2024, at 18:57, Russ Allbery  wrote:
> 
> Hakan Bayındır  writes:
>> Dear Russ,
> 
>>> If you are running a long-running task that produces data that you
>>> care about, make a directory for it to use, whether in your home
>>> directory, /opt, /srv, whatever.
> 
>> Sorry but, clusters, batch systems and other automated systems doesn't
>> work that way.
> 
> Yours might not, but I spent 20 years maintaining clusters and batch
> systems and I assure you that's how mine worked.
> 
That’s nice. We’re in it for the same duration. 
>> That's not an extension of the home directory in any way. After users
>> submit their jobs to the cluster, they neither have access to the
>> execution node, nor they can pick and choose where to put their files.
> 
>> These files may stay there up to a couple of weeks, and deleting
>> everything periodically will probably corrupt the jobs of these users
>> somehow.
> 
> Using /var/tmp for this purpose is not a good design decision.
> Directories are free; they can make a new one and point the files of batch
> jobs there.  They don't have to overload a directory that historically has
> different semantics and is often periodically cleared.  I get that this
> may not be your design or something you have control over, so telling you
> this doesn't directly help, but the point still stands.
> 
> Again, obviously the people configuring that cluster can configure it
> however they want, including overriding the /var/tmp cleanup policy.  But
> they're playing with fire by training users to use /var/tmp, and it's
> going to result in someone getting their data deleted at some point,
> regardless of what Debian does.
> 
You still assume that we direct users' home directories to /var/tmp or /tmp. 
This is not true, users work on their own home folders, on a different storage 
system. Possibly I didn’t myself clear enough. 

The applications users use create these temporary files without users' 
knowledge. They work in their own directories, but applications create another 
job dependent state files in both /tmp and /var/tmp. These are different 
programs and I assure you they’re not created there because user (or we) 
configured something. These files live there during the lifetime of the job, 
and cleaned afterwards by the application. 
> -- 
> Russ Allbery (r...@debian.org)  
> 

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Hakan Bayındır  writes:
> Dear Russ,

>> If you are running a long-running task that produces data that you
>> care about, make a directory for it to use, whether in your home
>> directory, /opt, /srv, whatever.

> Sorry but, clusters, batch systems and other automated systems doesn't
> work that way.

Yours might not, but I spent 20 years maintaining clusters and batch
systems and I assure you that's how mine worked.

> That's not an extension of the home directory in any way. After users
> submit their jobs to the cluster, they neither have access to the
> execution node, nor they can pick and choose where to put their files.

> These files may stay there up to a couple of weeks, and deleting
> everything periodically will probably corrupt the jobs of these users
> somehow.

Using /var/tmp for this purpose is not a good design decision.
Directories are free; they can make a new one and point the files of batch
jobs there.  They don't have to overload a directory that historically has
different semantics and is often periodically cleared.  I get that this
may not be your design or something you have control over, so telling you
this doesn't directly help, but the point still stands.

Again, obviously the people configuring that cluster can configure it
however they want, including overriding the /var/tmp cleanup policy.  But
they're playing with fire by training users to use /var/tmp, and it's
going to result in someone getting their data deleted at some point,
regardless of what Debian does.

-- 
Russ Allbery (r...@debian.org)  

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Holger Levsen]

On Tue, May 07, 2024 at 04:24:06PM +0300, Hakan Bayındır wrote:
> Consider a long running task, which will take days or weeks (which is the
> norm in simulation and science domains in general). System emitted a warning
> after three days, that it'll delete my files in three days. My job won't be
> finished, and I'll be losing three days of work unless I catch that warning.

Then it will be high time you learn not to abuse /tmp that way and
work in your (or your services) home/data directory.

Problem easily avoided. plus you don't need to make /tmp 20 TB because you
have lots of data. ;)

I'm a bit surprised how many people seem to really rely on data in /tmp
to survive for weeks or even months. I wonder if they backup /tmp?


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

"When one man dies it's a tragedy. When thousands die it's statistics." 
(Stalin commenting the worlds reaction on Covid 19.)


signature.asc
Description: PGP signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Hakan Bayındır]

Dear Russ,

It's not *me* using /var/tmp for my own temporary files, it's the 
applications other people use. I just logged in one of the nodes we have 
and there were job-dependent files created by a particular, high end 
scientific application (which is developed by another prominent 
company). This is neither in mine or users' control. It's the 
application they use, and I bet it has no setting for that.


> If you are running a long-running task that produces data that you
> care about, make a directory for it to use, whether in your home
> directory, /opt, /srv, whatever.

Sorry but, clusters, batch systems and other automated systems doesn't 
work that way.


> Not as an extension of people's home directory.

That's not an extension of the home directory in any way. After users 
submit their jobs to the cluster, they neither have access to the 
execution node, nor they can pick and choose where to put their files.


These files may stay there up to a couple of weeks, and deleting 
everything periodically will probably corrupt the jobs of these users 
somehow.


I for one understand that all the folders in a UNIX system have historic 
reasons and customs. I prefer to stick to the traditions and 
specifications come with them, however when you run a big system with 
tons of users who use tons of applications, I can't expect every 
software developer under the sun to know, understand and respect that 
conventions.


I just wanted to highlight a very prominent scenario from my vantage 
point, because it's the domain I'm working in.


> Your system is your system, so of course you can configure /var/tmp
> however you want and no one is going to stop you, but a lot of people
> on this thread are describing habits that are going to lose their data
> if they use a different distribution or even a differently-configured
> Debian distribution with tmpreaper installed.

Again, what I'm not describing is a *habit of mine*, but how many of the 
systems I interact with work, and there's no way to change that. I'm 
just pointing out how the systems we work with behave. We don't 
configure them that way. Heck, some of the applications our users use 
have no configuration file whatsoever.


I'm all for progress and a better, self-healing system, but I'm very 
against breaking things while doing that.


Cheers,

H.

On 7.05.2024 ÖS 5:32, Russ Allbery wrote:

Hakan Bayındır  writes:


Consider a long running task, which will take days or weeks (which is
the norm in simulation and science domains in general). System emitted a
warning after three days, that it'll delete my files in three days. My
job won't be finished, and I'll be losing three days of work unless I
catch that warning.


I have to admit that I'm a little surprised at the number of people who
are apparently using /var/tmp for things that are clearly not temporary
files in the traditional UNIX sense.  Clearly this bit of folk knowledge
is not as widespread as I thought, so we have to figure out how to deal
with that, but periodically deleting files out of /var/tmp has been common
(not universal, but common) UNIX practice for at least thirty years.

Whatever we do with /var/tmp retention, I beg people to stop using
/var/tmp for data you're keeping for longer than a few days and care about
losing.  That's not what it's for, and you *will* be bitten by this
someday, somewhere, because even with existing Debian configuration many
people run tmpreaper or similar programs.  If you are running a
long-running task that produces data that you care about, make a directory
for it to use, whether in your home directory, /opt, /srv, whatever.

/var/tmp's primary purpose historically was to support things like
temporary recovery files that needed to survive a system crash, but which
were still expected to be *temporary* in that one would then either use
the recovery file or expect it to be deleted.  Not as an extension of
people's home directory.

Your system is your system, so of course you can configure /var/tmp
however you want and no one is going to stop you, but a lot of people on
this thread are describing habits that are going to lose their data if
they use a different distribution or even a differently-configured Debian
distribution with tmpreaper installed.





OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Tue, 7 May 2024 at 15:53, Sam Hartman  wrote:
>
> > "Johannes" == Johannes Schauer Marin Rodrigues  
> > writes:
> >> > > If [files can be deleted automatically while mmdebstrap is using 
> them],
> >> > > how should applications guard against that from
> >> > > happening?
> >> >
> >> > As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
> >> > flock(2) lock on its chroot's root directory, systemd-tmpfiles should
> >> > fail to take out its own lock on the directory during cleanup, and
> >> > respond to that by treating the directory as "in use" and skipping 
> it.
> >>
> >> That also works, but only as long as mmdebootstrap is actually
> >> running, and as far as I understand it is not a long-running service,
> >> not sure if it works for this use case
>
> Note that according to the man page, ctime is used as well as mtime.
> So for roots that are actually temporary, I don't think much needs to be
> done.
> It won't matter that the mtime might be old because the ctime should be
> consistent with when the root is unpacked.
>
> I do wish there were a way to specify for /var/tmp that directories
> under /var/tmp should be deleted in their entirety or entirely left
> alone.
> I realize we'd have a big debate about whether that was a good default,
> but I'd find it useful for my systems at least.

This is a reasonable RFE, and it has already been proposed some days
ago (in the right place, upstream):
https://github.com/systemd/systemd/issues/32674

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Sam Hartman]

> "Johannes" == Johannes Schauer Marin Rodrigues  writes:
>> > > If [files can be deleted automatically while mmdebstrap is using 
them],
>> > > how should applications guard against that from
>> > > happening?
>> >
>> > As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
>> > flock(2) lock on its chroot's root directory, systemd-tmpfiles should
>> > fail to take out its own lock on the directory during cleanup, and
>> > respond to that by treating the directory as "in use" and skipping it.
>> 
>> That also works, but only as long as mmdebootstrap is actually
>> running, and as far as I understand it is not a long-running service,
>> not sure if it works for this use case

Note that according to the man page, ctime is used as well as mtime.
So for roots that are actually temporary, I don't think much needs to be
done.
It won't matter that the mtime might be old because the ctime should be
consistent with when the root is unpacked.

I do wish there were a way to specify for /var/tmp that directories
under /var/tmp should be deleted in their entirety or entirely left
alone.
I realize we'd have a big debate about whether that was a good default,
but I'd find it useful for my systems at least.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Andrey Rakhmatullin]

On Tue, May 07, 2024 at 04:24:06PM +0300, Hakan Bayındır wrote:
> On the other hand, if we need to change the configuration 99% of the time,
[citation needed]




-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Hakan Bayındır  writes:

> Consider a long running task, which will take days or weeks (which is
> the norm in simulation and science domains in general). System emitted a
> warning after three days, that it'll delete my files in three days. My
> job won't be finished, and I'll be losing three days of work unless I
> catch that warning.

I have to admit that I'm a little surprised at the number of people who
are apparently using /var/tmp for things that are clearly not temporary
files in the traditional UNIX sense.  Clearly this bit of folk knowledge
is not as widespread as I thought, so we have to figure out how to deal
with that, but periodically deleting files out of /var/tmp has been common
(not universal, but common) UNIX practice for at least thirty years.

Whatever we do with /var/tmp retention, I beg people to stop using
/var/tmp for data you're keeping for longer than a few days and care about
losing.  That's not what it's for, and you *will* be bitten by this
someday, somewhere, because even with existing Debian configuration many
people run tmpreaper or similar programs.  If you are running a
long-running task that produces data that you care about, make a directory
for it to use, whether in your home directory, /opt, /srv, whatever.

/var/tmp's primary purpose historically was to support things like
temporary recovery files that needed to survive a system crash, but which
were still expected to be *temporary* in that one would then either use
the recovery file or expect it to be deleted.  Not as an extension of
people's home directory.

Your system is your system, so of course you can configure /var/tmp
however you want and no one is going to stop you, but a lot of people on
this thread are describing habits that are going to lose their data if
they use a different distribution or even a differently-configured Debian
distribution with tmpreaper installed.

-- 
Russ Allbery (r...@debian.org)  

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

> ...3) I would put a file in any auto-cleaned space named "1-AUTOCLEAN.txt" 
> that contains some verbage explaining that things in this directory will be 
> wiped based on rules set in (wherever).

You know, that's a pretty good idea!

Put a 00README-TMP.txt in /tmp/ and /var/tmp/ which briefly states the
default deletion policy, the policy in place if it's not the default,
and a pointer to info about altering it. "/tmp's contents are deleted
at boot while /var/tmp is preserved across rebooting." Maybe in
/var/tmp suggest /var/scratch/ or /var/cache/tmp or such as a place
sysadmins might want to set up for not-backed-up but not-auto-deleted
material.

If the contents aren't dynamic, maybe they could be links to files in
/usr/share/doc/systemd/.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Alexandru Mihail]

> Consider a long running task, which will take days or weeks (which is
> the norm in simulation and science domains in general). System
> emitted a 
> warning after three days, that it'll delete my files in three days.
> My 
> job won't be finished, and I'll be losing three days of work unless I
> catch that warning.
> 
> Now consider these tasks are run on (dark) servers, where users'
> daemons 
> login to run the tasks but users do not. How can the user know? What
> can 
> they do? Same can be said for long running daemons like mail servers,
> CI 
> runners and such.
> 
> One may argue that we can change the configuration, which is true.
> 
You're making a strong argument here, indeed. I personally manage a
horde of bookworm VMs which I really don't want to watch for precisely
minute changes that break the system in subtle ways like this.
Putting /tmp in RAM also won't work, as discussed here. Apart from
SBCs, all other Debian use cases would not benefit from this. There's
plenty of people using cloud virtual machines happily running Debian
with 1 GB of ram, running clamav, mail servers, apache, etc, etc. Let's
not eat up that ram further if we really don't have to. I know because
I am such an user and certainly not the only one. I don't know how this
would impact people who run a boatload of containers with Debian too.

> On the other hand, if we need to change the configuration 99% of the 
> time, why are we making the change to a worse one in the first place?
> 
There's been quite some debate here which is good. This sets us apart
from corporate-run Linux in that there's technical democracy in
decisions impacting users. Maybe listing and weighing, calmly, the pros
and cons of this decision and how bad/good will the impact be on
programs and users could help drive a decision.
For instance, adopting this behavior:
- aligns us with upstream (neutral in my opinion)
- prevents clutter in /var/tmp by misbehaving applications mainly  
(users filling up their drive is their fault; they can still dd of=/  
anyway; we shouldn't put baby wheels on our OS) (good)
- might surprise users who got used to /var/tmp as being a scratch
space since a long time ago and might cause frustration if their files
randomly disappear. Yes, they shouldn't be storing files there anyway,
but deleting them without them explicitly setting that mechanism up or
hitting Y on a big prompt isn't helpful. (regression)
- might require updates to older applications which wrongly use
/var/tmp as discussed above. (neutral, that wasn't a sturdy mechanism
anyway)
Thanks,
Alexandru


signature.asc
Description: This is a digitally signed message part

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[rhys]

The /tmp/ as tmpfs discussion is funny to me because while we've been kicking 
around the idea of whether or not to clean /tmp/, having /tmp/ as a tmpfs makes 
that whole argument moot. It all goes away at boot time! Problem solved! :D

Honestly, I see this one as a much easier topic, assuming that no one is 
talking about changing existing systems. (I haven't seen anyone say that.)

So for new systems, /tmp/ as a tmpfs strikes me as a legitimate option, and the 
partition layout is something that any good admin pays close attention to on 
any new system, particularly a new distribution or even distro version. (That 
is, even going from 12.1 to 12.2, I'm going to be on the lookout for changes in 
the installer.)

Whether you want /tmp as a tmpfs is a decision that's going to be made at the 
same time as whether or not /home should be on a separate partition. The admin 
is going to do whatever makes the most sense for this system. 

To me, it's all about the display. I want to see what partition will be mounted 
as root, what partition will be mounted as /home, which will be swap (if any), 
and so on. But I don't need to see /proc and /sys. Those aren't optional. 

So if /tmp is not part of the root partition, it doesn't matter if it's a 
separate partition or a tmpfs. It should just appear in the screen that 
displays the filesystem layout, and then the admin can decide whether or not 
that's a good idea. 

I have no opinion on whether or not it should be the default. If /tmp/ as tmpfs 
becomes the default, I would probably only override that on certain low-memory 
systems that I run and just leave it on most others. I've seen it done before 
and it seemed to work fine in some cases and not in others. 

As long as it's somewhere that I can SEE it in the installer, I'd be happy. 
That's definitely a thing the admin can change later on with few consequences. 

Sent from my mobile device.


From: Hakan Bayındır 
Sent: Tuesday, May 7, 2024 05:45
To: 966...@bugs.debian.org; debian-devel@lists.debian.org
Cc: Carsten Leonhardt; Luca Boccassi; Peter Pentchev
Subject: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default 
[was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

Similarly, I’m following the thread for a couple of days now, and wondering 
about its implications. 

When I consider server scenarios, pushing /tmp to RAM looks highly undesirable 
from my perspective. All the servers I manage use their whole RAMs and using 
the unused space as a disk cache is far more desirable than a /tmp mount. When 
servers are virtualized, RAM is at premium, and a disk cache is way more usable 
rather than /tmp in the RAM. 

The other scenario I think is HPC, where applications use all the RAM 
available, squeezing the hardware dry. Again, /tmp in the RAM is very 
undesirable, because /tmp/$USER is also heavily used and an OOM situation 
creates a lose-lose situation where you either delete runtime files, or lose 
the executing job, which results in job failure in any case. 

On the other hand, I personally use my desktop computer as a development 
workstation and use tons of RAM either with software or with VMs. Again a /tmp 
in RAM is an inferior scenario to my current setup. 

The only useful state where /tmp is in RAM is single board computers where temp 
is both lightly utilized and maximizing SD/eMMC life is important. These 
systems even mount /var/log to a tmpfs and sync on boot/reboot/shutdown, 
reducing flash wear. 

Deleting /var/tmp has the same problems since long running tasks on the servers 
might need a file once in a month, but it can be crucial for functions of the 
software. 

I can’t see any scenario where these two are useful in typical or niche 
installations of Debian. 

FWIW, RedHat family doesn’t mount /tmp as a tmpfs on its default installation. 

Cheers, 

H. 

Re: Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Alexandru Mihail]

Maybe putting the cleanup task for /var/tmp on a longer timer and
warning users ahead of time of impending deletion (maybe 3 days before,
2 days, etc) would help with files of unsuspecting users getting
deleted. A log entry could also be emitted. I could see a gentle
warning on ssh login (minimal, one or two lines) and desktop
notification (for desktop only users who never see the terminal) be
helpful. A smarter implementation could perhaps only warn if dirs/files
that are going to be deleted are not systemd generated random items.
This does not fix issues with applications depending on stuff being
there long term; yet again nothing's perfect in software


signature.asc
Description: This is a digitally signed message part

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[rhys]

I'm always in favor of logging system changes. 

Notification at run time is really tricky. No one ever logs into several of my 
Debian servers. Other systems have interactive GUI or CLI users, some of whom 
are admins and others not. 

I don't know if login notices are terribly reliable as no one may ever see 
them. I see nothing wrong with having them, but I wouldn't consider them to be 
a good *primary* notification.

Personally, I would resort to email, as it's most likely that a good admin has 
at least set up root mail to go somewhere appropriate. 

Whether or not root should get copied on notifications sent to other users 
strikes me as a security question, though if the data to be deleted is in a 
shared scratch space like /tmp/ then perhaps that's not a concern (and the 
potential for lost data might override any such concerns anyway, plus the fact 
that root is, well, root in the first place). I would argue that the role of 
"root as helpful admin" should prevail in this case and root should get copied. 

I, for one, don't want a lot of email, but once the data is gone, it's GONE. 
I'd rather be notified and have to deal with it than not be told at all. 

Sent from my mobile device.

From: Alexandru Mihail 
Sent: Tuesday, May 7, 2024 07:59
To: debian-devel@lists.debian.org
Subject: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default 
[was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

Maybe putting the cleanup task for /var/tmp on a longer timer and warning users 
ahead of time of impending deletion (maybe 3 days before, 2 days, etc) would 
help with files of unsuspecting users getting deleted. A log entry could also 
be emitted. I could see a gentle warning on ssh login (minimal, one or two 
lines) and desktop notification (for desktop only users who never see the 
terminal) be helpful. A smarter implementation could perhaps only warn if 
dirs/files that are going to be deleted are not systemd generated random items. 
This does not fix issues with applications depending on stuff being there long 
term; yet again nothing's perfect in software 

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Hakan Bayındır]

Consider a long running task, which will take days or weeks (which is 
the norm in simulation and science domains in general). System emitted a 
warning after three days, that it'll delete my files in three days. My 
job won't be finished, and I'll be losing three days of work unless I 
catch that warning.


Now consider these tasks are run on (dark) servers, where users' daemons 
login to run the tasks but users do not. How can the user know? What can 
they do? Same can be said for long running daemons like mail servers, CI 
runners and such.


One may argue that we can change the configuration, which is true.

On the other hand, if we need to change the configuration 99% of the 
time, why are we making the change to a worse one in the first place?


On 7.05.2024 ÖS 3:59, Alexandru Mihail wrote:

Maybe putting the cleanup task for /var/tmp on a longer timer and warning users 
ahead of time of impending deletion (maybe 3 days before, 2 days, etc) would 
help with files of unsuspecting users getting deleted. A log entry could also 
be emitted. I could see a gentle warning on ssh login (minimal, one or two 
lines) and desktop notification (for desktop only users who never see the 
terminal) be helpful. A smarter implementation could perhaps only warn if 
dirs/files that are going to be deleted are not systemd generated random items. 
This does not fix issues with applications depending on stuff being there long 
term; yet again nothing's perfect in software




OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[rhys]

Perhaps ones *intentions* are good, but when you're making decisions for 
someone else, it is often necessary to take a step back and look at the bigger 
picture of what one is doing. 

I do not consider myself to be immune to the effects of this thought process. I 
have kids. They remind me constantly that I'm just as subject to the "I know 
better" mindset as anyone else. 

But you said the best thing to do is "keep an open mind." What Luca has 
literally said multiple times is "I've already made my decision and I'm only 
looking for comments that support the technical aspects of that decision."

That squashes discussion. Only one person has made such comments, and that 
person was not me. 

On a more procedural note...

I have repeatedly voiced my opinion that automatically deleting files is a bad 
idea and I stand by that opinion for reasons already stated. 

However, recognizing that I don't always get my way, I've thought about how one 
might "do it anyway" while still addressing my underlying concerns and here's 
what I came up with:

The two biggest things that are underneath are a) applications that don't clean 
up after themselves and b) a change in how the system behaves vs. what the 
users expect. 

To address a) in a coherent way, if I had infinite resources, I would create a 
package similar to popularity-contest that is obvious and optional and reports 
back what commonly appears in various scratch spaces. That is how I would 
gather a wide range of data on what packages don't behave well. D-i can allow 
the admin to opt in or not on the same screen as the one that asks about 
popularity-contest. (Those that opt in or out will likely do so for that same 
reasons, after all.)

As for b) the underlying problem is the change in *expected behavior* of the 
system. The real problem has nothing to do with whether or not it's technically 
a good idea. It's a shift in expectation with potentially disastrous 
consequences. Deleted files are often just gone. 

So to mitigate that, I would 1) only implement it on new installs (we'll come 
back to this), 2) mention it at least twice in the various d-i screens, and...

...3) I would put a file in any auto-cleaned space named "1-AUTOCLEAN.txt" that 
contains some verbage explaining that things in this directory will be wiped 
based on rules set in (wherever). 

This is how files like /etc/resolv.conf read when they are controlled by other 
processes. They just have text that tells you, "Don't change this directly. 
Your changes will be overwritten. Make your changes in (canonical place)."

Last but not least, I would go ahead and deploy the packages that automatically 
clean tmp spaces even to existing systems, but their default configuration 
would be disabled. The only thing that would enable them would be a) 
debian-installer (optionally, possibly as default), or b) admins who have heard 
about this and decide it's a good idea. 

Some will opt in, some won't. But the packages and their default configs could 
be pushed out safely. No one (even me) would have a reasonable complaint about 
such an arrangement. 

That would allow the expectation to shift over time while significantly 
reducing the number of surprised users who get their data deleted. 

In the end, this is all for people to use. So to me, it's much more an issue of 
making sure the people know what they're using and how to use it than anything 
else. If it were just a technical issue, this would be a much shorter 
conversation. ;)

--J

Sent from my mobile device.


From: "Barak A. Pearlmutter" 
Sent: Tuesday, May 7, 2024 07:18
To: r...@neoquasar.org
Cc: Luca Boccassi; debian-devel@lists.debian.org
Subject: Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default 
[was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

Rhys, I think you're being unfair. We have a *technical* disagreement 
here. But our hearts are all in the same place: Luca, myself, and all 
the other DDs discussing this, all want what's best for our users, we 
all want to build the best OS possible, and are all discussing the 
issue in good faith. 

There is an unavoidable tension, and we're hashing it out. Upstream 
has fielded a default behaviour which requires adjustment of a variety 
of other programs and workflows. Basically, anything that stores stuff 
in /tmp or /var/tmp needs to be made might-be-deleted-aware. There are 
mechanisms for dealing with this, but they're pretty complicated, and 
differ wildly for different file lifetimes etc. Other distributions 
have adopted that default, and rather than using exposed mechanisms 
for avoiding unexpected deletion, are just telling people not to count 
on files in /var/tmp/ surviving a reboot if the computer is shut down 
more than a month, or whatever. What should Debian do? You can make 
arguments both ways, and we are. Generally we follo

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Hakan Bayındır]

Similarly, I’m following the thread for a couple of days now, and wondering 
about its implications.

When I consider server scenarios, pushing /tmp to RAM looks highly undesirable 
from my perspective. All the servers I manage use their whole RAMs and using 
the unused space as a disk cache is far more desirable than a /tmp mount. When 
servers are virtualized, RAM is at premium, and a disk cache is way more usable 
rather than /tmp in the RAM.

The other scenario I think is HPC, where applications use all the RAM 
available, squeezing the hardware dry. Again, /tmp in the RAM is very 
undesirable, because /tmp/$USER is also heavily used and an OOM situation 
creates a lose-lose situation where you either delete runtime files, or lose 
the executing job, which results in job failure in any case.

On the other hand, I personally use my desktop computer as a development 
workstation and use tons of RAM either with software or with VMs. Again a /tmp 
in RAM is an inferior scenario to my current setup.

The only useful state where /tmp is in RAM is single board computers where temp 
is both lightly utilized and maximizing SD/eMMC life is important. These 
systems even mount /var/log to a tmpfs and sync on boot/reboot/shutdown, 
reducing flash wear.

Deleting /var/tmp has the same problems since long running tasks on the servers 
might need a file once in a month, but it can be crucial for functions of the 
software.

I can’t see any scenario where these two are useful in typical or niche 
installations of Debian.

FWIW, RedHat family doesn’t mount /tmp as a tmpfs on its default installation.

Cheers,

H.

> On 7 May 2024, at 12:42, Peter Pentchev  wrote:
> 
> On Tue, May 07, 2024 at 10:38:14AM +0200, Carsten Leonhardt wrote:
>> Luca Boccassi  writes:
>> 
>>> Defaults are defaults, they are trivially and fully overridable where
>>> needed if needed. Especially container and VM managers these days can
>>> super trivially override them via SMBIOS Type11 strings or
>>> Credentials, ephemerally and without changing the guest image at all.
>> 
>> That argument goes both ways and I prefer safe defaults. What
>> you/upstream propose are unsafe defaults, as was shown by several
>> comments in this thread. Whoever wants the unsafe defaults of deleting
>> old files and risking OOM situations can than "trivially and fully
>> override" the safe defaults.
> 
> So I've been wondering for a couple of days now, following this thread...
> ...would it be a good idea to make this a debconf prompt, high priority,
> default "yes", so that it is activated on new automatically installed
> systems, but people who upgrade their current Debian installations can
> choose to keep the old behavior?
> 
> I do realize that more debconf prompts are not always desirable, and
> such decisions must be taken on a case-by-case basis, so... yeah.
> 
> G'luck,
> Peter
> 
> -- 
> Peter Pentchev  r...@ringlet.net r...@debian.org pe...@morpheusly.com
> PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
> Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon McVittie]

On Tue, 07 May 2024 at 07:34:54 -0500, r...@neoquasar.org wrote:
> possibly convince those applications to use their own
> scratch space such as /tmp// that is more easily identifiable

This would be a denial of service at best, and a privilege escalation
vulnerability at worst. To be safe, it would have to be more like
/tmp/.XX where the XX is replaced by a random string
by mkstemp() or similar.

(For example my system currently has /var/tmp/flatpak-cache-5X58M2/ which
is fine, but using /var/tmp/flatpak-cache/ would be wrong.)

smcv

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Alexandru Mihail]

Maybe putting the cleanup task for /var/tmp on a longer timer and warning users 
ahead of time of impending deletion (maybe 3 days before, 2 days, etc) would 
help with files of unsuspecting users getting deleted. A log entry could also 
be emitted. I could see a gentle warning on ssh login (minimal, one or two 
lines) and desktop notification (for desktop only users who never see the 
terminal) be helpful. A smarter implementation could perhaps only warn if 
dirs/files that are going to be deleted are not systemd generated random items. 
This does not fix issues with applications depending on stuff being there long 
term; yet again nothing's perfect in software

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[rhys]

This, in my opinion, is the correct view. 

If the users/admins of a system are putting files somewhere, those are their 
files and therefore their responsibility. It is not up to anyone else to claim 
they know better and clean up after them. 

If the files are abandoned by applications that don't clean up after 
themselves, the applications should be updated to clean up properly, since only 
those applications understand the full context of what is or isn't still "in 
use" or needed (and possibly convince those applications to use their own 
scratch space such as /tmp// that is more easily identifiable). 
Checking the documentation of those applications for descriptions of what ends 
up in [/var]/tmp/ would also be useful, since that informs the admin's decision 
on how to deal with possibly abandoned temp files. 

Making the applications behave properly and trusting the system owners to run 
their systems as they see fit will always be the better choice. 

Otherwise, all that will happen is that over time, another scratch space that 
does not automatically get reaped will appear, users and apps will migrate to 
that space, and we will all be back where we started. (No, they won't just 
"change the defaults" because that's not a stable process. One admin may allow 
that while another doesn't.)

--J

Sent from my mobile device.


From: Philip Hands 
Sent: Tuesday, May 7, 2024 06:31

This makes me wonder what it is that we're expecting to need to delete.

Is this a symptom of sloppy applications that fail to clear up the
debris they create in /var/tmp?  If so, is that not a bug in that
application?

I'd suggest that rather than clearing up after the sloppy behaviour of
buggy applications, we instead leave it visible, in the hope that it can
then be fixed.

Of course, that's obviously not worked in some (many?) cases, so where
we know of problematic packages, could we not add per-package tmpfiles.d
files that name the specific paths that those packages are known to
litter the system with, with appropriate deletion timeouts chosen by the
Maintainer?

That ought to achieve the benefit you're looking for, without hiding
symptoms of future problems with other packages, and without
inconveniencing anyone that's using /var/tmp as scratch space.

Cheers, Phil.
--
Philip Hands -- https://hands.com/~phil

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

Rhys, I think you're being unfair. We have a *technical* disagreement
here. But our hearts are all in the same place: Luca, myself, and all
the other DDs discussing this, all want what's best for our users, we
all want to build the best OS possible, and are all discussing the
issue in good faith.

There is an unavoidable tension, and we're hashing it out. Upstream
has fielded a default behaviour which requires adjustment of a variety
of other programs and workflows. Basically, anything that stores stuff
in /tmp or /var/tmp needs to be made might-be-deleted-aware. There are
mechanisms for dealing with this, but they're pretty complicated, and
differ wildly for different file lifetimes etc. Other distributions
have adopted that default, and rather than using exposed mechanisms
for avoiding unexpected deletion, are just telling people not to count
on files in /var/tmp/ surviving a reboot if the computer is shut down
more than a month, or whatever. What should Debian do? You can make
arguments both ways, and we are. Generally we follow upstream unless
there's a compelling reason not to. You can suggest various strategies
for making things reliable despite following upstream. You can discuss
why maybe upstream should not be followed in this case. This is
precisely the kind of discussion that leads to good decisions, with
everyone keeping an open mind and sharing information and ideas.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Philip Hands]

Luca Boccassi  writes:

> On Mon, 6 May 2024 at 11:33, Barak A. Pearlmutter  wrote:
>>
>> > We have two separate issues here:
>>
>> > a/ /tmp-on-tmpfs
>> > b/ time based clean-up of /tmp and /var/tmp
>>
>> > I think it makes sense to discuss/handle those separately.
>>
>> Agreed.
>>
>> I also don't see any issue with a/, at worst people will be annoyed
>> with it for some reason and can then change it back.
>>
>> > Regarding b/: ...
>>
>> > The tmpfiles rule tmp.conf as shipped by systemd upstream ...
>> > Files that are older then 10 days or 30 days are automatically cleaned up.
>>
>> This seems like a rather dangerous thing to spring on people.
>>
>> First of all, time can be pretty fluid on user machines.
>
> Then upon reading the release notes, on such a machine, one can simply do:
>
> touch /etc/tmpfiles.d/tmp.conf
>
> And they get no automated cleanups. This stuff is designed to be
> trivially overridable, both by end-users and image builders. What I am
> looking for is, what packages need bugs/MRs filed to deal with this
> change, if any.

Isn't this change (as presented) effectively about masking bugs?

We've had people suggesting that implementing this will surprise them
and disrupt their existing use of /var/tmp as scratch storage, and I've
got a lot of sympathy with that, so I'm guessing that the people that
are expected to benefit from this are not those that remember systems
where the main distinction between /tmp and /var/tmp was that /tmp got
emptied at boot time, whereas /var/tmp did not.

That makes me assume that those that would be most likely benefit from
such a change will mainly be users that are never going to type "/tmp"
(with or without a preceding "/var"), and are therefore not going to
have any idea what is being deleting for them, but will be happy never
to get their disk filled.

This makes me wonder what it is that we're expecting to need to delete.

Is this a symptom of sloppy applications that fail to clear up the
debris they create in /var/tmp?  If so, is that not a bug in that
application?

I'd suggest that rather than clearing up after the sloppy behaviour of
buggy applications, we instead leave it visible, in the hope that it can
then be fixed.

Of course, that's obviously not worked in some (many?) cases, so where
we know of problematic packages, could we not add per-package tmpfiles.d
files that name the specific paths that those packages are known to
litter the system with, with appropriate deletion timeouts chosen by the
Maintainer?

That ought to achieve the benefit you're looking for, without hiding
symptoms of future problems with other packages, and without
inconveniencing anyone that's using /var/tmp as scratch space.

Cheers, Phil.
-- 
Philip Hands -- https://hands.com/~phil


signature.asc
Description: PGP signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Peter Pentchev]

On Tue, May 07, 2024 at 10:38:14AM +0200, Carsten Leonhardt wrote:
> Luca Boccassi  writes:
> 
> > Defaults are defaults, they are trivially and fully overridable where
> > needed if needed. Especially container and VM managers these days can
> > super trivially override them via SMBIOS Type11 strings or
> > Credentials, ephemerally and without changing the guest image at all.
> 
> That argument goes both ways and I prefer safe defaults. What
> you/upstream propose are unsafe defaults, as was shown by several
> comments in this thread. Whoever wants the unsafe defaults of deleting
> old files and risking OOM situations can than "trivially and fully
> override" the safe defaults.

So I've been wondering for a couple of days now, following this thread...
...would it be a good idea to make this a debconf prompt, high priority,
default "yes", so that it is activated on new automatically installed
systems, but people who upgrade their current Debian installations can
choose to keep the old behavior?

I do realize that more debconf prompts are not always desirable, and
such decisions must be taken on a case-by-case basis, so... yeah.

G'luck,
Peter

-- 
Peter Pentchev  r...@ringlet.net r...@debian.org pe...@morpheusly.com
PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13


signature.asc
Description: PGP signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Carsten Leonhardt]

Luca Boccassi  writes:

> Defaults are defaults, they are trivially and fully overridable where
> needed if needed. Especially container and VM managers these days can
> super trivially override them via SMBIOS Type11 strings or
> Credentials, ephemerally and without changing the guest image at all.

That argument goes both ways and I prefer safe defaults. What
you/upstream propose are unsafe defaults, as was shown by several
comments in this thread. Whoever wants the unsafe defaults of deleting
old files and risking OOM situations can than "trivially and fully
override" the safe defaults.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[rhys]

You're now at the stage where you're not just MISSING the point of what people 
are trying to tell you, you're actively IGNORING it. 

Automatically deleting files is a bad idea. Those files aren't yours. You don't 
know why they are there. Leave them alone. 

--J

Sent from my mobile device.


From: Luca Boccassi 
Sent: Monday, May 6, 2024 08:20
To: Barak A. Pearlmutter
Cc: debian-devel@lists.debian.org
Subject: Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default 
[was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

On Mon, 6 May 2024 at 13:42, Barak A. Pearlmutter  wrote: 
> 
> > Then upon reading the release notes, on such a machine, one can simply do: 
> > 
> > touch /etc/tmpfiles.d/tmp.conf 
> > 
> > And they get no automated cleanups. 
> 
> This also disables on-boot cleaning of /tmp/. 

Yes, as it's going to be a tmpfs, so it is no longer needed. Trivial 
to maintain though if one wants to do so, though. 

> The root issue here is that deleting not-read-in-a-while 
> but-maybe-stat'ed-recently-by-make-that-doesn't-count files from 
> /var/tmp/ by default, particularly when the system didn't used to, 
> violates the principle of least surprise. 

Which is what release notes are for, if everything was always the same 
we wouldn't spend time to put those together

> There's an old debugging story 

While personal anecdotes and stories can be interesting and amusing in 
many circumstances, I am not really looking for those at this very 
moment. What I am looking for right now is packages or internal 
infrastructure that need 
an update to cope with these two changes before I upload them, so if 
you know of any please do let me know and I'll happily look into it 
and at least file a bug, if not a MR. Thanks. 

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Luca Boccassi  writes:
> Richard Lewis  wrote:

>> - tmux stores sockets in /tmp/tmux-$UID
>> - I think screen might use /tmp/screens

>> I suppose if you detached for a long time you might find yourself
>> unable to reattach.

>> I think you can change the location of these.

> And those are useful only as long as screen/tmux are still running,
> right (I don't really use either that much)? If so, a flock is the right
> solution for these

Also, using /tmp as a path for those sockets was always a questionable
decision.  I believe current versions of screen use /run/screen, which is
a more reasonable location.  Using a per-user directory would be even
better, although I think screen intentionally supports shared screens
between users (which is a somewhat terrifying feature from a security
standpoint, but that's a different argument).

-- 
Russ Allbery (r...@debian.org)  

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Luca Boccassi (2024-05-07 00:09:51)
> To be more specific, as per documentation:
> 
> https://www.freedesktop.org/software/systemd/man/latest/tmpfiles.d.html
> 
> 'x' lines can be used to override cleanup rules, and support globbing,
> so something like:
> 
> x /tmp/mmdebstrap.*

thank you for being patient with me. I saw the man page but I also tried using
codesearch to look for other packages doing the same thing already and was
unable to find one. This made me doubt whether I had understood this correctly.
For example, tmpfiles.d(5) makes it look like there are several minuses
required after the path but apparently those are optional. I've never written
such files before, so your input is useful for me, thank you!

> > And just to confirm (I read this elsewhere in this thread): if my
> > /etc/fstab has an entry for /tmp (with a tmpfs) does this automatically
> > mean that no cleanup will happen or do i still have to put something into
> > /etc to disable the periodic cleanup?
> That's something different, fstab is about whether /tmp is a tmpfs or not,
> cleanups still happen regardless of the filesystem type.

I shall put the following into my local /etc/tmpfiles.d/josch.conf to disable
the cleanup of /tmp completely, then:

x /tmp

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 23:03, Richard Lewis
 wrote:
>
> Luca Boccassi  writes:
>
> > Hence, I am not really looking for philosophical discussions or lists
> > of personal preferences or hypotheticals, but for facts: what would
> > break where, and how to fix it?
>
> - tmux stores sockets in /tmp/tmux-$UID
> - I think screen might use /tmp/screens
>
> I suppose if you detached for a long time you might find yourself unable
> to reattach.
>
> I think you can change the location of these.

And those are useful only as long as screen/tmux are still running,
right (I don't really use either that much)? If so, a flock is the
right solution for these

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 23:00, Johannes Schauer Marin Rodrigues
 wrote:
>
> Quoting Luca Boccassi (2024-05-06 23:28:59)
> > On Mon, 6 May 2024 at 22:27, Simon McVittie  wrote:
> > >
> > > On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues 
> > > wrote:
> > > > If [files can be deleted automatically while mmdebstrap is using them],
> > > > how should applications guard against that from
> > > > happening?
> > >
> > > As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
> > > flock(2) lock on its chroot's root directory, systemd-tmpfiles should
> > > fail to take out its own lock on the directory during cleanup, and
> > > respond to that by treating the directory as "in use" and skipping it.
> >
> > That also works, but only as long as mmdebootstrap is actually
> > running, and as far as I understand it is not a long-running service,
> > not sure if it works for this use case
>
> I guess those users who want to place their chroots in /tmp would have to
> disable the cleanup timer on their systems.
>
> For the mmdebstrap user who just wants to create some tarballs, I think the
> flock solution would be sufficient, thank you.
>
> > More specific settings win, so your application can ship its own tmpfiles.d
> > snippet that defines a longer (or infinite) cleanup time for directories
> > under /tmp or /var/tmp.
>
> How would that look like? Say, my application creates temporary directories
> with the pattern /tmp/mmdebstrap. -- how would the recommended line
> which disables cleanup in /usr/lib/tmpfiles.d/mmdebstrap.conf look like?

To be more specific, as per documentation:

https://www.freedesktop.org/software/systemd/man/latest/tmpfiles.d.html

'x' lines can be used to override cleanup rules, and support globbing,
so something like:

x /tmp/mmdebstrap.*

> And just to confirm (I read this elsewhere in this thread): if my /etc/fstab
> has an entry for /tmp (with a tmpfs) does this automatically mean that no
> cleanup will happen or do i still have to put something into /etc to disable
> the periodic cleanup?

That's something different, fstab is about whether /tmp is a tmpfs or
not, cleanups still happen regardless of the filesystem type.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Quoting Luca Boccassi (2024-05-06 23:28:59)
> On Mon, 6 May 2024 at 22:27, Simon McVittie  wrote:
> >
> > On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues 
> > wrote:
> > > If [files can be deleted automatically while mmdebstrap is using them],
> > > how should applications guard against that from
> > > happening?
> >
> > As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
> > flock(2) lock on its chroot's root directory, systemd-tmpfiles should
> > fail to take out its own lock on the directory during cleanup, and
> > respond to that by treating the directory as "in use" and skipping it.
> 
> That also works, but only as long as mmdebootstrap is actually
> running, and as far as I understand it is not a long-running service,
> not sure if it works for this use case

I guess those users who want to place their chroots in /tmp would have to
disable the cleanup timer on their systems.

For the mmdebstrap user who just wants to create some tarballs, I think the
flock solution would be sufficient, thank you.

> More specific settings win, so your application can ship its own tmpfiles.d
> snippet that defines a longer (or infinite) cleanup time for directories
> under /tmp or /var/tmp.

How would that look like? Say, my application creates temporary directories
with the pattern /tmp/mmdebstrap. -- how would the recommended line
which disables cleanup in /usr/lib/tmpfiles.d/mmdebstrap.conf look like?

And just to confirm (I read this elsewhere in this thread): if my /etc/fstab
has an entry for /tmp (with a tmpfs) does this automatically mean that no
cleanup will happen or do i still have to put something into /etc to disable
the periodic cleanup?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Richard Lewis]

Luca Boccassi  writes:

> Hence, I am not really looking for philosophical discussions or lists
> of personal preferences or hypotheticals, but for facts: what would
> break where, and how to fix it?

- tmux stores sockets in /tmp/tmux-$UID
- I think screen might use /tmp/screens

I suppose if you detached for a long time you might find yourself unable
to reattach.

I think you can change the location of these.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 22:27, Simon McVittie  wrote:
>
> On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues wrote:
> > If [files can be deleted automatically while mmdebstrap is using them],
> > how should applications guard against that from
> > happening?
>
> As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
> flock(2) lock on its chroot's root directory, systemd-tmpfiles should
> fail to take out its own lock on the directory during cleanup, and
> respond to that by treating the directory as "in use" and skipping it.

That also works, but only as long as mmdebootstrap is actually
running, and as far as I understand it is not a long-running service,
not sure if it works for this use case

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 21:08, Johannes Schauer Marin Rodrigues
 wrote:
>
> Hi,
>
> Quoting Luca Boccassi (2024-05-06 15:20:08)
> > While personal anecdotes and stories can be interesting and amusing in many
> > circumstances, I am not really looking for those at this very moment. What I
> > am looking for right now is packages or internal infrastructure that need an
> > update to cope with these two changes before I upload them, so if you know 
> > of
> > any please do let me know and I'll happily look into it and at least file a
> > bug, if not a MR. Thanks.
>
> mmdebstrap uses $TMPDIR (because that's what Perl's File::Temp uses) and falls
> back to /tmp for storing the chroot directories it creates. The modification
> times on files in that chroot directory are potentially much older than 10 
> days
> as they match the timestamps on the packages installed in the chroot. Do I
> understand it correctly that with the proposed change and an unfortunate 
> timing
> it can happen that files older than 10 days get removed from a temporary 
> chroot
> directory in /tmp? If yes, how should applications guard against that from
> happening?

More specific settings win, so your application can ship its own
tmpfiles.d snippet that defines a longer (or infinite) cleanup time
for directories under /tmp or /var/tmp.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon McVittie]

On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues wrote:
> If [files can be deleted automatically while mmdebstrap is using them],
> how should applications guard against that from
> happening?

As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
flock(2) lock on its chroot's root directory, systemd-tmpfiles should
fail to take out its own lock on the directory during cleanup, and
respond to that by treating the directory as "in use" and skipping it.

smcv

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 21:30, Salvo Tomaselli  wrote:
>
> On a fresh installed fedora system I downloaded a .iso in /tmp, then the
> OOMkiller killed wayland, so everything died.
>
> If you know you won't ever fill it up, I guess it's fine. But I'd go for the
> safer (and sadly slower) option.

You can just configure it (or disable it) on older devices or where
you don't want it, it's very easy to do as already mentioned

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Richard Lewis]

Luca Boccassi  writes:

> On Mon, 6 May 2024 at 15:42, Richard Lewis
>  wrote:
>>
>> Luca Boccassi  writes:
>>
>> > Hence, I am not really looking for philosophical discussions or lists
>> > of personal preferences or hypotheticals, but for facts: what would
>> > break where, and how to fix it?
>>
>> cleaning /tmp or /var/tmp: users may lose files if they dont realise a
>>   directory tmp can be cleaned without a reboot. something in /var/tmp
>>   that's been in there for 35 days before an upgrade might be deleted
>>   before the user reads the NEWS.Debian email, meaning they have no
>>   chance to react). Maybe you could postpone the very first deletion
>>   until after the next reboot?
>>
>> using a tmpfs: is there a risk of losing unrelated data due to more
>>   frequent OOM killing random other programmes due to /tmp using all the
>>   memory?  is there a case to only use a tmpfs if the system has
>>   "enough" memory?
>
> Again, those are all hypotheticals, and one can construct similarly
> contrived thought exercises for most possible permutations of most
> configurations, and the answer is always the same: customize the
> configuration accordingly. Hence, not relevant right now.

> What is relevant is: which packages, if any, or which DSA-owned
> systems, if any, are actually affected and how?

Why do you think that the impact on users is less important than the
impact on debian packages?

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Luca Boccassi (2024-05-06 15:20:08)
> While personal anecdotes and stories can be interesting and amusing in many
> circumstances, I am not really looking for those at this very moment. What I
> am looking for right now is packages or internal infrastructure that need an
> update to cope with these two changes before I upload them, so if you know of
> any please do let me know and I'll happily look into it and at least file a
> bug, if not a MR. Thanks.

mmdebstrap uses $TMPDIR (because that's what Perl's File::Temp uses) and falls
back to /tmp for storing the chroot directories it creates. The modification
times on files in that chroot directory are potentially much older than 10 days
as they match the timestamps on the packages installed in the chroot. Do I
understand it correctly that with the proposed change and an unfortunate timing
it can happen that files older than 10 days get removed from a temporary chroot
directory in /tmp? If yes, how should applications guard against that from
happening?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Andrey Rakhmatullin]

On Mon, May 06, 2024 at 04:50:50PM +0100, Barak A. Pearlmutter wrote:
> > tmpfiles.d snippets can be defined to cleanup on a timer _anything_,
> 
> It's a question of what the *default* behaviour should be.
> 
> For whatever reason, a lot of people who process large data use
> /var/tmp/FOO/ as a place to store information that should not be
> backed up, but also should not just disappear.
To be honest I'm greatly surprised by this idea, and by the suggestion
that a lot of people do this, to me this is very similar by that half-joke
about people storing useful files in the Recycle Bin.
But I also thought that "/tmp is on tmpfs, /var/tmp isn't, and that's
their main difference" are our defaults for years, when they apparently
aren't (now I understand why people sometimes suggest using /dev/shm or
/run for this or that to increase performance).

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 16:51, Barak A. Pearlmutter  wrote:
>
> > tmpfiles.d snippets can be defined to cleanup on a timer _anything_,
>
> It's a question of what the *default* behaviour should be.

No, it is not, at least not for the strawman you conjured. So I gather
that git doesn't warn when cloning on other directories that might be
cleaned up - great, that's a non-issue too.
If you want it to do so, then the appropriate way is for you to file
an RFE upstream and ask them to parse the installed tmpfiles.d on a
system before cloning. This is not that hard to do, and the format is
stable, and if they ask about that I'll happily back you up on this
stability guarantee.
But what git does or does not allow is then completely unrelated to
anything happening here.

> For whatever reason, a lot of people who process large data use /var/tmp/FOO/ 
> as a place to store information that should not be backed up, but also should 
> not just disappear.

Then such people, assuming they actually exist, can configure their
custom systems accordingly upon reading the release notes before
upgrading, as they would do anyway if installing on CentOS or any
other major OS. Hence, not an issue either.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

> tmpfiles.d snippets can be defined to cleanup on a timer _anything_,

It's a question of what the *default* behaviour should be.

For whatever reason, a lot of people who process large data use
/var/tmp/FOO/ as a place to store information that should not be
backed up, but also should not just disappear.

Maybe they should use ~/FOO/ but put a CACHEDIR.TAG file with the
right magic contents there. Or touch ~/FOO/.nobackup and use duplicity
--exclude-if-present .nobackup. Or ~/.cache/FOO/, although that's a
bit counterintuitive because it's "hidden" and not really meant for
this purpose.

There's a reason people don't want these files below ~/, which is that
some backup mechanisms, like btrfs snapshots or LVMs snapshots, may
snapshot files there regardless; and often /home/ is on more
stable/expensive storage (raid k for k≥1) while /var/tmp/ is on a
cheaper substrate.

Given the era of large data we're living in, this is a ubiquitous
problem. Since we don't give users a standard solution, they've rolled
their own. If we think that solution is wrong, we should (a) make an
actual official suggested location for this sort of material,
/var/scratch/ or /var/scratch/${USER}/ with a link there from
~/Scratch or something, and (b) transition them to it, ideally without
unexpectedly deleting their files and messing up their work in the
process.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 16:42, Simon Richter  wrote:
>
> Hi,
>
> On 5/6/24 19:57, Michael Biebl wrote:
>
> > Afaik, /var/tmp has never been cleaned up on /boot.
> > So I'm not sure what you mean with "no longer"?
>
> Oof, you're right, it was /tmp, /var/run, /var/lock:
>
>  [ "$VERBOSE" != no ] && echo -n "Cleaning"
>  [ -d /tmp ] && cleantmp
>  [ -d /var/run ] && cleanrun
>  [ -d /var/lock ] && cleanlock
>  [ "$VERBOSE" != no ] && echo "."
>
> Would it make sense to make it a bug for a package to use /var/tmp (on
> my system, I can see files from audacity and reportbug there) and
> declare that this directory is for the use of the sysadmin only?

In general it should be the opposite, it's fine for packages to use
/tmp or /var/tmp (using appropriate measures such as random file/dir
names to avoid clashes and attacks), as long as it's really for
temporary stuff, as in, if they are lost it's not a problem and they
are recreated. If you are storing your precious irreplaceable state on
/tmp or /var/tmp, then you are doing it wrong (TM).
In general it's users who should be careful and know what they are
doing, especially due to the world-writable nature of both
directories.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 16:30, Simon Richter  wrote:
>
> Hi,
>
> On 5/6/24 20:19, Luca Boccassi wrote:
>
> > Is that the default layout, or a selectable option?
>
> When you create a partition manually, it asks for the mount point, and
> makes a number of suggestions in a dropdown, and /tmp is one of these.
>
> There is also a "enter manually" option.
>
> > If the latter, to me it makes sense to keep it, maybe with updated
> > wording. Making things customizable is not really an issue, as long as
> > it's clear what happens where.
>
> The only wording there is "/tmp - Temporary files" or something like
> that (has been a while since I saw that particular menu) -- it's an
> option in a menu, so space is limited.
>
> FWIW, I don't see much of a use case for leaving /tmp on the root
> partition, so if no /tmp mount point is configured, mounting a tmpfs is
> a good choice. Basically anything that can boot the installer has enough
> RAM.

Found it:

https://salsa.debian.org/installer-team/partman-basicfilesystems/-/blob/master/debian/partman-basicfilesystems.templates?ref_type=heads#L97

So  unless someone has a better suggestion, I can change it from:

"/tmp - temporary files"

To something like:

"/tmp - temporary files (default: tmpfs)"

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 15:42, Richard Lewis
 wrote:
>
> Luca Boccassi  writes:
>
> > Hence, I am not really looking for philosophical discussions or lists
> > of personal preferences or hypotheticals, but for facts: what would
> > break where, and how to fix it?
>
> cleaning /tmp or /var/tmp: users may lose files if they dont realise a
>   directory tmp can be cleaned without a reboot. something in /var/tmp
>   that's been in there for 35 days before an upgrade might be deleted
>   before the user reads the NEWS.Debian email, meaning they have no
>   chance to react). Maybe you could postpone the very first deletion
>   until after the next reboot?
>
> using a tmpfs: is there a risk of losing unrelated data due to more
>   frequent OOM killing random other programmes due to /tmp using all the
>   memory?  is there a case to only use a tmpfs if the system has
>   "enough" memory?

Again, those are all hypotheticals, and one can construct similarly
contrived thought exercises for most possible permutations of most
configurations, and the answer is always the same: customize the
configuration accordingly. Hence, not relevant right now.
What is relevant is: which packages, if any, or which DSA-owned
systems, if any, are actually affected and how?

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon Richter]

Hi,

On 5/6/24 19:57, Michael Biebl wrote:


Afaik, /var/tmp has never been cleaned up on /boot.
So I'm not sure what you mean with "no longer"?


Oof, you're right, it was /tmp, /var/run, /var/lock:

[ "$VERBOSE" != no ] && echo -n "Cleaning"
[ -d /tmp ] && cleantmp
[ -d /var/run ] && cleanrun
[ -d /var/lock ] && cleanlock
[ "$VERBOSE" != no ] && echo "."

Would it make sense to make it a bug for a package to use /var/tmp (on 
my system, I can see files from audacity and reportbug there) and 
declare that this directory is for the use of the sysadmin only?


   Simon

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon Richter]

Hi,

On 5/6/24 20:19, Luca Boccassi wrote:


Is that the default layout, or a selectable option?


When you create a partition manually, it asks for the mount point, and 
makes a number of suggestions in a dropdown, and /tmp is one of these.


There is also a "enter manually" option.


If the latter, to me it makes sense to keep it, maybe with updated
wording. Making things customizable is not really an issue, as long as
it's clear what happens where.


The only wording there is "/tmp - Temporary files" or something like 
that (has been a while since I saw that particular menu) -- it's an 
option in a menu, so space is limited.


FWIW, I don't see much of a use case for leaving /tmp on the root 
partition, so if no /tmp mount point is configured, mounting a tmpfs is 
a good choice. Basically anything that can boot the installer has enough 
RAM.


   Simon

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 16:03, Barak A. Pearlmutter  wrote:
>
> If it clones into /tmp the *entire* tree will either be reaped (upon
> reboot) or not.
>
> But having just some files deleted from a git dir or git working dir
> is much more dangerous, because various git commands can treat files
> deleted from the working tree as deliberate changes to be committed,
> and files deleted from the git dir can render it unusable.

It's not really that different, it's a variation of the same problem.
I take it that git doesn't warn when cloning on tmpfs, then, so that's
not an issue.
tmpfiles.d snippets can be defined to cleanup on a timer _anything_,
and have done so for ~15 years. Does git currently parse all installed
tmpfiles.d snippets and warn if cloning on an arbitrary directory
covered by one of them?

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Andrey Rakhmatullin]

On Mon, May 06, 2024 at 07:42:11AM -0700, Russ Allbery wrote:
> >> I'm not sure if we have software on long running servers which place
> >> files in /tmp and /var/tmp and expect files to not be deleted during
> >> runtime, even if not accessed for a long time. This is certainly an
> >> issue to be aware of and keep an eye on.
> 
> > Note that FHS mandates it for /var/tmp: "Files and directories located
> > in /var/tmp must not be deleted when the system is booted. Although data
> > stored in /var/tmp is typically deleted in a site-specific manner, it is
> > recommended that deletions occur at a less frequent interval than /tmp."
> 
> It mandates that it not be cleaned on *boot*.  Not that it never be
> cleaned during runtime.  
Ah, I've read that as "while the system is booted".


-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

If it clones into /tmp the *entire* tree will either be reaped (upon
reboot) or not.

But having just some files deleted from a git dir or git working dir
is much more dangerous, because various git commands can treat files
deleted from the working tree as deliberate changes to be committed,
and files deleted from the git dir can render it unusable.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Andrey Rakhmatullin  writes:
> On Mon, May 06, 2024 at 10:40:00AM +0200, Michael Biebl wrote:

>> I'm not sure if we have software on long running servers which place
>> files in /tmp and /var/tmp and expect files to not be deleted during
>> runtime, even if not accessed for a long time. This is certainly an
>> issue to be aware of and keep an eye on.

> Note that FHS mandates it for /var/tmp: "Files and directories located
> in /var/tmp must not be deleted when the system is booted. Although data
> stored in /var/tmp is typically deleted in a site-specific manner, it is
> recommended that deletions occur at a less frequent interval than /tmp."

It mandates that it not be cleaned on *boot*.  Not that it never be
cleaned during runtime.  It anticipates that it be cleaned periodically,
just less frequently than /tmp.

There is a specific prohibition against clearing /var/tmp on reboot
because /var/tmp historically has been used to store temporary files whose
whole reason for existence is that they need to survive a reboot, such as
vi recover files, but are still safe to delete periodically.

Historically, deleting anything in /var/tmp that hadn't been accessed in
over seven days was a perfectly reasonable and typical configuration.
These days, we have the complication that it's fairly common to turn off
atime updates for performance reasons, which makes it a bit harder to
implement that policy when /var/tmp isn't its own partition and thus
inherits that setting from the rest of the system.

-- 
Russ Allbery (r...@debian.org)  

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 15:31, Barak A. Pearlmutter  wrote:
>
> > What I am looking for right now is packages or internal
> > infrastructure that need
> > an update to cope with these two changes before I upload them, so if
> > you know of any please do let me know and I'll happily look into it
> > and at least file a bug, if not a MR. Thanks.
>
> Okay.
>
> git and other version control systems should be modified to warn if
> the work tree or git dir are in /var/tmp and therefore exposed to
> unexpected deletion of files, and possibly modified to refuse to clone
> there without an explicit --force-unstable-storage option.

Does it do something similar when cloning into /tmp/ or any other
tmpfs right now?

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

> What I am looking for right now is packages or internal
> infrastructure that need
> an update to cope with these two changes before I upload them, so if
> you know of any please do let me know and I'll happily look into it
> and at least file a bug, if not a MR. Thanks.

Okay.

git and other version control systems should be modified to warn if
the work tree or git dir are in /var/tmp and therefore exposed to
unexpected deletion of files, and possibly modified to refuse to clone
there without an explicit --force-unstable-storage option.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon McVittie]

On Mon, 06 May 2024 at 13:41:58 +0100, Barak A. Pearlmutter wrote:
> As someone who regularly deals with large datasets, and keeps them in
> the "approved" don't-back-these-up location /var/tmp

Independent of whether we make the change Luca is suggesting or not,
I don't think /var/tmp is a good recommendation for users storing
temporary data manually, because safe use of "obvious" filenames in a
world-writeable location like /var/tmp requires you to be aware of the
possibility of symlink attacks, and either ensure that your kernel has a
mitigation for them (as Debian's does with default runtime configuration),
or be careful to avoid them, or be OK with any human user account and
any system uid on the system being able to achieve at least denial of
service and maybe privilege escalation to your account.

/tmp and /var/tmp are fine places for *programs* to store temporary data -
for example Flatpak uses a subdirectory in /var/tmp for half-downloaded
apps and runtimes - but only if they're using mkstemp(), mkdtemp()
or equivalent to generate an unpredictable location and allocate it
without race conditions, which is the only way these directories are
safe to use in portable software. This is not particularly user-friendly
to do manually.

For files you don't want backed up, I'd personally recommend storing
them somewhere in your home directory, perhaps ~/tmp, with a CACHEDIR.TAG
(http://justsolve.archiveteam.org/wiki/Cache_Directory_Tagging_Standard),
and using backup software that understands that convention (for example
borg, restic, tar).

smcv

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Richard Lewis]

Luca Boccassi  writes:

> Hence, I am not really looking for philosophical discussions or lists
> of personal preferences or hypotheticals, but for facts: what would
> break where, and how to fix it?

cleaning /tmp or /var/tmp: users may lose files if they dont realise a
  directory tmp can be cleaned without a reboot. something in /var/tmp
  that's been in there for 35 days before an upgrade might be deleted
  before the user reads the NEWS.Debian email, meaning they have no
  chance to react). Maybe you could postpone the very first deletion
  until after the next reboot?

using a tmpfs: is there a risk of losing unrelated data due to more
  frequent OOM killing random other programmes due to /tmp using all the
  memory?  is there a case to only use a tmpfs if the system has
  "enough" memory?