Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

> > what would break where, and how to fix it? I only found autopkgtest
> so
> > far, which uses /tmp/ in the guest and expects it to survive across
> > reboots, and I have a MR up already for that. Anything else?
> 
> Perhaps whatever makes these files in /tmp? i think something to do
> with
> X/gdm/gnome?
> 
> /tmp/.X0-lock
> /tmp/.X1024-lock
> /tmp/.X1025-lock
> 
> /tmp/.X11-unix
> /tmp/.X1-lock
> 
> /tmp/.XIM-unix
> 
> /tmp/.font-unix
> 
> /tmp/.ICE-unix

These are all already covered by /usr/lib/tmpfiles.d/x11.conf

-- 
Kind regards,
Luca Boccassi


signature.asc
Description: This is a digitally signed message part

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Michael Biebl]

Am 13.05.24 um 11:42 schrieb Johannes Schauer Marin Rodrigues:

If we want to try and weigh cost against benefit, do the benefits really
outweigh the cost? How costly is it to carry a patch in Debian and deviate from
upstream versus all the problems that participants of this thread now listed?
My gut feeling is, that the cost of these hard to debug problems is far greater
than continuing to deviate from upstream and carry a Debian-specific patch, no?


Concerning the /var/tmp (and /tmp) cleaning, the Debian specific patch 
is 
https://salsa.debian.org/systemd-team/systemd/-/blob/debian/master/debian/patches/debian/Bring-tmpfiles.d-tmp.conf-in-line-with-Debian-defaul.patch?ref_type=heads


This patch/change was added in 2012 and has required very little to no 
maintenance since that time. So the cost of keeping it is very low 
maintenance wise.


Michael



OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

Unless somebody's already put it there, I'm going to move these
suggestions to a wishlist bug against systemd. Not sure if it should
be one bug or a few, one for each suggestion.

Currently discussion about reaping /var/tmp/ is in
https://bugs.debian.org/966621 and
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1870585 but
these are discussing "should we turn on /var/tmp/ reaping" rather than
"if we do turn it on, should we take measures to make it safer".

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Barak A. Pearlmutter (2024-05-13 10:47:43)
> > I'd like to hear some arguments *in favour* of making this change.
> > Alignment with systemd-upstream, reduced package maintenance burden
> > are two that I can think of, but perhaps I've missed more. These two,
> > IMHO, are significantly outweighed by the risks.
> Let me see if I understand the arguments being made in favor.

thank you, I'd also like to understand them.

> 1. Compatibility with upstream. This means all the upstream logic is sort of
>imported by reference, so the below is mainly the upstream logic, as I
>understand it.

Yes, I also think that there is value in doing the same thing that upstream or
other distros are doing. There is a cost if Debian decides to deviate from what
others decided to be the default.

> 2. Defend the system against buggy programs that leave debris in /var/tmp/,
>and against debris left there when programs are terminated prematurely.
>These are programs which use /var/tmp/ internally, but not as part of
>their API, so the user would have no particular way of knowing that they
>are leaving things there, would have no particular reason to check for and
>delete such files, and might not be able to easily recognize which files
>should be removed.

But I do not understand this as an advantage. In my mind it is quite the
opposite. The buggy programs which leave files in /tmp will now have their bugs
not noticed anymore because the files get cleaned up by systemd. On the other
hand, we are now introducing new bugs in programs which should do an flock on
the temporary directory but do not do so yet. Imagine I would not've read
debian-devel. How would I as the mmdebstrap author have noticed that my tool as
a user of /tmp should set up this flock? I imagine the bug report of somebody
who has a weird problem with the chroot but somehow they are unable to
reproduce it because it depends on the cleanup timing. Are the bugs we are
introducing by regularly cleaning up /tmp not potentially super hard to
diagnose and might thus just not get fixed? Is there an effort to go around and
identify programs we ship with long lasting use of /tmp and is filing bugs so
that they are performing an flock? And at the same time we are now ignoring the
bugs in programs who leave files in /tmp and forget to clean them up. Is this
not a disadvantage of this change instead of an advantage?

> I looked at the upstream bug report
> https://github.com/systemd/systemd/issues/32674 which suggests that deletion
> of directory trees in /var/tmp/ be atomic, and trigger only when everything
> in the tree meets criteria for deletion. I added a comment suggesting that
> the policy be tweaked in two ways. (a) Use system-up time rather than
> wall-clock time for measuring file age, to address the "suspending or
> shutting down for over 30 days breaks running data processing scripts that
> uses /var/tmp/ for intermediate files" issue. I sort of have an invariant in
> my head, which is that suspending the computer doesn't break things, and also
> the whole point of /var/tmp/ is that files there are preserved across boots.
> And (b) check if a file is open by some process, the same way fuser(1) does,
> and if so, don't delete it. Could also preserve directories which are the
> current directory of some process, if you want to be even more user friendly.
> 
> The only response I got was "don't use temporary directories for things that
> you cannot afford to lose and recreate", which I don't really understand.
> It's like saying "you should have good backups, so it's not a problem to just
> delete anything in /home older than two days". Bottom line, it's not clear to
> me that upstream has really thought this through with users in mind. I'd
> suggest that Debian may wish to tweak the defaults on this stuff pretty hard
> to be more user friendly.

Thank you, the suspend issue is another issue that is created by this change.
If we want to try and weigh cost against benefit, do the benefits really
outweigh the cost? How costly is it to carry a patch in Debian and deviate from
upstream versus all the problems that participants of this thread now listed?
My gut feeling is, that the cost of these hard to debug problems is far greater
than continuing to deviate from upstream and carry a Debian-specific patch, no?

> - as discussed earlier, add /tmp/00-README and /var/tmp/00-README to explain
> this old-file-deletion policy

I think this is a really good idea.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

> I'd like to hear some arguments *in favour* of making this change.
> Alignment with systemd-upstream, reduced package maintenance burden
> are two that I can think of, but perhaps I've missed more. These two,
> IMHO, are significantly outweighed by the risks.

Let me see if I understand the arguments being made in favor.

1. Compatibility with upstream. This means all the upstream logic is
sort of imported by reference, so the below is mainly the upstream
logic, as I understand it.

2. Defend the system against buggy programs that leave debris in
/var/tmp/, and against debris left there when programs are terminated
prematurely. These are programs which use /var/tmp/ internally, but
not as part of their API, so the user would have no particular way of
knowing that they are leaving things there, would have no particular
reason to check for and delete such files, and might not be able to
easily recognize which files should be removed.

3. Defend the system against forgetful users, who create files in
/var/tmp/ but neglect to delete them when they're no longer needed.

Unfortunately 2 & 3 cannot be mechanically distinguished, so even if
you wanted to have separate policies for these two classes of files,
it's not really technically possible. So upstream is optimizing for
case (2), and suggests that /var/tmp/ be sort of reserved for programs
and scripts that are aware of this policy, and users not manually
create files there.

I hope that's an accurate characterization.

I looked at the upstream bug report
https://github.com/systemd/systemd/issues/32674 which suggests that
deletion of directory trees in /var/tmp/ be atomic, and trigger only
when everything in the tree meets criteria for deletion. I added a
comment suggesting that the policy be tweaked in two ways. (a) Use
system-up time rather than wall-clock time for measuring file age, to
address the "suspending or shutting down for over 30 days breaks
running data processing scripts that uses /var/tmp/ for intermediate
files" issue. I sort of have an invariant in my head, which is that
suspending the computer doesn't break things, and also the whole point
of /var/tmp/ is that files there are preserved across boots. And (b)
check if a file is open by some process, the same way fuser(1) does,
and if so, don't delete it. Could also preserve directories which are
the current directory of some process, if you want to be even more
user friendly.

The only response I got was "don't use temporary directories for
things that you cannot afford to lose and recreate", which I don't
really understand. It's like saying "you should have good backups, so
it's not a problem to just delete anything in /home older than two
days". Bottom line, it's not clear to me that upstream has really
thought this through with users in mind. I'd suggest that Debian may
wish to tweak the defaults on this stuff pretty hard to be more user
friendly.

Here are a couple cheap tweaks I'd suggest. My hope is that these
would avoid some of the worst case scenarios discussed while still
satisfying the goals 1/2/3 above, and being super easy to implement.

- lengthen the reap time for /var/tmp/ to eight weeks, since Europeans
often take six-week vacations.

- make a "tempdir" command, philosophically similar to tempfile in
debianutils, which creates a fresh directory in /var/tmp/ and drops
the user into a shell with that as the current directory, directory
flocked until that subshell terminates. Could give an optional
directory argument to lock so you can get back into a directory, and
maybe have options to run a program there before or instead of the
subshell.

- following a resume-from-suspend or boot, shut off the
delete-old-files-in-var-tmp mechanism for a while, maybe eight hours
or something like that. Maybe shorten the delay if it doesn't get to
run across multiple resume-or-boot cycles for a week or two.

- as discussed earlier, add /tmp/00-README and /var/tmp/00-README to
explain this old-file-deletion policy

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Bill Allombert]

Le Mon, May 06, 2024 at 11:15:35AM +0100, Barak A. Pearlmutter a écrit :
> > We have two separate issues here:
> 
> > a/ /tmp-on-tmpfs
 
Note that /tmp-on-tmpfs and cleanup-tmp-at-boot are not equivalent.

With cleanup-tmp-at-boot, if your system crashes, you can still backup
/tmp before rebooting.

Cheers,
-- 
Bill. 

Imagine a large red swirl here.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Stéphane Blondon]

Le mar. 7 mai 2024, 20:18,  a écrit :

> Even after a reboot, I would be upset to lose the debug files that I've
> been accumulating for several days while trying to track down an
> intermittent problem with this stupid VPN...
>


At reboot, /tmp isautomatically flushed. It's the default behaviour since
years (at least on physical machines).

-- 
Stephane

Avoiding /var/tmp for long-running compute (was: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default])

[Russ Allbery]

"Jonathan Dowland"  writes:

> Else-thread, Russ begs people to stop doing this. I agree people
> shouldn't! We should also work on education and promotion of the
> alternatives.

Also, helping people use better tools for managing workloads like this
that make their lives easier and have better semantics, thus improving
life for everyone.

I'm suggesting solutions that I don't have time to help implement, and of
course it will take a long time for better tools to filter into all those
clusters, so this doesn't address the immediate problem of this thread
(hence the subject change).  But based on my past experience with these
types of systems, I bet a lot of the patterns captured in software are
older ones.  Linux has a *lot* of facilities today that it didn't have, or
at least weren't widely used, five years ago.  It would be great to help
some of those improvements filter down, because they can make a lot of
these problems go away.

For example, take the case of scratch space for batch computing.  The
logical lifespan for temporary files for a batch computing job is the
lifetime of the job, whatever that may be.  (I know there are exceptions,
but here I'm just talking about defaults.)  Previously one would have to
build support into the batch job management system for creating and
managing those per-job temporary directories, and ensure the jobs support
TMPDIR or other environment variables to control where they store data,
and everyone was doing this independently.  (I've done a *lot* of this
kind of thing, once upon a time.)

But now we have mount namespaces, and systemd has PrivateTmp that builds
on top of that.  So if the job is managed by an execution manager, it can
create per-job temporary directories and it may already support (as
systemd does) the semantics of deleting the contents of those directories
on job exit, and it bind-mounts those into the process space and the
process is none the wiser.  I think all of the desirable glue may not
fully be there (controlling what underlying file system is used for
PrivateTmp, ensuring they're also excluded from normal cleanup, etc.), but
this is very close to a much better way of handling this problem that
still exposes /tmp and /var/tmp to the job so that none of the
often-crufty scientific computing software has to change.

The new capabilities that Linux now has due to namespaces are marvellous
and solve a whole lot of problems that I didn't realize were even
solvable, and right now I suspect there are huge opportunities for
substantial improvements without a whole lot of effort by just plumbing
those facilities through to higher-level layers like batch systems.  Whole
classes of long-standing problems would just disappear, or at least be
far, far easier to manage.

Substantial, substantial caveat: I have been out of this world for a
while, and maybe most of this work has already been done?  That would be
amazing.  The best possible response to this post would be for someone to
tell me I'm five years behind and the batch systems have already picked up
this work and we can just point people at them.

-- 
Russ Allbery (r...@debian.org)  

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Jonathan Dowland]

On Mon May 6, 2024 at 5:01 PM BST, Luca Boccassi wrote:
> On Mon, 6 May 2024 at 16:51, Barak A. Pearlmutter 
> wrote:
> > For whatever reason, a lot of people who process large data use
> > /var/tmp/FOO/ as a place to store information that should not be
> > backed up, but also should not just disappear.
>
> Then such people, assuming they actually exist, can configure their
> custom systems accordingly upon reading the release notes before
> upgrading, as they would do anyway if installing on CentOS or any
> other major OS. Hence, not an issue either.

They actually exist. I'm been one of them, I've worked with many of
them, it's an incredibly common pattern in academic computing at least,
and changing it in Debian should be a very carefully explored decision.

You've pointed out that changing the behaviour from the default, in
either direction, is trivial. The issue is not one of individual
preference but of what is default. The long-established status quo
is not to clean /var/tmp. There is serious risk here: to users data
and correspondingly to Debian's reputation for stability, which
many of us have worked hard to maintain for a very long time.

If you think we need hard data to quantify this practice, then let's
work on a plan for how to gather that going forward, rather than
dismiss this outright because we haven't collected it.

Else-thread, Russ begs people to stop doing this. I agree people
shouldn't! We should also work on education and promotion of the
alternatives.

I'd like to hear some arguments *in favour* of making this change.
Alignment with systemd-upstream, reduced package maintenance burden
are two that I can think of, but perhaps I've missed more. These two,
IMHO, are significantly outweighed by the risks.

Please hold off making this change now and let this discussion continue.


-- 
Please do not CC me for listmail.

  Jonathan Dowland
✎j...@debian.org
   https://jmtd.net

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Emanuele Rocca]

Hi,

On 2024-05-07 09:43, Russ Allbery wrote:
> I understand your point, which is that this pattern is out there in the
> wild and Debian is in danger of breaking existing usage patterns by
> matching the defaults of other distributions.  This is a valid point, and
> I appreciate you making it.

The more general point being that if systems have certain properties,
whether by design or by accident, people tend to rely on them if these
properties are useful for whatever reasons.

In the specific case of /var/tmp in Debian, for a very long time now the
properties have been: (1) persistent, world-writable storage (2) outside
of /home (3) available by default on all systems without any
configuration. To many, these properties make for a good place where
transient-ish work can be done without the risk of losing it upon reboot
(or power loss, or similar). Not being in /home is an important one,
because for instance /home may be regularly backed up, or it may be on a
NFS share, or who knows what else, and you may not want that who knows
why.

All of that being said, I do see the value in uniformity with other
distros, also because it surely makes things easier for maintainers.
And yes, https://xkcd.com/1172/.

It's just that changes are usually a costs/benefits tradeoff -- in the
xkcd a CPU is overheating, whereas in this case the problem to fix seems
a bit less obvious.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Marc Haber]

On Tue, 07 May 2024 22:29:30 +0100, Richard Lewis
 wrote:
>Holger Levsen  writes:
>> I'm a bit surprised how many people seem to really rely on data in /tmp
>> to survive for weeks or even months. I wonder if they backup /tmp?
>
>I use /tmp for things that fall somewhere between "needs a backup" and
>"unimportant, can be deleted whenever".

For me there is a difference between /tmp and /var/tmp. On all my
systems, /tmp has been a tmpfs for decades, /var/tmp being used for
data that is too large for tmpfs.

Even losing /tmp would probably affect some of my running programs
including the X11 session.

Greetings
Marc
-- 

Marc Haber |   " Questions are the | Mailadresse im Header
Rhein-Neckar, DE   | Beginning of Wisdom " | 
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 6224 1600402

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Simon Richter  writes:
> On 5/8/24 07:05, Russ Allbery wrote:

>> It sounds like that is what kicked off this discussion, but moving /tmp
>> to tmpfs also usually makes programs that use /tmp run faster.  I
>> believe that was the original motivation for tmpfs back in the day.

> IIRC it started out as an implementation of POSIX SHM, and was later
> generalized.

I believe you're correct for Linux specifically but not in general for
UNIX.  For example, I'm fairly sure this is not the case on Solaris, which
was the first place I encountered tmpfs and where tmpfs /tmp was the
default starting in Solaris 2.1 in 1992.  tmpfs was present in SunOS in
1987, so I'm pretty sure it predates POSIX shared memory.

Linux was very, very late to the tmpfs world.

> When /var runs full, the problem is probably initrd building.

I'm not quite sure what to make of this statement.  On my systems, /var
contains all sorts of rather large things, such as PostgreSQL databases,
INN spool files, and mail spools.  I have filled up /var on many systems
over the years, and it's never been by building initrd images.

> Taking a quick look around all my machines, the accumulated cruft in
> /var/tmp is on the order of kilobytes -- mostly reportbug files, and a
> few from audacity -- and these machines have not been reinstalled in the
> last ten years.

Yes, I don't think many programs use it.  I think that's a good thing; the
specific semantics of /var/tmp are only useful in fairly narrow
situations, and overfilling it is fairly dangerous.

Back in the day, /var/tmp was the thing that you used if /tmp was too
small (because it was usually tmpfs).  For example, using sort -T /var/tmp
to sort large files is an old UNIX rune.  And, of course, students would
use it because they ran out of quota in their home directories and then
get upset when their files got deleted automatically, back in the days of
shared UNIX login clusters.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Tue, 7 May 2024 at 17:33, Sam Hartman  wrote:
>
> > "Luca" == Luca Boccassi  writes:
>
> Luca> On Mon, 6 May 2024 at 15:42, Richard Lewis
> Luca>  wrote:
> >>
> >> Luca Boccassi  writes:
> >>
> >> > Hence, I am not really looking for philosophical discussions or
> >> lists > of personal preferences or hypotheticals, but for facts:
> >> what would > break where, and how to fix it?
>
> ssh-agent appears to default to creating a socket under /tmp.
> I think respecting $XDG_RUNTIME_DIR would be better.
>
> /etc/X11/Xsession.d/90x11-common_ssh-agent also doesn't override where
> the socket ends up.
> I definitely think for session scripts like that $XDG_RUNTIME_DIR would
> be better.
>
>
> gnome-keyring's ssh-agent handles this better, although last time I
> checked, it did not support pkcs11, so I could not use it with PIV
> cards.
> (Other parts of gnome-keyring do support pkcs11).

The ssh agent provided by gnupg also behaves correctly and creates the
socket in XDG_RUNTIME_DIR. I have filed a bug for openssh-client's
ssh-agent.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon Richter]

Hi,

On 5/8/24 07:05, Russ Allbery wrote:


It sounds like that is what kicked off this discussion, but moving /tmp to
tmpfs also usually makes programs that use /tmp run faster.  I believe
that was the original motivation for tmpfs back in the day.


IIRC it started out as an implementation of POSIX SHM, and was later 
generalized. The connection to SHM is still there -- POSIX SHM only 
works if a tmpfs is mounted anywhere in the system. Canonically, a tmpfs 
is mounted on /dev/shm for that purpose, but if /tmp is a tmpfs, then 
/dev/shm doesn't need to exist.


I agree that it makes a lot of things run faster (especially compiling, 
which creates temporary files in /tmp), but it has also caused 
situations that required pressing SysRq to resolve (also during compiling).



For /var/tmp, I think the primary motivation to garbage-collect those
files is that filling up /var/tmp is often quite bad for the system.  It's
frequently not on its own partition, but is shared with at least /var, and
filling up /var can be very bad.  It can result in bounced mail, unstable
services, and other serious problems.


When /var runs full, the problem is probably initrd building.

Taking a quick look around all my machines, the accumulated cruft in 
/var/tmp is on the order of kilobytes -- mostly reportbug files, and a 
few from audacity -- and these machines have not been reinstalled in the 
last ten years.


   Simon

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Tue, 7 May 2024 at 22:57, Russ Allbery  wrote:
>
> Richard Lewis  writes:
> > Luca Boccassi  writes:
>
> >> what would break where, and how to fix it?
>
> > Another one for you to investigate: I believe apt source and 'apt-get
> > source' download and extract things into /tmp, as in the mmdebootstap
> > example mentioned by someone else, this will create "old" files that
> > could immediately be flagged for deletion causing surprises.
>
> > (People restoring from backups might also find this an issue)
>
> systemd-tmpfiles respects atime and ctime by default, not just mtime, so I
> think this would only be a problem on file systems that didn't support
> those attributes.  atime is often turned off, but I believe support for
> ctime is fairly universal among the likely file systems for /var/tmp, and
> I believe tmpfs supports all three.  (I'm not 100% sure, though, so please
> correct me if I'm wrong.)

Yes atime/ctime are used too, so things that are really in the process
of being used are not really an issue.

I checked screen and even in bookworm it uses /run/screen/ as you
said, so it's fine.

I checked tmux and indeed it uses /tmp/tmux-UID/ - which is a terrible
choice given it's predictable so if something manages to run first it
can hijack it, but that's really a pre-existing issue. I've filed a
bug to notify that it needs to start flocking the file in /tmp/ while
running to avoid them being deleted while in use.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Richard Lewis  writes:

> btw, i'm not trying to argue against the change, but i dont yet
> understand the rationale (which id like to be put into the
> release-notes): is there perhaps something more compelling than "other
> distributions and upstream already do this"?

It sounds like that is what kicked off this discussion, but moving /tmp to
tmpfs also usually makes programs that use /tmp run faster.  I believe
that was the original motivation for tmpfs back in the day.

For /var/tmp, I think the primary motivation to garbage-collect those
files is that filling up /var/tmp is often quite bad for the system.  It's
frequently not on its own partition, but is shared with at least /var, and
filling up /var can be very bad.  It can result in bounced mail, unstable
services, and other serious problems.

Most modern desktop systems now have large enough drives that this isn't
as much of a concern as it used to be, but VMs often still have quite
small / partitions and put /var/tmp on that partition.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Sven Mueller]

Am 07.05.2024 22:56 schrieb Richard Lewis :Luca Boccassi  writes:



> qwhat would

> break where, and how to fix it?



Another one for you to investigate: I believe apt source and 'apt-get

source' download and extract things into /tmp, as in the mmdebootstap

example mentioned by someone else, this will create "old" files that

could immediately be flagged for deletion causing surprises.`apt download` and `apt source` download to your current working directory. Same for apt-get.
(People restoring from backups might also find this an issue)I would not expect people to restore to /tmp and expecting that restore to work across a reboot. And to be honest, I find the expectation to have any guarantee on files in /tmp or /var/tmp across a reboot quite surprising. The directories are named "tmp" because they are meant as temporary storage, which implies automatic deleting at some point IMHO.Now, bad choices by various tools have been mentioned, so a cleaner for these directories that runs outside a reboot has to be careful anyhow. But during a reboot? I don't think that should be too much of a problem. Cheers, Sven

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Richard Lewis  writes:
> Luca Boccassi  writes:

>> what would break where, and how to fix it?

> Another one for you to investigate: I believe apt source and 'apt-get
> source' download and extract things into /tmp, as in the mmdebootstap
> example mentioned by someone else, this will create "old" files that
> could immediately be flagged for deletion causing surprises.

> (People restoring from backups might also find this an issue)

systemd-tmpfiles respects atime and ctime by default, not just mtime, so I
think this would only be a problem on file systems that didn't support
those attributes.  atime is often turned off, but I believe support for
ctime is fairly universal among the likely file systems for /var/tmp, and
I believe tmpfs supports all three.  (I'm not 100% sure, though, so please
correct me if I'm wrong.)

-- 
Russ Allbery (r...@debian.org)  

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Richard Lewis]

Holger Levsen  writes:

> I'm a bit surprised how many people seem to really rely on data in /tmp
> to survive for weeks or even months. I wonder if they backup /tmp?

I use /tmp for things that fall somewhere between "needs a backup" and
"unimportant, can be deleted whenever". I think all of the issues raised
(disappearing files from git checkouts, old files in unpacked
tarballs/debootstraps/downloads, autopkgtests, 3rd-party-software, bad
choices in tmux/ssh-agent/etc) fall into that spectrum: It's not that
the data is critical, but losing it creates more work -- what is the
reason for accepting that 'risk'?

btw, i'm not trying to argue against the change, but i dont yet
understand the rationale (which id like to be put into the
release-notes): is there perhaps something more compelling than "other
distributions and upstream already do this"?

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Richard Lewis]

Luca Boccassi  writes:

> qwhat would
> break where, and how to fix it?

Another one for you to investigate: I believe apt source and 'apt-get
source' download and extract things into /tmp, as in the mmdebootstap
example mentioned by someone else, this will create "old" files that
could immediately be flagged for deletion causing surprises.

(People restoring from backups might also find this an issue)

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Andrey Rakhmatullin]

On Tue, May 07, 2024 at 09:49:17PM +0200, Johannes Schauer Marin Rodrigues 
wrote:
> Quoting Andrey Rakhmatullin (2024-05-06 19:14:40)
> > On Mon, May 06, 2024 at 04:50:50PM +0100, Barak A. Pearlmutter wrote:
> > > > tmpfiles.d snippets can be defined to cleanup on a timer _anything_,
> > > 
> > > It's a question of what the *default* behaviour should be.
> > > 
> > > For whatever reason, a lot of people who process large data use
> > > /var/tmp/FOO/ as a place to store information that should not be
> > > backed up, but also should not just disappear.
> > To be honest I'm greatly surprised by this idea, and by the suggestion
> > that a lot of people do this, to me this is very similar by that half-joke
> > about people storing useful files in the Recycle Bin.
> 
> I'm doing exactly that. I use paper I already printed stuff on but which were
> misprints or which are no longer useful to me and are just destined for the
> recycling bin to write stuff on that is important to me. After I'm done with 
> my
> work on this scrap paper I decide what I want to keep and copy to permanent
> storage and what I want to really throw away.
I actually meant the Windows feature :)


-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Quoting Andrey Rakhmatullin (2024-05-06 19:14:40)
> On Mon, May 06, 2024 at 04:50:50PM +0100, Barak A. Pearlmutter wrote:
> > > tmpfiles.d snippets can be defined to cleanup on a timer _anything_,
> > 
> > It's a question of what the *default* behaviour should be.
> > 
> > For whatever reason, a lot of people who process large data use
> > /var/tmp/FOO/ as a place to store information that should not be
> > backed up, but also should not just disappear.
> To be honest I'm greatly surprised by this idea, and by the suggestion
> that a lot of people do this, to me this is very similar by that half-joke
> about people storing useful files in the Recycle Bin.

I'm doing exactly that. I use paper I already printed stuff on but which were
misprints or which are no longer useful to me and are just destined for the
recycling bin to write stuff on that is important to me. After I'm done with my
work on this scrap paper I decide what I want to keep and copy to permanent
storage and what I want to really throw away.

I would not like if (suppose I had a person cleaning up my stuff) a cleaning
person came to my desk, saw that there was obvious scrap paper ultimately
destined for the bin and threw that away every once in a while. I might be
taking my notes on what is ultimately for the trash but I want to be the one to
decide when to bring the trash out.

Thank you for this great example! :)

cheers, josch

signature.asc
Description: signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Holger Levsen (2024-05-07 17:22:48)
> On Tue, May 07, 2024 at 04:24:06PM +0300, Hakan Bayındır wrote:
> > Consider a long running task, which will take days or weeks (which is the
> > norm in simulation and science domains in general). System emitted a warning
> > after three days, that it'll delete my files in three days. My job won't be
> > finished, and I'll be losing three days of work unless I catch that warning.
> Then it will be high time you learn not to abuse /tmp that way and work in
> your (or your services) home/data directory.
> 
> Problem easily avoided. plus you don't need to make /tmp 20 TB because you
> have lots of data. ;)
> 
> I'm a bit surprised how many people seem to really rely on data in /tmp to
> survive for weeks or even months. I wonder if they backup /tmp?

I like using /tmp because it's a tmpfs which makes some things faster. For
quite a few things I do not want them to be stored long-term on my SSD so I
resort to using /tmp and not the directory I called ~/tmp inside my $HOME.

This is also not only about data surviving for weeks and months. Elsewhere in
this thread i mentioned mmdebstrap as an application which creates files in
/tmp which have a modification time far in the past. The same happens when
using other tools, for example lets say I want to have a small scratch space
into which I wget some files:

$ wget https://www.debian.org/Pics/debian-logo-1024x576.png
$ stat -c %y debian-logo-1024x576.png
2020-12-17 10:59:08.0 +0100

Will this mean that debian-logo-1024x576.png might accidentally get cleaned up
unless I disable that mechanism? The problem is not limited to people with a
crazy large /tmp either. My system has 3.7 GB of ram and having /tmp be a tmpfs
(even though it's very small) is still beneficial for me because my maximum
read speed from my SSD is 140 MB/s. My small RAM is much faster than that.

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

I guess sometimes when people discuss technical matters, good ideas pop up.

(Although I still think that its problematic interactions with lengthy
suspends makes the whole idea of auto-deletion based purely on
timestamps problematic. I can imagine more coherent mechanisms, which
doesn't count time the machine was suspended against the removal
clock, which check if any files are open, check if any process has its
current directory in a tree, which treats trees as a whole instead of
deleting leaves piecemeal only deleting the tree if all its contents
are ripe, etc. That would introduce considerable complexity though.
However, it is the sort of thing a good sysadmin would do before
manually removing stuff in /var/tmp/, so ...)

Re: Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Josh Triplett]

Barak A. Pearlmutter wrote:
> You know, that's a pretty good idea!
>
> Put a 00README-TMP.txt in /tmp/ and /var/tmp/ which briefly states the
> default deletion policy, the policy in place if it's not the default,
> and a pointer to info about altering it. "/tmp's contents are deleted
> at boot while /var/tmp is preserved across rebooting." Maybe in
> /var/tmp suggest /var/scratch/ or /var/cache/tmp or such as a place
> sysadmins might want to set up for not-backed-up but not-auto-deleted
> material.
>
> If the contents aren't dynamic, maybe they could be links to files in
> /usr/share/doc/systemd/.

This seems like a *great* idea. systemd-tmpfiles configuration can
easily create such a file, either with contents or as a symlink to a
documentation file in /usr/share/doc.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[rhys]

It is not "abuse of /tmp" to put files there, even if they need to be there for 
a long time. That is an unnecessary characterization. 

Yes, /tmp gets backed up along with the rest of the system on every VM in my 
environment. 

Sometimes "temporary" CAN mean "weeks or even months." That's not something 
that needs to be determined in advance by someone else. 

This is keeping in mind that I, myself, said that I would be fine with /tmp as 
a tmpfs. I wouldn't cry at all if /tmp were cleaned out at boot time (which 
could be weeks or even months).

But a) at boot time, all processes have been restarted. It would be an 
exceptional process that needs to resume after a reboot using data in /tmp 
(though not impossible) and...

...perhaps more importantly, b) I wouldn't want to declare on behalf of all 
Debian users that doing so Is Wrong And You Should Not Do It. That strikes me 
as unnecessarily presumptuous. 

It's one thing to support a change that MY systems can cope with. It's quite 
another to declare that EVERYONE will now use these rules, particularly when it 
means deleting their files. 

Even after a reboot, I would be upset to lose the debug files that I've been 
accumulating for several days while trying to track down an intermittent 
problem with this stupid VPN...

Sent from my mobile device.


From: Holger Levsen 
Sent: Tuesday, May 7, 2024 10:22
To: debian-devel@lists.debian.org
Subject: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default 
[was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

Then it will be high time you learn not to abuse /tmp that way 

I'm a bit surprised how many people seem to really rely on data in /tmp
to survive for weeks or even months. I wonder if they backup /tmp?


--
cheers,
Holger

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄

"When one man dies it's a tragedy. When thousands die it's statistics."
(Stalin commenting the worlds reaction on Covid 19.)

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Marvin Renich]

Early in this meta-thread it was suggested to separate /tmp-is-tmpfs
from cleanup-of-{,/var}/tmp.  I am really surprised that nobody has
suggested the obvious separation of new installs from upgrades.

Changing the local configuration for either feature is trivial either
way.  I think the proposed changes are reasonable for _new_
_installations_.  However, making configuration changes on upgrade that
may potentially cause significant problems, even if only for a small
number of users, and even if they are currently abusing tmp directories,
seems unnecessary to me.

The release notes can say "The defaults for new installations have
changed.  For upgrades, if it will not cause a disruption on your
system, you can get the new behavior by changing these configuration
files"

Personally, I believe (for upgrades) changing tmp-is-tmpfs will have
much less disruptive effect overall than the cleanup of /var/tmp, but I
don't see any reason to force either change on upgrade.

...Marvin

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Hakan Bayındır  writes:

> The applications users use create these temporary files without users'
> knowledge. They work in their own directories, but applications create
> another job dependent state files in both /tmp and /var/tmp. These are
> different programs and I assure you they’re not created there because
> user (or we) configured something. These files live there during the
> lifetime of the job, and cleaned afterwards by the application.

Then someone should fix those applications, because that behavior will
result in user data loss if they're not fixed.  However, first one should
check whether the applications are just honoring TMPDIR or equivalent
variables, in which case TMPDIR on batch systems often should be set to a
user-specific or job-specific persistent directory for exactly this
reason.  That way you can use a user-specific cleanup strategy, such as
purging that directory when all of the user's jobs have finished.

I understand your point, which is that this pattern is out there in the
wild and Debian is in danger of breaking existing usage patterns by
matching the defaults of other distributions.  This is a valid point, and
I appreciate you making it.

My replies are not intended to dispute that point, but to say that the
burden of addressing this buggy behavior should not rest entirely on
Debian.  What the combination of batch system and application is doing is
semantically incorrect and is dangerous, and it really should be fixed.
Even if Debian changes nothing, at some point someone will deploy workers
with a different base operating system and be very surprised when these
files are automatically deleted.

We were automatically cleaning /tmp and /var/tmp on commercial UNIX
systems in 1995 and fixing broken applications that didn't honor TMPDIR.
This is not a new problem.  Nor is having /var/tmp fill up and cause all
sorts of system problems because someone turned off /var/tmp cleaning
while trying to work around broken applications.

-- 
Russ Allbery (r...@debian.org)  

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Sam Hartman]

> "Luca" == Luca Boccassi  writes:

Luca> On Mon, 6 May 2024 at 15:42, Richard Lewis
Luca>  wrote:
>> 
>> Luca Boccassi  writes:
>> 
>> > Hence, I am not really looking for philosophical discussions or
>> lists > of personal preferences or hypotheticals, but for facts:
>> what would > break where, and how to fix it?

ssh-agent appears to default to creating a socket under /tmp.
I think respecting $XDG_RUNTIME_DIR would be better.

/etc/X11/Xsession.d/90x11-common_ssh-agent also doesn't override where
the socket ends up.
I definitely think for session scripts like that $XDG_RUNTIME_DIR would
be better.


gnome-keyring's ssh-agent handles this better, although last time I
checked, it did not support pkcs11, so I could not use it with PIV
cards.
(Other parts of gnome-keyring do support pkcs11).

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Hakan Bayındır]

Sent from my iPhone

> On 7 May 2024, at 18:39, Holger Levsen  wrote:
> 
> On Tue, May 07, 2024 at 04:24:06PM +0300, Hakan Bayındır wrote:
>> Consider a long running task, which will take days or weeks (which is the
>> norm in simulation and science domains in general). System emitted a warning
>> after three days, that it'll delete my files in three days. My job won't be
>> finished, and I'll be losing three days of work unless I catch that warning.
> 
> Then it will be high time you learn not to abuse /tmp that way and
> work in your (or your services) home/data directory.
> 
> Problem easily avoided. plus you don't need to make /tmp 20 TB because you
> have lots of data. ;)
> 
> I'm a bit surprised how many people seem to really rely on data in /tmp
> to survive for weeks or even months. I wonder if they backup /tmp?
Me is figurative here. Neither me, nor my code nor our users abuse these 
folders. The applications they use create these files without users’ knowledge. 

And yes, these applications rely on the data they saved on /tmp during the job. 
Again, let me repeat. These are not users' files, but applications internal 
data which they automatically create.

And sometimes these /tmp folders are put on high speed internal NVMe RAIDs to 
allow multiple GPUs work together with lower latency, for weeks. 
> -- 
> cheers,
>Holger
> 
> ⢀⣴⠾⠻⢶⣦⠀
> ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
> ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
> ⠈⠳⣄
> 
> "When one man dies it's a tragedy. When thousands die it's statistics." 
> (Stalin commenting the worlds reaction on Covid 19.)

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Hakan Bayındır]

> On 7 May 2024, at 18:57, Russ Allbery  wrote:
> 
> Hakan Bayındır  writes:
>> Dear Russ,
> 
>>> If you are running a long-running task that produces data that you
>>> care about, make a directory for it to use, whether in your home
>>> directory, /opt, /srv, whatever.
> 
>> Sorry but, clusters, batch systems and other automated systems doesn't
>> work that way.
> 
> Yours might not, but I spent 20 years maintaining clusters and batch
> systems and I assure you that's how mine worked.
> 
That’s nice. We’re in it for the same duration. 
>> That's not an extension of the home directory in any way. After users
>> submit their jobs to the cluster, they neither have access to the
>> execution node, nor they can pick and choose where to put their files.
> 
>> These files may stay there up to a couple of weeks, and deleting
>> everything periodically will probably corrupt the jobs of these users
>> somehow.
> 
> Using /var/tmp for this purpose is not a good design decision.
> Directories are free; they can make a new one and point the files of batch
> jobs there.  They don't have to overload a directory that historically has
> different semantics and is often periodically cleared.  I get that this
> may not be your design or something you have control over, so telling you
> this doesn't directly help, but the point still stands.
> 
> Again, obviously the people configuring that cluster can configure it
> however they want, including overriding the /var/tmp cleanup policy.  But
> they're playing with fire by training users to use /var/tmp, and it's
> going to result in someone getting their data deleted at some point,
> regardless of what Debian does.
> 
You still assume that we direct users' home directories to /var/tmp or /tmp. 
This is not true, users work on their own home folders, on a different storage 
system. Possibly I didn’t myself clear enough. 

The applications users use create these temporary files without users' 
knowledge. They work in their own directories, but applications create another 
job dependent state files in both /tmp and /var/tmp. These are different 
programs and I assure you they’re not created there because user (or we) 
configured something. These files live there during the lifetime of the job, 
and cleaned afterwards by the application. 
> -- 
> Russ Allbery (r...@debian.org)  
> 

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Hakan Bayındır  writes:
> Dear Russ,

>> If you are running a long-running task that produces data that you
>> care about, make a directory for it to use, whether in your home
>> directory, /opt, /srv, whatever.

> Sorry but, clusters, batch systems and other automated systems doesn't
> work that way.

Yours might not, but I spent 20 years maintaining clusters and batch
systems and I assure you that's how mine worked.

> That's not an extension of the home directory in any way. After users
> submit their jobs to the cluster, they neither have access to the
> execution node, nor they can pick and choose where to put their files.

> These files may stay there up to a couple of weeks, and deleting
> everything periodically will probably corrupt the jobs of these users
> somehow.

Using /var/tmp for this purpose is not a good design decision.
Directories are free; they can make a new one and point the files of batch
jobs there.  They don't have to overload a directory that historically has
different semantics and is often periodically cleared.  I get that this
may not be your design or something you have control over, so telling you
this doesn't directly help, but the point still stands.

Again, obviously the people configuring that cluster can configure it
however they want, including overriding the /var/tmp cleanup policy.  But
they're playing with fire by training users to use /var/tmp, and it's
going to result in someone getting their data deleted at some point,
regardless of what Debian does.

-- 
Russ Allbery (r...@debian.org)  

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Holger Levsen]

On Tue, May 07, 2024 at 04:24:06PM +0300, Hakan Bayındır wrote:
> Consider a long running task, which will take days or weeks (which is the
> norm in simulation and science domains in general). System emitted a warning
> after three days, that it'll delete my files in three days. My job won't be
> finished, and I'll be losing three days of work unless I catch that warning.

Then it will be high time you learn not to abuse /tmp that way and
work in your (or your services) home/data directory.

Problem easily avoided. plus you don't need to make /tmp 20 TB because you
have lots of data. ;)

I'm a bit surprised how many people seem to really rely on data in /tmp
to survive for weeks or even months. I wonder if they backup /tmp?


-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

"When one man dies it's a tragedy. When thousands die it's statistics." 
(Stalin commenting the worlds reaction on Covid 19.)


signature.asc
Description: PGP signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Hakan Bayındır]

Dear Russ,

It's not *me* using /var/tmp for my own temporary files, it's the 
applications other people use. I just logged in one of the nodes we have 
and there were job-dependent files created by a particular, high end 
scientific application (which is developed by another prominent 
company). This is neither in mine or users' control. It's the 
application they use, and I bet it has no setting for that.


> If you are running a long-running task that produces data that you
> care about, make a directory for it to use, whether in your home
> directory, /opt, /srv, whatever.

Sorry but, clusters, batch systems and other automated systems doesn't 
work that way.


> Not as an extension of people's home directory.

That's not an extension of the home directory in any way. After users 
submit their jobs to the cluster, they neither have access to the 
execution node, nor they can pick and choose where to put their files.


These files may stay there up to a couple of weeks, and deleting 
everything periodically will probably corrupt the jobs of these users 
somehow.


I for one understand that all the folders in a UNIX system have historic 
reasons and customs. I prefer to stick to the traditions and 
specifications come with them, however when you run a big system with 
tons of users who use tons of applications, I can't expect every 
software developer under the sun to know, understand and respect that 
conventions.


I just wanted to highlight a very prominent scenario from my vantage 
point, because it's the domain I'm working in.


> Your system is your system, so of course you can configure /var/tmp
> however you want and no one is going to stop you, but a lot of people
> on this thread are describing habits that are going to lose their data
> if they use a different distribution or even a differently-configured
> Debian distribution with tmpreaper installed.

Again, what I'm not describing is a *habit of mine*, but how many of the 
systems I interact with work, and there's no way to change that. I'm 
just pointing out how the systems we work with behave. We don't 
configure them that way. Heck, some of the applications our users use 
have no configuration file whatsoever.


I'm all for progress and a better, self-healing system, but I'm very 
against breaking things while doing that.


Cheers,

H.

On 7.05.2024 ÖS 5:32, Russ Allbery wrote:

Hakan Bayındır  writes:


Consider a long running task, which will take days or weeks (which is
the norm in simulation and science domains in general). System emitted a
warning after three days, that it'll delete my files in three days. My
job won't be finished, and I'll be losing three days of work unless I
catch that warning.


I have to admit that I'm a little surprised at the number of people who
are apparently using /var/tmp for things that are clearly not temporary
files in the traditional UNIX sense.  Clearly this bit of folk knowledge
is not as widespread as I thought, so we have to figure out how to deal
with that, but periodically deleting files out of /var/tmp has been common
(not universal, but common) UNIX practice for at least thirty years.

Whatever we do with /var/tmp retention, I beg people to stop using
/var/tmp for data you're keeping for longer than a few days and care about
losing.  That's not what it's for, and you *will* be bitten by this
someday, somewhere, because even with existing Debian configuration many
people run tmpreaper or similar programs.  If you are running a
long-running task that produces data that you care about, make a directory
for it to use, whether in your home directory, /opt, /srv, whatever.

/var/tmp's primary purpose historically was to support things like
temporary recovery files that needed to survive a system crash, but which
were still expected to be *temporary* in that one would then either use
the recovery file or expect it to be deleted.  Not as an extension of
people's home directory.

Your system is your system, so of course you can configure /var/tmp
however you want and no one is going to stop you, but a lot of people on
this thread are describing habits that are going to lose their data if
they use a different distribution or even a differently-configured Debian
distribution with tmpreaper installed.





OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Tue, 7 May 2024 at 15:53, Sam Hartman  wrote:
>
> > "Johannes" == Johannes Schauer Marin Rodrigues  
> > writes:
> >> > > If [files can be deleted automatically while mmdebstrap is using 
> them],
> >> > > how should applications guard against that from
> >> > > happening?
> >> >
> >> > As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
> >> > flock(2) lock on its chroot's root directory, systemd-tmpfiles should
> >> > fail to take out its own lock on the directory during cleanup, and
> >> > respond to that by treating the directory as "in use" and skipping 
> it.
> >>
> >> That also works, but only as long as mmdebootstrap is actually
> >> running, and as far as I understand it is not a long-running service,
> >> not sure if it works for this use case
>
> Note that according to the man page, ctime is used as well as mtime.
> So for roots that are actually temporary, I don't think much needs to be
> done.
> It won't matter that the mtime might be old because the ctime should be
> consistent with when the root is unpacked.
>
> I do wish there were a way to specify for /var/tmp that directories
> under /var/tmp should be deleted in their entirety or entirely left
> alone.
> I realize we'd have a big debate about whether that was a good default,
> but I'd find it useful for my systems at least.

This is a reasonable RFE, and it has already been proposed some days
ago (in the right place, upstream):
https://github.com/systemd/systemd/issues/32674

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Sam Hartman]

> "Johannes" == Johannes Schauer Marin Rodrigues  writes:
>> > > If [files can be deleted automatically while mmdebstrap is using 
them],
>> > > how should applications guard against that from
>> > > happening?
>> >
>> > As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
>> > flock(2) lock on its chroot's root directory, systemd-tmpfiles should
>> > fail to take out its own lock on the directory during cleanup, and
>> > respond to that by treating the directory as "in use" and skipping it.
>> 
>> That also works, but only as long as mmdebootstrap is actually
>> running, and as far as I understand it is not a long-running service,
>> not sure if it works for this use case

Note that according to the man page, ctime is used as well as mtime.
So for roots that are actually temporary, I don't think much needs to be
done.
It won't matter that the mtime might be old because the ctime should be
consistent with when the root is unpacked.

I do wish there were a way to specify for /var/tmp that directories
under /var/tmp should be deleted in their entirety or entirely left
alone.
I realize we'd have a big debate about whether that was a good default,
but I'd find it useful for my systems at least.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Andrey Rakhmatullin]

On Tue, May 07, 2024 at 04:24:06PM +0300, Hakan Bayındır wrote:
> On the other hand, if we need to change the configuration 99% of the time,
[citation needed]




-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Hakan Bayındır  writes:

> Consider a long running task, which will take days or weeks (which is
> the norm in simulation and science domains in general). System emitted a
> warning after three days, that it'll delete my files in three days. My
> job won't be finished, and I'll be losing three days of work unless I
> catch that warning.

I have to admit that I'm a little surprised at the number of people who
are apparently using /var/tmp for things that are clearly not temporary
files in the traditional UNIX sense.  Clearly this bit of folk knowledge
is not as widespread as I thought, so we have to figure out how to deal
with that, but periodically deleting files out of /var/tmp has been common
(not universal, but common) UNIX practice for at least thirty years.

Whatever we do with /var/tmp retention, I beg people to stop using
/var/tmp for data you're keeping for longer than a few days and care about
losing.  That's not what it's for, and you *will* be bitten by this
someday, somewhere, because even with existing Debian configuration many
people run tmpreaper or similar programs.  If you are running a
long-running task that produces data that you care about, make a directory
for it to use, whether in your home directory, /opt, /srv, whatever.

/var/tmp's primary purpose historically was to support things like
temporary recovery files that needed to survive a system crash, but which
were still expected to be *temporary* in that one would then either use
the recovery file or expect it to be deleted.  Not as an extension of
people's home directory.

Your system is your system, so of course you can configure /var/tmp
however you want and no one is going to stop you, but a lot of people on
this thread are describing habits that are going to lose their data if
they use a different distribution or even a differently-configured Debian
distribution with tmpreaper installed.

-- 
Russ Allbery (r...@debian.org)  

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

> ...3) I would put a file in any auto-cleaned space named "1-AUTOCLEAN.txt" 
> that contains some verbage explaining that things in this directory will be 
> wiped based on rules set in (wherever).

You know, that's a pretty good idea!

Put a 00README-TMP.txt in /tmp/ and /var/tmp/ which briefly states the
default deletion policy, the policy in place if it's not the default,
and a pointer to info about altering it. "/tmp's contents are deleted
at boot while /var/tmp is preserved across rebooting." Maybe in
/var/tmp suggest /var/scratch/ or /var/cache/tmp or such as a place
sysadmins might want to set up for not-backed-up but not-auto-deleted
material.

If the contents aren't dynamic, maybe they could be links to files in
/usr/share/doc/systemd/.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Alexandru Mihail]

> Consider a long running task, which will take days or weeks (which is
> the norm in simulation and science domains in general). System
> emitted a 
> warning after three days, that it'll delete my files in three days.
> My 
> job won't be finished, and I'll be losing three days of work unless I
> catch that warning.
> 
> Now consider these tasks are run on (dark) servers, where users'
> daemons 
> login to run the tasks but users do not. How can the user know? What
> can 
> they do? Same can be said for long running daemons like mail servers,
> CI 
> runners and such.
> 
> One may argue that we can change the configuration, which is true.
> 
You're making a strong argument here, indeed. I personally manage a
horde of bookworm VMs which I really don't want to watch for precisely
minute changes that break the system in subtle ways like this.
Putting /tmp in RAM also won't work, as discussed here. Apart from
SBCs, all other Debian use cases would not benefit from this. There's
plenty of people using cloud virtual machines happily running Debian
with 1 GB of ram, running clamav, mail servers, apache, etc, etc. Let's
not eat up that ram further if we really don't have to. I know because
I am such an user and certainly not the only one. I don't know how this
would impact people who run a boatload of containers with Debian too.

> On the other hand, if we need to change the configuration 99% of the 
> time, why are we making the change to a worse one in the first place?
> 
There's been quite some debate here which is good. This sets us apart
from corporate-run Linux in that there's technical democracy in
decisions impacting users. Maybe listing and weighing, calmly, the pros
and cons of this decision and how bad/good will the impact be on
programs and users could help drive a decision.
For instance, adopting this behavior:
- aligns us with upstream (neutral in my opinion)
- prevents clutter in /var/tmp by misbehaving applications mainly  
(users filling up their drive is their fault; they can still dd of=/  
anyway; we shouldn't put baby wheels on our OS) (good)
- might surprise users who got used to /var/tmp as being a scratch
space since a long time ago and might cause frustration if their files
randomly disappear. Yes, they shouldn't be storing files there anyway,
but deleting them without them explicitly setting that mechanism up or
hitting Y on a big prompt isn't helpful. (regression)
- might require updates to older applications which wrongly use
/var/tmp as discussed above. (neutral, that wasn't a sturdy mechanism
anyway)
Thanks,
Alexandru


signature.asc
Description: This is a digitally signed message part

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[rhys]

The /tmp/ as tmpfs discussion is funny to me because while we've been kicking 
around the idea of whether or not to clean /tmp/, having /tmp/ as a tmpfs makes 
that whole argument moot. It all goes away at boot time! Problem solved! :D

Honestly, I see this one as a much easier topic, assuming that no one is 
talking about changing existing systems. (I haven't seen anyone say that.)

So for new systems, /tmp/ as a tmpfs strikes me as a legitimate option, and the 
partition layout is something that any good admin pays close attention to on 
any new system, particularly a new distribution or even distro version. (That 
is, even going from 12.1 to 12.2, I'm going to be on the lookout for changes in 
the installer.)

Whether you want /tmp as a tmpfs is a decision that's going to be made at the 
same time as whether or not /home should be on a separate partition. The admin 
is going to do whatever makes the most sense for this system. 

To me, it's all about the display. I want to see what partition will be mounted 
as root, what partition will be mounted as /home, which will be swap (if any), 
and so on. But I don't need to see /proc and /sys. Those aren't optional. 

So if /tmp is not part of the root partition, it doesn't matter if it's a 
separate partition or a tmpfs. It should just appear in the screen that 
displays the filesystem layout, and then the admin can decide whether or not 
that's a good idea. 

I have no opinion on whether or not it should be the default. If /tmp/ as tmpfs 
becomes the default, I would probably only override that on certain low-memory 
systems that I run and just leave it on most others. I've seen it done before 
and it seemed to work fine in some cases and not in others. 

As long as it's somewhere that I can SEE it in the installer, I'd be happy. 
That's definitely a thing the admin can change later on with few consequences. 

Sent from my mobile device.


From: Hakan Bayındır 
Sent: Tuesday, May 7, 2024 05:45
To: 966...@bugs.debian.org; debian-devel@lists.debian.org
Cc: Carsten Leonhardt; Luca Boccassi; Peter Pentchev
Subject: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default 
[was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

Similarly, I’m following the thread for a couple of days now, and wondering 
about its implications. 

When I consider server scenarios, pushing /tmp to RAM looks highly undesirable 
from my perspective. All the servers I manage use their whole RAMs and using 
the unused space as a disk cache is far more desirable than a /tmp mount. When 
servers are virtualized, RAM is at premium, and a disk cache is way more usable 
rather than /tmp in the RAM. 

The other scenario I think is HPC, where applications use all the RAM 
available, squeezing the hardware dry. Again, /tmp in the RAM is very 
undesirable, because /tmp/$USER is also heavily used and an OOM situation 
creates a lose-lose situation where you either delete runtime files, or lose 
the executing job, which results in job failure in any case. 

On the other hand, I personally use my desktop computer as a development 
workstation and use tons of RAM either with software or with VMs. Again a /tmp 
in RAM is an inferior scenario to my current setup. 

The only useful state where /tmp is in RAM is single board computers where temp 
is both lightly utilized and maximizing SD/eMMC life is important. These 
systems even mount /var/log to a tmpfs and sync on boot/reboot/shutdown, 
reducing flash wear. 

Deleting /var/tmp has the same problems since long running tasks on the servers 
might need a file once in a month, but it can be crucial for functions of the 
software. 

I can’t see any scenario where these two are useful in typical or niche 
installations of Debian. 

FWIW, RedHat family doesn’t mount /tmp as a tmpfs on its default installation. 

Cheers, 

H. 

Re: Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Alexandru Mihail]

Maybe putting the cleanup task for /var/tmp on a longer timer and
warning users ahead of time of impending deletion (maybe 3 days before,
2 days, etc) would help with files of unsuspecting users getting
deleted. A log entry could also be emitted. I could see a gentle
warning on ssh login (minimal, one or two lines) and desktop
notification (for desktop only users who never see the terminal) be
helpful. A smarter implementation could perhaps only warn if dirs/files
that are going to be deleted are not systemd generated random items.
This does not fix issues with applications depending on stuff being
there long term; yet again nothing's perfect in software


signature.asc
Description: This is a digitally signed message part

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[rhys]

I'm always in favor of logging system changes. 

Notification at run time is really tricky. No one ever logs into several of my 
Debian servers. Other systems have interactive GUI or CLI users, some of whom 
are admins and others not. 

I don't know if login notices are terribly reliable as no one may ever see 
them. I see nothing wrong with having them, but I wouldn't consider them to be 
a good *primary* notification.

Personally, I would resort to email, as it's most likely that a good admin has 
at least set up root mail to go somewhere appropriate. 

Whether or not root should get copied on notifications sent to other users 
strikes me as a security question, though if the data to be deleted is in a 
shared scratch space like /tmp/ then perhaps that's not a concern (and the 
potential for lost data might override any such concerns anyway, plus the fact 
that root is, well, root in the first place). I would argue that the role of 
"root as helpful admin" should prevail in this case and root should get copied. 

I, for one, don't want a lot of email, but once the data is gone, it's GONE. 
I'd rather be notified and have to deal with it than not be told at all. 

Sent from my mobile device.

From: Alexandru Mihail 
Sent: Tuesday, May 7, 2024 07:59
To: debian-devel@lists.debian.org
Subject: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default 
[was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

Maybe putting the cleanup task for /var/tmp on a longer timer and warning users 
ahead of time of impending deletion (maybe 3 days before, 2 days, etc) would 
help with files of unsuspecting users getting deleted. A log entry could also 
be emitted. I could see a gentle warning on ssh login (minimal, one or two 
lines) and desktop notification (for desktop only users who never see the 
terminal) be helpful. A smarter implementation could perhaps only warn if 
dirs/files that are going to be deleted are not systemd generated random items. 
This does not fix issues with applications depending on stuff being there long 
term; yet again nothing's perfect in software 

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Hakan Bayındır]

Consider a long running task, which will take days or weeks (which is 
the norm in simulation and science domains in general). System emitted a 
warning after three days, that it'll delete my files in three days. My 
job won't be finished, and I'll be losing three days of work unless I 
catch that warning.


Now consider these tasks are run on (dark) servers, where users' daemons 
login to run the tasks but users do not. How can the user know? What can 
they do? Same can be said for long running daemons like mail servers, CI 
runners and such.


One may argue that we can change the configuration, which is true.

On the other hand, if we need to change the configuration 99% of the 
time, why are we making the change to a worse one in the first place?


On 7.05.2024 ÖS 3:59, Alexandru Mihail wrote:

Maybe putting the cleanup task for /var/tmp on a longer timer and warning users 
ahead of time of impending deletion (maybe 3 days before, 2 days, etc) would 
help with files of unsuspecting users getting deleted. A log entry could also 
be emitted. I could see a gentle warning on ssh login (minimal, one or two 
lines) and desktop notification (for desktop only users who never see the 
terminal) be helpful. A smarter implementation could perhaps only warn if 
dirs/files that are going to be deleted are not systemd generated random items. 
This does not fix issues with applications depending on stuff being there long 
term; yet again nothing's perfect in software




OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[rhys]

Perhaps ones *intentions* are good, but when you're making decisions for 
someone else, it is often necessary to take a step back and look at the bigger 
picture of what one is doing. 

I do not consider myself to be immune to the effects of this thought process. I 
have kids. They remind me constantly that I'm just as subject to the "I know 
better" mindset as anyone else. 

But you said the best thing to do is "keep an open mind." What Luca has 
literally said multiple times is "I've already made my decision and I'm only 
looking for comments that support the technical aspects of that decision."

That squashes discussion. Only one person has made such comments, and that 
person was not me. 

On a more procedural note...

I have repeatedly voiced my opinion that automatically deleting files is a bad 
idea and I stand by that opinion for reasons already stated. 

However, recognizing that I don't always get my way, I've thought about how one 
might "do it anyway" while still addressing my underlying concerns and here's 
what I came up with:

The two biggest things that are underneath are a) applications that don't clean 
up after themselves and b) a change in how the system behaves vs. what the 
users expect. 

To address a) in a coherent way, if I had infinite resources, I would create a 
package similar to popularity-contest that is obvious and optional and reports 
back what commonly appears in various scratch spaces. That is how I would 
gather a wide range of data on what packages don't behave well. D-i can allow 
the admin to opt in or not on the same screen as the one that asks about 
popularity-contest. (Those that opt in or out will likely do so for that same 
reasons, after all.)

As for b) the underlying problem is the change in *expected behavior* of the 
system. The real problem has nothing to do with whether or not it's technically 
a good idea. It's a shift in expectation with potentially disastrous 
consequences. Deleted files are often just gone. 

So to mitigate that, I would 1) only implement it on new installs (we'll come 
back to this), 2) mention it at least twice in the various d-i screens, and...

...3) I would put a file in any auto-cleaned space named "1-AUTOCLEAN.txt" that 
contains some verbage explaining that things in this directory will be wiped 
based on rules set in (wherever). 

This is how files like /etc/resolv.conf read when they are controlled by other 
processes. They just have text that tells you, "Don't change this directly. 
Your changes will be overwritten. Make your changes in (canonical place)."

Last but not least, I would go ahead and deploy the packages that automatically 
clean tmp spaces even to existing systems, but their default configuration 
would be disabled. The only thing that would enable them would be a) 
debian-installer (optionally, possibly as default), or b) admins who have heard 
about this and decide it's a good idea. 

Some will opt in, some won't. But the packages and their default configs could 
be pushed out safely. No one (even me) would have a reasonable complaint about 
such an arrangement. 

That would allow the expectation to shift over time while significantly 
reducing the number of surprised users who get their data deleted. 

In the end, this is all for people to use. So to me, it's much more an issue of 
making sure the people know what they're using and how to use it than anything 
else. If it were just a technical issue, this would be a much shorter 
conversation. ;)

--J

Sent from my mobile device.


From: "Barak A. Pearlmutter" 
Sent: Tuesday, May 7, 2024 07:18
To: r...@neoquasar.org
Cc: Luca Boccassi; debian-devel@lists.debian.org
Subject: Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default 
[was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

Rhys, I think you're being unfair. We have a *technical* disagreement 
here. But our hearts are all in the same place: Luca, myself, and all 
the other DDs discussing this, all want what's best for our users, we 
all want to build the best OS possible, and are all discussing the 
issue in good faith. 

There is an unavoidable tension, and we're hashing it out. Upstream 
has fielded a default behaviour which requires adjustment of a variety 
of other programs and workflows. Basically, anything that stores stuff 
in /tmp or /var/tmp needs to be made might-be-deleted-aware. There are 
mechanisms for dealing with this, but they're pretty complicated, and 
differ wildly for different file lifetimes etc. Other distributions 
have adopted that default, and rather than using exposed mechanisms 
for avoiding unexpected deletion, are just telling people not to count 
on files in /var/tmp/ surviving a reboot if the computer is shut down 
more than a month, or whatever. What should Debian do? You can make 
arguments both ways, and we are. Generally we follow upstream unless 
there's a compelling reason not to. You can suggest 

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Hakan Bayındır]

Similarly, I’m following the thread for a couple of days now, and wondering 
about its implications.

When I consider server scenarios, pushing /tmp to RAM looks highly undesirable 
from my perspective. All the servers I manage use their whole RAMs and using 
the unused space as a disk cache is far more desirable than a /tmp mount. When 
servers are virtualized, RAM is at premium, and a disk cache is way more usable 
rather than /tmp in the RAM.

The other scenario I think is HPC, where applications use all the RAM 
available, squeezing the hardware dry. Again, /tmp in the RAM is very 
undesirable, because /tmp/$USER is also heavily used and an OOM situation 
creates a lose-lose situation where you either delete runtime files, or lose 
the executing job, which results in job failure in any case.

On the other hand, I personally use my desktop computer as a development 
workstation and use tons of RAM either with software or with VMs. Again a /tmp 
in RAM is an inferior scenario to my current setup.

The only useful state where /tmp is in RAM is single board computers where temp 
is both lightly utilized and maximizing SD/eMMC life is important. These 
systems even mount /var/log to a tmpfs and sync on boot/reboot/shutdown, 
reducing flash wear.

Deleting /var/tmp has the same problems since long running tasks on the servers 
might need a file once in a month, but it can be crucial for functions of the 
software.

I can’t see any scenario where these two are useful in typical or niche 
installations of Debian.

FWIW, RedHat family doesn’t mount /tmp as a tmpfs on its default installation.

Cheers,

H.

> On 7 May 2024, at 12:42, Peter Pentchev  wrote:
> 
> On Tue, May 07, 2024 at 10:38:14AM +0200, Carsten Leonhardt wrote:
>> Luca Boccassi  writes:
>> 
>>> Defaults are defaults, they are trivially and fully overridable where
>>> needed if needed. Especially container and VM managers these days can
>>> super trivially override them via SMBIOS Type11 strings or
>>> Credentials, ephemerally and without changing the guest image at all.
>> 
>> That argument goes both ways and I prefer safe defaults. What
>> you/upstream propose are unsafe defaults, as was shown by several
>> comments in this thread. Whoever wants the unsafe defaults of deleting
>> old files and risking OOM situations can than "trivially and fully
>> override" the safe defaults.
> 
> So I've been wondering for a couple of days now, following this thread...
> ...would it be a good idea to make this a debconf prompt, high priority,
> default "yes", so that it is activated on new automatically installed
> systems, but people who upgrade their current Debian installations can
> choose to keep the old behavior?
> 
> I do realize that more debconf prompts are not always desirable, and
> such decisions must be taken on a case-by-case basis, so... yeah.
> 
> G'luck,
> Peter
> 
> -- 
> Peter Pentchev  r...@ringlet.net r...@debian.org pe...@morpheusly.com
> PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
> Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon McVittie]

On Tue, 07 May 2024 at 07:34:54 -0500, r...@neoquasar.org wrote:
> possibly convince those applications to use their own
> scratch space such as /tmp// that is more easily identifiable

This would be a denial of service at best, and a privilege escalation
vulnerability at worst. To be safe, it would have to be more like
/tmp/.XX where the XX is replaced by a random string
by mkstemp() or similar.

(For example my system currently has /var/tmp/flatpak-cache-5X58M2/ which
is fine, but using /var/tmp/flatpak-cache/ would be wrong.)

smcv

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Alexandru Mihail]

Maybe putting the cleanup task for /var/tmp on a longer timer and warning users 
ahead of time of impending deletion (maybe 3 days before, 2 days, etc) would 
help with files of unsuspecting users getting deleted. A log entry could also 
be emitted. I could see a gentle warning on ssh login (minimal, one or two 
lines) and desktop notification (for desktop only users who never see the 
terminal) be helpful. A smarter implementation could perhaps only warn if 
dirs/files that are going to be deleted are not systemd generated random items. 
This does not fix issues with applications depending on stuff being there long 
term; yet again nothing's perfect in software

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[rhys]

This, in my opinion, is the correct view. 

If the users/admins of a system are putting files somewhere, those are their 
files and therefore their responsibility. It is not up to anyone else to claim 
they know better and clean up after them. 

If the files are abandoned by applications that don't clean up after 
themselves, the applications should be updated to clean up properly, since only 
those applications understand the full context of what is or isn't still "in 
use" or needed (and possibly convince those applications to use their own 
scratch space such as /tmp// that is more easily identifiable). 
Checking the documentation of those applications for descriptions of what ends 
up in [/var]/tmp/ would also be useful, since that informs the admin's decision 
on how to deal with possibly abandoned temp files. 

Making the applications behave properly and trusting the system owners to run 
their systems as they see fit will always be the better choice. 

Otherwise, all that will happen is that over time, another scratch space that 
does not automatically get reaped will appear, users and apps will migrate to 
that space, and we will all be back where we started. (No, they won't just 
"change the defaults" because that's not a stable process. One admin may allow 
that while another doesn't.)

--J

Sent from my mobile device.


From: Philip Hands 
Sent: Tuesday, May 7, 2024 06:31

This makes me wonder what it is that we're expecting to need to delete.

Is this a symptom of sloppy applications that fail to clear up the
debris they create in /var/tmp?  If so, is that not a bug in that
application?

I'd suggest that rather than clearing up after the sloppy behaviour of
buggy applications, we instead leave it visible, in the hope that it can
then be fixed.

Of course, that's obviously not worked in some (many?) cases, so where
we know of problematic packages, could we not add per-package tmpfiles.d
files that name the specific paths that those packages are known to
litter the system with, with appropriate deletion timeouts chosen by the
Maintainer?

That ought to achieve the benefit you're looking for, without hiding
symptoms of future problems with other packages, and without
inconveniencing anyone that's using /var/tmp as scratch space.

Cheers, Phil.
--
Philip Hands -- https://hands.com/~phil

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

Rhys, I think you're being unfair. We have a *technical* disagreement
here. But our hearts are all in the same place: Luca, myself, and all
the other DDs discussing this, all want what's best for our users, we
all want to build the best OS possible, and are all discussing the
issue in good faith.

There is an unavoidable tension, and we're hashing it out. Upstream
has fielded a default behaviour which requires adjustment of a variety
of other programs and workflows. Basically, anything that stores stuff
in /tmp or /var/tmp needs to be made might-be-deleted-aware. There are
mechanisms for dealing with this, but they're pretty complicated, and
differ wildly for different file lifetimes etc. Other distributions
have adopted that default, and rather than using exposed mechanisms
for avoiding unexpected deletion, are just telling people not to count
on files in /var/tmp/ surviving a reboot if the computer is shut down
more than a month, or whatever. What should Debian do? You can make
arguments both ways, and we are. Generally we follow upstream unless
there's a compelling reason not to. You can suggest various strategies
for making things reliable despite following upstream. You can discuss
why maybe upstream should not be followed in this case. This is
precisely the kind of discussion that leads to good decisions, with
everyone keeping an open mind and sharing information and ideas.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Philip Hands]

Luca Boccassi  writes:

> On Mon, 6 May 2024 at 11:33, Barak A. Pearlmutter  wrote:
>>
>> > We have two separate issues here:
>>
>> > a/ /tmp-on-tmpfs
>> > b/ time based clean-up of /tmp and /var/tmp
>>
>> > I think it makes sense to discuss/handle those separately.
>>
>> Agreed.
>>
>> I also don't see any issue with a/, at worst people will be annoyed
>> with it for some reason and can then change it back.
>>
>> > Regarding b/: ...
>>
>> > The tmpfiles rule tmp.conf as shipped by systemd upstream ...
>> > Files that are older then 10 days or 30 days are automatically cleaned up.
>>
>> This seems like a rather dangerous thing to spring on people.
>>
>> First of all, time can be pretty fluid on user machines.
>
> Then upon reading the release notes, on such a machine, one can simply do:
>
> touch /etc/tmpfiles.d/tmp.conf
>
> And they get no automated cleanups. This stuff is designed to be
> trivially overridable, both by end-users and image builders. What I am
> looking for is, what packages need bugs/MRs filed to deal with this
> change, if any.

Isn't this change (as presented) effectively about masking bugs?

We've had people suggesting that implementing this will surprise them
and disrupt their existing use of /var/tmp as scratch storage, and I've
got a lot of sympathy with that, so I'm guessing that the people that
are expected to benefit from this are not those that remember systems
where the main distinction between /tmp and /var/tmp was that /tmp got
emptied at boot time, whereas /var/tmp did not.

That makes me assume that those that would be most likely benefit from
such a change will mainly be users that are never going to type "/tmp"
(with or without a preceding "/var"), and are therefore not going to
have any idea what is being deleting for them, but will be happy never
to get their disk filled.

This makes me wonder what it is that we're expecting to need to delete.

Is this a symptom of sloppy applications that fail to clear up the
debris they create in /var/tmp?  If so, is that not a bug in that
application?

I'd suggest that rather than clearing up after the sloppy behaviour of
buggy applications, we instead leave it visible, in the hope that it can
then be fixed.

Of course, that's obviously not worked in some (many?) cases, so where
we know of problematic packages, could we not add per-package tmpfiles.d
files that name the specific paths that those packages are known to
litter the system with, with appropriate deletion timeouts chosen by the
Maintainer?

That ought to achieve the benefit you're looking for, without hiding
symptoms of future problems with other packages, and without
inconveniencing anyone that's using /var/tmp as scratch space.

Cheers, Phil.
-- 
Philip Hands -- https://hands.com/~phil


signature.asc
Description: PGP signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Peter Pentchev]

On Tue, May 07, 2024 at 10:38:14AM +0200, Carsten Leonhardt wrote:
> Luca Boccassi  writes:
> 
> > Defaults are defaults, they are trivially and fully overridable where
> > needed if needed. Especially container and VM managers these days can
> > super trivially override them via SMBIOS Type11 strings or
> > Credentials, ephemerally and without changing the guest image at all.
> 
> That argument goes both ways and I prefer safe defaults. What
> you/upstream propose are unsafe defaults, as was shown by several
> comments in this thread. Whoever wants the unsafe defaults of deleting
> old files and risking OOM situations can than "trivially and fully
> override" the safe defaults.

So I've been wondering for a couple of days now, following this thread...
...would it be a good idea to make this a debconf prompt, high priority,
default "yes", so that it is activated on new automatically installed
systems, but people who upgrade their current Debian installations can
choose to keep the old behavior?

I do realize that more debconf prompts are not always desirable, and
such decisions must be taken on a case-by-case basis, so... yeah.

G'luck,
Peter

-- 
Peter Pentchev  r...@ringlet.net r...@debian.org pe...@morpheusly.com
PGP key:http://people.FreeBSD.org/~roam/roam.key.asc
Key fingerprint 2EE7 A7A5 17FC 124C F115  C354 651E EFB0 2527 DF13


signature.asc
Description: PGP signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Carsten Leonhardt]

Luca Boccassi  writes:

> Defaults are defaults, they are trivially and fully overridable where
> needed if needed. Especially container and VM managers these days can
> super trivially override them via SMBIOS Type11 strings or
> Credentials, ephemerally and without changing the guest image at all.

That argument goes both ways and I prefer safe defaults. What
you/upstream propose are unsafe defaults, as was shown by several
comments in this thread. Whoever wants the unsafe defaults of deleting
old files and risking OOM situations can than "trivially and fully
override" the safe defaults.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[rhys]

You're now at the stage where you're not just MISSING the point of what people 
are trying to tell you, you're actively IGNORING it. 

Automatically deleting files is a bad idea. Those files aren't yours. You don't 
know why they are there. Leave them alone. 

--J

Sent from my mobile device.


From: Luca Boccassi 
Sent: Monday, May 6, 2024 08:20
To: Barak A. Pearlmutter
Cc: debian-devel@lists.debian.org
Subject: Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default 
[was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

On Mon, 6 May 2024 at 13:42, Barak A. Pearlmutter  wrote: 
> 
> > Then upon reading the release notes, on such a machine, one can simply do: 
> > 
> > touch /etc/tmpfiles.d/tmp.conf 
> > 
> > And they get no automated cleanups. 
> 
> This also disables on-boot cleaning of /tmp/. 

Yes, as it's going to be a tmpfs, so it is no longer needed. Trivial 
to maintain though if one wants to do so, though. 

> The root issue here is that deleting not-read-in-a-while 
> but-maybe-stat'ed-recently-by-make-that-doesn't-count files from 
> /var/tmp/ by default, particularly when the system didn't used to, 
> violates the principle of least surprise. 

Which is what release notes are for, if everything was always the same 
we wouldn't spend time to put those together

> There's an old debugging story 

While personal anecdotes and stories can be interesting and amusing in 
many circumstances, I am not really looking for those at this very 
moment. What I am looking for right now is packages or internal 
infrastructure that need 
an update to cope with these two changes before I upload them, so if 
you know of any please do let me know and I'll happily look into it 
and at least file a bug, if not a MR. Thanks. 

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Luca Boccassi  writes:
> Richard Lewis  wrote:

>> - tmux stores sockets in /tmp/tmux-$UID
>> - I think screen might use /tmp/screens

>> I suppose if you detached for a long time you might find yourself
>> unable to reattach.

>> I think you can change the location of these.

> And those are useful only as long as screen/tmux are still running,
> right (I don't really use either that much)? If so, a flock is the right
> solution for these

Also, using /tmp as a path for those sockets was always a questionable
decision.  I believe current versions of screen use /run/screen, which is
a more reasonable location.  Using a per-user directory would be even
better, although I think screen intentionally supports shared screens
between users (which is a somewhat terrifying feature from a security
standpoint, but that's a different argument).

-- 
Russ Allbery (r...@debian.org)  

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Luca Boccassi (2024-05-07 00:09:51)
> To be more specific, as per documentation:
> 
> https://www.freedesktop.org/software/systemd/man/latest/tmpfiles.d.html
> 
> 'x' lines can be used to override cleanup rules, and support globbing,
> so something like:
> 
> x /tmp/mmdebstrap.*

thank you for being patient with me. I saw the man page but I also tried using
codesearch to look for other packages doing the same thing already and was
unable to find one. This made me doubt whether I had understood this correctly.
For example, tmpfiles.d(5) makes it look like there are several minuses
required after the path but apparently those are optional. I've never written
such files before, so your input is useful for me, thank you!

> > And just to confirm (I read this elsewhere in this thread): if my
> > /etc/fstab has an entry for /tmp (with a tmpfs) does this automatically
> > mean that no cleanup will happen or do i still have to put something into
> > /etc to disable the periodic cleanup?
> That's something different, fstab is about whether /tmp is a tmpfs or not,
> cleanups still happen regardless of the filesystem type.

I shall put the following into my local /etc/tmpfiles.d/josch.conf to disable
the cleanup of /tmp completely, then:

x /tmp

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 23:03, Richard Lewis
 wrote:
>
> Luca Boccassi  writes:
>
> > Hence, I am not really looking for philosophical discussions or lists
> > of personal preferences or hypotheticals, but for facts: what would
> > break where, and how to fix it?
>
> - tmux stores sockets in /tmp/tmux-$UID
> - I think screen might use /tmp/screens
>
> I suppose if you detached for a long time you might find yourself unable
> to reattach.
>
> I think you can change the location of these.

And those are useful only as long as screen/tmux are still running,
right (I don't really use either that much)? If so, a flock is the
right solution for these

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 23:00, Johannes Schauer Marin Rodrigues
 wrote:
>
> Quoting Luca Boccassi (2024-05-06 23:28:59)
> > On Mon, 6 May 2024 at 22:27, Simon McVittie  wrote:
> > >
> > > On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues 
> > > wrote:
> > > > If [files can be deleted automatically while mmdebstrap is using them],
> > > > how should applications guard against that from
> > > > happening?
> > >
> > > As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
> > > flock(2) lock on its chroot's root directory, systemd-tmpfiles should
> > > fail to take out its own lock on the directory during cleanup, and
> > > respond to that by treating the directory as "in use" and skipping it.
> >
> > That also works, but only as long as mmdebootstrap is actually
> > running, and as far as I understand it is not a long-running service,
> > not sure if it works for this use case
>
> I guess those users who want to place their chroots in /tmp would have to
> disable the cleanup timer on their systems.
>
> For the mmdebstrap user who just wants to create some tarballs, I think the
> flock solution would be sufficient, thank you.
>
> > More specific settings win, so your application can ship its own tmpfiles.d
> > snippet that defines a longer (or infinite) cleanup time for directories
> > under /tmp or /var/tmp.
>
> How would that look like? Say, my application creates temporary directories
> with the pattern /tmp/mmdebstrap. -- how would the recommended line
> which disables cleanup in /usr/lib/tmpfiles.d/mmdebstrap.conf look like?

To be more specific, as per documentation:

https://www.freedesktop.org/software/systemd/man/latest/tmpfiles.d.html

'x' lines can be used to override cleanup rules, and support globbing,
so something like:

x /tmp/mmdebstrap.*

> And just to confirm (I read this elsewhere in this thread): if my /etc/fstab
> has an entry for /tmp (with a tmpfs) does this automatically mean that no
> cleanup will happen or do i still have to put something into /etc to disable
> the periodic cleanup?

That's something different, fstab is about whether /tmp is a tmpfs or
not, cleanups still happen regardless of the filesystem type.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Quoting Luca Boccassi (2024-05-06 23:28:59)
> On Mon, 6 May 2024 at 22:27, Simon McVittie  wrote:
> >
> > On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues 
> > wrote:
> > > If [files can be deleted automatically while mmdebstrap is using them],
> > > how should applications guard against that from
> > > happening?
> >
> > As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
> > flock(2) lock on its chroot's root directory, systemd-tmpfiles should
> > fail to take out its own lock on the directory during cleanup, and
> > respond to that by treating the directory as "in use" and skipping it.
> 
> That also works, but only as long as mmdebootstrap is actually
> running, and as far as I understand it is not a long-running service,
> not sure if it works for this use case

I guess those users who want to place their chroots in /tmp would have to
disable the cleanup timer on their systems.

For the mmdebstrap user who just wants to create some tarballs, I think the
flock solution would be sufficient, thank you.

> More specific settings win, so your application can ship its own tmpfiles.d
> snippet that defines a longer (or infinite) cleanup time for directories
> under /tmp or /var/tmp.

How would that look like? Say, my application creates temporary directories
with the pattern /tmp/mmdebstrap. -- how would the recommended line
which disables cleanup in /usr/lib/tmpfiles.d/mmdebstrap.conf look like?

And just to confirm (I read this elsewhere in this thread): if my /etc/fstab
has an entry for /tmp (with a tmpfs) does this automatically mean that no
cleanup will happen or do i still have to put something into /etc to disable
the periodic cleanup?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Richard Lewis]

Luca Boccassi  writes:

> Hence, I am not really looking for philosophical discussions or lists
> of personal preferences or hypotheticals, but for facts: what would
> break where, and how to fix it?

- tmux stores sockets in /tmp/tmux-$UID
- I think screen might use /tmp/screens

I suppose if you detached for a long time you might find yourself unable
to reattach.

I think you can change the location of these.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 22:27, Simon McVittie  wrote:
>
> On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues wrote:
> > If [files can be deleted automatically while mmdebstrap is using them],
> > how should applications guard against that from
> > happening?
>
> As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
> flock(2) lock on its chroot's root directory, systemd-tmpfiles should
> fail to take out its own lock on the directory during cleanup, and
> respond to that by treating the directory as "in use" and skipping it.

That also works, but only as long as mmdebootstrap is actually
running, and as far as I understand it is not a long-running service,
not sure if it works for this use case

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 21:08, Johannes Schauer Marin Rodrigues
 wrote:
>
> Hi,
>
> Quoting Luca Boccassi (2024-05-06 15:20:08)
> > While personal anecdotes and stories can be interesting and amusing in many
> > circumstances, I am not really looking for those at this very moment. What I
> > am looking for right now is packages or internal infrastructure that need an
> > update to cope with these two changes before I upload them, so if you know 
> > of
> > any please do let me know and I'll happily look into it and at least file a
> > bug, if not a MR. Thanks.
>
> mmdebstrap uses $TMPDIR (because that's what Perl's File::Temp uses) and falls
> back to /tmp for storing the chroot directories it creates. The modification
> times on files in that chroot directory are potentially much older than 10 
> days
> as they match the timestamps on the packages installed in the chroot. Do I
> understand it correctly that with the proposed change and an unfortunate 
> timing
> it can happen that files older than 10 days get removed from a temporary 
> chroot
> directory in /tmp? If yes, how should applications guard against that from
> happening?

More specific settings win, so your application can ship its own
tmpfiles.d snippet that defines a longer (or infinite) cleanup time
for directories under /tmp or /var/tmp.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon McVittie]

On Mon, 06 May 2024 at 22:08:56 +0200, Johannes Schauer Marin Rodrigues wrote:
> If [files can be deleted automatically while mmdebstrap is using them],
> how should applications guard against that from
> happening?

As documented in tmpfiles.d(5), if mmdebstrap takes out an exclusive
flock(2) lock on its chroot's root directory, systemd-tmpfiles should
fail to take out its own lock on the directory during cleanup, and
respond to that by treating the directory as "in use" and skipping it.

smcv

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 21:30, Salvo Tomaselli  wrote:
>
> On a fresh installed fedora system I downloaded a .iso in /tmp, then the
> OOMkiller killed wayland, so everything died.
>
> If you know you won't ever fill it up, I guess it's fine. But I'd go for the
> safer (and sadly slower) option.

You can just configure it (or disable it) on older devices or where
you don't want it, it's very easy to do as already mentioned

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Richard Lewis]

Luca Boccassi  writes:

> On Mon, 6 May 2024 at 15:42, Richard Lewis
>  wrote:
>>
>> Luca Boccassi  writes:
>>
>> > Hence, I am not really looking for philosophical discussions or lists
>> > of personal preferences or hypotheticals, but for facts: what would
>> > break where, and how to fix it?
>>
>> cleaning /tmp or /var/tmp: users may lose files if they dont realise a
>>   directory tmp can be cleaned without a reboot. something in /var/tmp
>>   that's been in there for 35 days before an upgrade might be deleted
>>   before the user reads the NEWS.Debian email, meaning they have no
>>   chance to react). Maybe you could postpone the very first deletion
>>   until after the next reboot?
>>
>> using a tmpfs: is there a risk of losing unrelated data due to more
>>   frequent OOM killing random other programmes due to /tmp using all the
>>   memory?  is there a case to only use a tmpfs if the system has
>>   "enough" memory?
>
> Again, those are all hypotheticals, and one can construct similarly
> contrived thought exercises for most possible permutations of most
> configurations, and the answer is always the same: customize the
> configuration accordingly. Hence, not relevant right now.

> What is relevant is: which packages, if any, or which DSA-owned
> systems, if any, are actually affected and how?

Why do you think that the impact on users is less important than the
impact on debian packages?

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Johannes Schauer Marin Rodrigues]

Hi,

Quoting Luca Boccassi (2024-05-06 15:20:08)
> While personal anecdotes and stories can be interesting and amusing in many
> circumstances, I am not really looking for those at this very moment. What I
> am looking for right now is packages or internal infrastructure that need an
> update to cope with these two changes before I upload them, so if you know of
> any please do let me know and I'll happily look into it and at least file a
> bug, if not a MR. Thanks.

mmdebstrap uses $TMPDIR (because that's what Perl's File::Temp uses) and falls
back to /tmp for storing the chroot directories it creates. The modification
times on files in that chroot directory are potentially much older than 10 days
as they match the timestamps on the packages installed in the chroot. Do I
understand it correctly that with the proposed change and an unfortunate timing
it can happen that files older than 10 days get removed from a temporary chroot
directory in /tmp? If yes, how should applications guard against that from
happening?

Thanks!

cheers, josch

signature.asc
Description: signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Andrey Rakhmatullin]

On Mon, May 06, 2024 at 04:50:50PM +0100, Barak A. Pearlmutter wrote:
> > tmpfiles.d snippets can be defined to cleanup on a timer _anything_,
> 
> It's a question of what the *default* behaviour should be.
> 
> For whatever reason, a lot of people who process large data use
> /var/tmp/FOO/ as a place to store information that should not be
> backed up, but also should not just disappear.
To be honest I'm greatly surprised by this idea, and by the suggestion
that a lot of people do this, to me this is very similar by that half-joke
about people storing useful files in the Recycle Bin.
But I also thought that "/tmp is on tmpfs, /var/tmp isn't, and that's
their main difference" are our defaults for years, when they apparently
aren't (now I understand why people sometimes suggest using /dev/shm or
/run for this or that to increase performance).

-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 16:51, Barak A. Pearlmutter  wrote:
>
> > tmpfiles.d snippets can be defined to cleanup on a timer _anything_,
>
> It's a question of what the *default* behaviour should be.

No, it is not, at least not for the strawman you conjured. So I gather
that git doesn't warn when cloning on other directories that might be
cleaned up - great, that's a non-issue too.
If you want it to do so, then the appropriate way is for you to file
an RFE upstream and ask them to parse the installed tmpfiles.d on a
system before cloning. This is not that hard to do, and the format is
stable, and if they ask about that I'll happily back you up on this
stability guarantee.
But what git does or does not allow is then completely unrelated to
anything happening here.

> For whatever reason, a lot of people who process large data use /var/tmp/FOO/ 
> as a place to store information that should not be backed up, but also should 
> not just disappear.

Then such people, assuming they actually exist, can configure their
custom systems accordingly upon reading the release notes before
upgrading, as they would do anyway if installing on CentOS or any
other major OS. Hence, not an issue either.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

> tmpfiles.d snippets can be defined to cleanup on a timer _anything_,

It's a question of what the *default* behaviour should be.

For whatever reason, a lot of people who process large data use
/var/tmp/FOO/ as a place to store information that should not be
backed up, but also should not just disappear.

Maybe they should use ~/FOO/ but put a CACHEDIR.TAG file with the
right magic contents there. Or touch ~/FOO/.nobackup and use duplicity
--exclude-if-present .nobackup. Or ~/.cache/FOO/, although that's a
bit counterintuitive because it's "hidden" and not really meant for
this purpose.

There's a reason people don't want these files below ~/, which is that
some backup mechanisms, like btrfs snapshots or LVMs snapshots, may
snapshot files there regardless; and often /home/ is on more
stable/expensive storage (raid k for k≥1) while /var/tmp/ is on a
cheaper substrate.

Given the era of large data we're living in, this is a ubiquitous
problem. Since we don't give users a standard solution, they've rolled
their own. If we think that solution is wrong, we should (a) make an
actual official suggested location for this sort of material,
/var/scratch/ or /var/scratch/${USER}/ with a link there from
~/Scratch or something, and (b) transition them to it, ideally without
unexpectedly deleting their files and messing up their work in the
process.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 16:42, Simon Richter  wrote:
>
> Hi,
>
> On 5/6/24 19:57, Michael Biebl wrote:
>
> > Afaik, /var/tmp has never been cleaned up on /boot.
> > So I'm not sure what you mean with "no longer"?
>
> Oof, you're right, it was /tmp, /var/run, /var/lock:
>
>  [ "$VERBOSE" != no ] && echo -n "Cleaning"
>  [ -d /tmp ] && cleantmp
>  [ -d /var/run ] && cleanrun
>  [ -d /var/lock ] && cleanlock
>  [ "$VERBOSE" != no ] && echo "."
>
> Would it make sense to make it a bug for a package to use /var/tmp (on
> my system, I can see files from audacity and reportbug there) and
> declare that this directory is for the use of the sysadmin only?

In general it should be the opposite, it's fine for packages to use
/tmp or /var/tmp (using appropriate measures such as random file/dir
names to avoid clashes and attacks), as long as it's really for
temporary stuff, as in, if they are lost it's not a problem and they
are recreated. If you are storing your precious irreplaceable state on
/tmp or /var/tmp, then you are doing it wrong (TM).
In general it's users who should be careful and know what they are
doing, especially due to the world-writable nature of both
directories.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 16:30, Simon Richter  wrote:
>
> Hi,
>
> On 5/6/24 20:19, Luca Boccassi wrote:
>
> > Is that the default layout, or a selectable option?
>
> When you create a partition manually, it asks for the mount point, and
> makes a number of suggestions in a dropdown, and /tmp is one of these.
>
> There is also a "enter manually" option.
>
> > If the latter, to me it makes sense to keep it, maybe with updated
> > wording. Making things customizable is not really an issue, as long as
> > it's clear what happens where.
>
> The only wording there is "/tmp - Temporary files" or something like
> that (has been a while since I saw that particular menu) -- it's an
> option in a menu, so space is limited.
>
> FWIW, I don't see much of a use case for leaving /tmp on the root
> partition, so if no /tmp mount point is configured, mounting a tmpfs is
> a good choice. Basically anything that can boot the installer has enough
> RAM.

Found it:

https://salsa.debian.org/installer-team/partman-basicfilesystems/-/blob/master/debian/partman-basicfilesystems.templates?ref_type=heads#L97

So  unless someone has a better suggestion, I can change it from:

"/tmp - temporary files"

To something like:

"/tmp - temporary files (default: tmpfs)"

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 15:42, Richard Lewis
 wrote:
>
> Luca Boccassi  writes:
>
> > Hence, I am not really looking for philosophical discussions or lists
> > of personal preferences or hypotheticals, but for facts: what would
> > break where, and how to fix it?
>
> cleaning /tmp or /var/tmp: users may lose files if they dont realise a
>   directory tmp can be cleaned without a reboot. something in /var/tmp
>   that's been in there for 35 days before an upgrade might be deleted
>   before the user reads the NEWS.Debian email, meaning they have no
>   chance to react). Maybe you could postpone the very first deletion
>   until after the next reboot?
>
> using a tmpfs: is there a risk of losing unrelated data due to more
>   frequent OOM killing random other programmes due to /tmp using all the
>   memory?  is there a case to only use a tmpfs if the system has
>   "enough" memory?

Again, those are all hypotheticals, and one can construct similarly
contrived thought exercises for most possible permutations of most
configurations, and the answer is always the same: customize the
configuration accordingly. Hence, not relevant right now.
What is relevant is: which packages, if any, or which DSA-owned
systems, if any, are actually affected and how?

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon Richter]

Hi,

On 5/6/24 19:57, Michael Biebl wrote:


Afaik, /var/tmp has never been cleaned up on /boot.
So I'm not sure what you mean with "no longer"?


Oof, you're right, it was /tmp, /var/run, /var/lock:

[ "$VERBOSE" != no ] && echo -n "Cleaning"
[ -d /tmp ] && cleantmp
[ -d /var/run ] && cleanrun
[ -d /var/lock ] && cleanlock
[ "$VERBOSE" != no ] && echo "."

Would it make sense to make it a bug for a package to use /var/tmp (on 
my system, I can see files from audacity and reportbug there) and 
declare that this directory is for the use of the sysadmin only?


   Simon

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon Richter]

Hi,

On 5/6/24 20:19, Luca Boccassi wrote:


Is that the default layout, or a selectable option?


When you create a partition manually, it asks for the mount point, and 
makes a number of suggestions in a dropdown, and /tmp is one of these.


There is also a "enter manually" option.


If the latter, to me it makes sense to keep it, maybe with updated
wording. Making things customizable is not really an issue, as long as
it's clear what happens where.


The only wording there is "/tmp - Temporary files" or something like 
that (has been a while since I saw that particular menu) -- it's an 
option in a menu, so space is limited.


FWIW, I don't see much of a use case for leaving /tmp on the root 
partition, so if no /tmp mount point is configured, mounting a tmpfs is 
a good choice. Basically anything that can boot the installer has enough 
RAM.


   Simon

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 16:03, Barak A. Pearlmutter  wrote:
>
> If it clones into /tmp the *entire* tree will either be reaped (upon
> reboot) or not.
>
> But having just some files deleted from a git dir or git working dir
> is much more dangerous, because various git commands can treat files
> deleted from the working tree as deliberate changes to be committed,
> and files deleted from the git dir can render it unusable.

It's not really that different, it's a variation of the same problem.
I take it that git doesn't warn when cloning on tmpfs, then, so that's
not an issue.
tmpfiles.d snippets can be defined to cleanup on a timer _anything_,
and have done so for ~15 years. Does git currently parse all installed
tmpfiles.d snippets and warn if cloning on an arbitrary directory
covered by one of them?

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Andrey Rakhmatullin]

On Mon, May 06, 2024 at 07:42:11AM -0700, Russ Allbery wrote:
> >> I'm not sure if we have software on long running servers which place
> >> files in /tmp and /var/tmp and expect files to not be deleted during
> >> runtime, even if not accessed for a long time. This is certainly an
> >> issue to be aware of and keep an eye on.
> 
> > Note that FHS mandates it for /var/tmp: "Files and directories located
> > in /var/tmp must not be deleted when the system is booted. Although data
> > stored in /var/tmp is typically deleted in a site-specific manner, it is
> > recommended that deletions occur at a less frequent interval than /tmp."
> 
> It mandates that it not be cleaned on *boot*.  Not that it never be
> cleaned during runtime.  
Ah, I've read that as "while the system is booted".


-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

If it clones into /tmp the *entire* tree will either be reaped (upon
reboot) or not.

But having just some files deleted from a git dir or git working dir
is much more dangerous, because various git commands can treat files
deleted from the working tree as deliberate changes to be committed,
and files deleted from the git dir can render it unusable.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Russ Allbery]

Andrey Rakhmatullin  writes:
> On Mon, May 06, 2024 at 10:40:00AM +0200, Michael Biebl wrote:

>> I'm not sure if we have software on long running servers which place
>> files in /tmp and /var/tmp and expect files to not be deleted during
>> runtime, even if not accessed for a long time. This is certainly an
>> issue to be aware of and keep an eye on.

> Note that FHS mandates it for /var/tmp: "Files and directories located
> in /var/tmp must not be deleted when the system is booted. Although data
> stored in /var/tmp is typically deleted in a site-specific manner, it is
> recommended that deletions occur at a less frequent interval than /tmp."

It mandates that it not be cleaned on *boot*.  Not that it never be
cleaned during runtime.  It anticipates that it be cleaned periodically,
just less frequently than /tmp.

There is a specific prohibition against clearing /var/tmp on reboot
because /var/tmp historically has been used to store temporary files whose
whole reason for existence is that they need to survive a reboot, such as
vi recover files, but are still safe to delete periodically.

Historically, deleting anything in /var/tmp that hadn't been accessed in
over seven days was a perfectly reasonable and typical configuration.
These days, we have the complication that it's fairly common to turn off
atime updates for performance reasons, which makes it a bit harder to
implement that policy when /var/tmp isn't its own partition and thus
inherits that setting from the rest of the system.

-- 
Russ Allbery (r...@debian.org)  

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 15:31, Barak A. Pearlmutter  wrote:
>
> > What I am looking for right now is packages or internal
> > infrastructure that need
> > an update to cope with these two changes before I upload them, so if
> > you know of any please do let me know and I'll happily look into it
> > and at least file a bug, if not a MR. Thanks.
>
> Okay.
>
> git and other version control systems should be modified to warn if
> the work tree or git dir are in /var/tmp and therefore exposed to
> unexpected deletion of files, and possibly modified to refuse to clone
> there without an explicit --force-unstable-storage option.

Does it do something similar when cloning into /tmp/ or any other
tmpfs right now?

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

> What I am looking for right now is packages or internal
> infrastructure that need
> an update to cope with these two changes before I upload them, so if
> you know of any please do let me know and I'll happily look into it
> and at least file a bug, if not a MR. Thanks.

Okay.

git and other version control systems should be modified to warn if
the work tree or git dir are in /var/tmp and therefore exposed to
unexpected deletion of files, and possibly modified to refuse to clone
there without an explicit --force-unstable-storage option.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon McVittie]

On Mon, 06 May 2024 at 13:41:58 +0100, Barak A. Pearlmutter wrote:
> As someone who regularly deals with large datasets, and keeps them in
> the "approved" don't-back-these-up location /var/tmp

Independent of whether we make the change Luca is suggesting or not,
I don't think /var/tmp is a good recommendation for users storing
temporary data manually, because safe use of "obvious" filenames in a
world-writeable location like /var/tmp requires you to be aware of the
possibility of symlink attacks, and either ensure that your kernel has a
mitigation for them (as Debian's does with default runtime configuration),
or be careful to avoid them, or be OK with any human user account and
any system uid on the system being able to achieve at least denial of
service and maybe privilege escalation to your account.

/tmp and /var/tmp are fine places for *programs* to store temporary data -
for example Flatpak uses a subdirectory in /var/tmp for half-downloaded
apps and runtimes - but only if they're using mkstemp(), mkdtemp()
or equivalent to generate an unpredictable location and allocate it
without race conditions, which is the only way these directories are
safe to use in portable software. This is not particularly user-friendly
to do manually.

For files you don't want backed up, I'd personally recommend storing
them somewhere in your home directory, perhaps ~/tmp, with a CACHEDIR.TAG
(http://justsolve.archiveteam.org/wiki/Cache_Directory_Tagging_Standard),
and using backup software that understands that convention (for example
borg, restic, tar).

smcv

Re: Bug#966621: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Richard Lewis]

Luca Boccassi  writes:

> Hence, I am not really looking for philosophical discussions or lists
> of personal preferences or hypotheticals, but for facts: what would
> break where, and how to fix it?

cleaning /tmp or /var/tmp: users may lose files if they dont realise a
  directory tmp can be cleaned without a reboot. something in /var/tmp
  that's been in there for 35 days before an upgrade might be deleted
  before the user reads the NEWS.Debian email, meaning they have no
  chance to react). Maybe you could postpone the very first deletion
  until after the next reboot?

using a tmpfs: is there a risk of losing unrelated data due to more
  frequent OOM killing random other programmes due to /tmp using all the
  memory?  is there a case to only use a tmpfs if the system has
  "enough" memory?

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 13:42, Barak A. Pearlmutter  wrote:
>
> > Then upon reading the release notes, on such a machine, one can simply do:
> >
> > touch /etc/tmpfiles.d/tmp.conf
> >
> > And they get no automated cleanups.
>
> This also disables on-boot cleaning of /tmp/.

Yes, as it's going to be a tmpfs, so it is no longer needed. Trivial
to maintain though if one wants to do so, though.

> The root issue here is that deleting not-read-in-a-while
> but-maybe-stat'ed-recently-by-make-that-doesn't-count files from
> /var/tmp/ by default, particularly when the system didn't used to,
> violates the principle of least surprise.

Which is what release notes are for, if everything was always the same
we wouldn't spend time to put those together.

> There's an old debugging story

While personal anecdotes and stories can be interesting and amusing in
many circumstances, I am not really looking for those at this very
moment. What I am looking for right now is packages or internal
infrastructure that need
an update to cope with these two changes before I upload them, so if
you know of any please do let me know and I'll happily look into it
and at least file a bug, if not a MR. Thanks.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

> Then upon reading the release notes, on such a machine, one can simply do:
>
> touch /etc/tmpfiles.d/tmp.conf
>
> And they get no automated cleanups.

This also disables on-boot cleaning of /tmp/.

The root issue here is that deleting not-read-in-a-while
but-maybe-stat'ed-recently-by-make-that-doesn't-count files from
/var/tmp/ by default, particularly when the system didn't used to,
violates the principle of least surprise. It will cause problems for
end users. And disabling it *properly* requires serious sysadmin
knowledge and is easy to get wrong.

There's an old debugging story: Maclisp on a DEC-10 was glitchy some
days and not others. Someone marked the crashes on a calendar, and
they correlated with the phase of the moon! Turns out the program
printed the lunar phase ("crescent", "waning gibbous", etc) in its
header, just for fun, and when the string was long it wrote past the
end of a buffer, etc.

The story is still told fifty years later because it's so funny, it's
a total violation of levels. People debugging in a windowless basement
should not need to know the phase of the moon!

Users want their systems to run reliably and reproducibly. Getting
different behaviour depending on how long it's been since some of the
files you're using have been *accessed* adds oodles of extra state
people have to keep track of in their mental model of what some
program needs to run. Sure, it can be administratively disabled on one
particular system; but then you run the same program or perform the
same work-flow on another system, and it fails. Something fails
reliably after running for over 30 days ... unless you keep an eye on
it.

As someone who regularly deals with large datasets, and keeps them in
the "approved" don't-back-these-up location /var/tmp/, this just seems
crazy to me. When I tell a grad student to install Debian on their
laptop, do I really need to spend time explaining to them to be sure
to turn off the "delete your old files for no reason" option which is
enabled by default? When they're using someone else's cluster, should
I remind them to run a cron job "find /var/tmp/foo -exec touch ..." so
the 25 Tb of data we pushed onto scratch storage on that cluster
doesn't partly disappear? Our priority should be our users. Our users
are not well served by having their files deleted. They put them there
for a reason!

--Barak.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Marvin Renich]

* Michael Biebl  [240506 07:15]:
> Am 06.05.24 um 12:35 schrieb Simon Richter:
> > Hi,
> > 
> > On 5/6/24 17:40, Michael Biebl wrote:
> > 
> > > If we go with a/, then I think d-i should be updated to no longer
> > > create /tmp as a separate partition.
> > 
> > I think if the admin explicitly configures tmpfs as a separate file
> > system, then that should be honored -- if there is memory pressure,
> > tmpfs+swap is pretty much the worst file system there is.
> > 
> 
> If you explicitly configure /tmp via /etc/fstab it will override the default
> tmp.mount that is shipped by systemd.
> 
> My point is a different one: If we enable /tmp-on-tmpfs by default, then
> offering a partition layout in d-i that places /tmp as a physical partition
> is confusing/inconsistent imho.

I don't think so at all, especially if the description of this option
explains that if you don't choose this option /tmp will be a tmpfs.  But
either way, this is an option that is different from the default, and it
doesn't seem either confusing or inconsistent to me.

...Marvin

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 12:15, Michael Biebl  wrote:
>
> Am 06.05.24 um 12:35 schrieb Simon Richter:
> > Hi,
> >
> > On 5/6/24 17:40, Michael Biebl wrote:
> >
> >> If we go with a/, then I think d-i should be updated to no longer
> >> create /tmp as a separate partition.
> >
> > I think if the admin explicitly configures tmpfs as a separate file
> > system, then that should be honored -- if there is memory pressure,
> > tmpfs+swap is pretty much the worst file system there is.
> >
>
> If you explicitly configure /tmp via /etc/fstab it will override the
> default tmp.mount that is shipped by systemd.
>
> My point is a different one: If we enable /tmp-on-tmpfs by default, then
> offering a partition layout in d-i that places /tmp as a physical
> partition is confusing/inconsistent imho.

Is that the default layout, or a selectable option?

If the latter, to me it makes sense to keep it, maybe with updated
wording. Making things customizable is not really an issue, as long as
it's clear what happens where.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Marvin Renich]

* Simon Richter  [240506 06:51]:
> Hi,
> 
> On 5/6/24 17:40, Michael Biebl wrote:
> 
> > If we go with a/, then I think d-i should be updated to no longer create
> > /tmp as a separate partition.
> 
> I think if the admin explicitly configures tmpfs as a separate file system,
> then that should be honored -- if there is memory pressure, tmpfs+swap is
> pretty much the worst file system there is.

I agree.  It has already been said that /tmp on tmpfs is not right for
all situations (I agree it is a good default for most systems), and
since d-i doesn't do this by default, what benefit is there to removing
that option from the installer?  There is clearly benefit to leaving it
in.

...Marvin

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 11:33, Barak A. Pearlmutter  wrote:
>
> > We have two separate issues here:
>
> > a/ /tmp-on-tmpfs
> > b/ time based clean-up of /tmp and /var/tmp
>
> > I think it makes sense to discuss/handle those separately.
>
> Agreed.
>
> I also don't see any issue with a/, at worst people will be annoyed
> with it for some reason and can then change it back.
>
> > Regarding b/: ...
>
> > The tmpfiles rule tmp.conf as shipped by systemd upstream ...
> > Files that are older then 10 days or 30 days are automatically cleaned up.
>
> This seems like a rather dangerous thing to spring on people.
>
> First of all, time can be pretty fluid on user machines.

Then upon reading the release notes, on such a machine, one can simply do:

touch /etc/tmpfiles.d/tmp.conf

And they get no automated cleanups. This stuff is designed to be
trivially overridable, both by end-users and image builders. What I am
looking for is, what packages need bugs/MRs filed to deal with this
change, if any.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 11:48, Michael Biebl  wrote:
>
> Am 06.05.24 um 12:18 schrieb Luca Boccassi:
> > Defaults are defaults, they are trivially and fully overridable where
> > needed if needed. Especially container and VM managers these days can
> > super trivially override them via SMBIOS Type11 strings or
> > Credentials, ephemerally and without changing the guest image at all.
>
>
> Aligning defaults across distros does have value.
> That said, a distro like Debian has a larger scope than say a desktop
> oriented one like Fedora.
> Debian is used on a broad spectrum of systems: from embedded to server
> to cloud to desktop.
> So I think it is valuable to gather feedback from all affected parties
> to make an informed decision.
>
> What upstream is doing should not be the only driving factor.

It's impossible to have defaults that make everyone happy, there will
always be someone who doesn't like any choice one might pick (there
are people unhappy with the current ones too).

Hence, I am not really looking for philosophical discussions or lists
of personal preferences or hypotheticals, but for facts: what would
break where, and how to fix it? I only found autopkgtest so far, which
uses /tmp/ in the guest and expects it to survive across reboots, and
I have a MR up already for that. Anything else?

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Andrey Rakhmatullin]

On Mon, May 06, 2024 at 10:40:00AM +0200, Michael Biebl wrote:
> I'm not sure if we have software on long running servers which place files
> in /tmp and /var/tmp and expect files to not be deleted during runtime, even
> if not accessed for a long time. This is certainly an issue to be aware of
> and keep an eye on.
Note that FHS mandates it for /var/tmp: "Files and directories located in
/var/tmp must not be deleted when the system is booted. Although data
stored in /var/tmp is typically deleted in a site-specific manner, it is
recommended that deletions occur at a less frequent interval than /tmp."




-- 
WBR, wRAR


signature.asc
Description: PGP signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Michael Biebl]

Am 06.05.24 um 12:35 schrieb Simon Richter:

Hi,

On 5/6/24 17:40, Michael Biebl wrote:

If we go with a/, then I think d-i should be updated to no longer 
create /tmp as a separate partition.


I think if the admin explicitly configures tmpfs as a separate file 
system, then that should be honored -- if there is memory pressure, 
tmpfs+swap is pretty much the worst file system there is.




If you explicitly configure /tmp via /etc/fstab it will override the 
default tmp.mount that is shipped by systemd.


My point is a different one: If we enable /tmp-on-tmpfs by default, then 
offering a partition layout in d-i that places /tmp as a physical 
partition is confusing/inconsistent imho.





Regarding b/:
The current setup as used in Debian is to only clean /tmp on boot 
(which is pointless with /tmp-on-tmpfs) and never clean up /var/tmp


There is also the tmpreaper package, which does periodic cleanup based 
on atime. I'd expect that to be installed on a lot of long-running 
machines, it is certainly installed on mine.


Also if /var/tmp is no longer cleaned on boot, that is a bug.


Afaik, /var/tmp has never been cleaned up on /boot.
So I'm not sure what you mean with "no longer"?

Michael


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Michael Biebl]

Am 06.05.24 um 12:18 schrieb Luca Boccassi:

Defaults are defaults, they are trivially and fully overridable where
needed if needed. Especially container and VM managers these days can
super trivially override them via SMBIOS Type11 strings or
Credentials, ephemerally and without changing the guest image at all.



Aligning defaults across distros does have value.
That said, a distro like Debian has a larger scope than say a desktop 
oriented one like Fedora.
Debian is used on a broad spectrum of systems: from embedded to server 
to cloud to desktop.
So I think it is valuable to gather feedback from all affected parties 
to make an informed decision.


What upstream is doing should not be the only driving factor.


Michael



OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Samuel Thibault]

Barak A. Pearlmutter, le lun. 06 mai 2024 11:15:35 +0100, a ecrit:
> To me, the purpose of /var/tmp/ when I have my "user" hat on is: a
> place to put files I don't want backed up, particularly large ones,
> and which if I run out of disk space is a place to look for stuff to
> delete. it's not "a place to put files that the system might
> capriciously delete just because."

Definite +1 on this.

My /var/tmp has various "I want to get rid of this at some point"
temporary stuff, and I would be quite annoyed if it would disappear
unexpectedly.

> One compromise would be to enable the /var/tmp/ reaper by default only
> on new installations.

At the very least, yes.

Samuel

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Michael Biebl]

Am 06.05.24 um 12:15 schrieb Barak A. Pearlmutter:

We have two separate issues here:



a/ /tmp-on-tmpfs
b/ time based clean-up of /tmp and /var/tmp



I think it makes sense to discuss/handle those separately.


Agreed.

I also don't see any issue with a/, at worst people will be annoyed
with it for some reason and can then change it back.


I forgot to mention, that while I was sceptical of /tmp-on-tmpfs in the 
past, I nowadays basically enable it everywhere, even on smaller systems 
like PIs.

So it's a change I support.

I did run into odd issues like autopkgtest filling up /tmp and then 
various programs like Firefox behaving oddly and worked around this by 
using TMPDIR=/var/tmp for stuff that needed larger tmp files.








OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Simon Richter]

Hi,

On 5/6/24 17:40, Michael Biebl wrote:

If we go with a/, then I think d-i should be updated to no longer create 
/tmp as a separate partition.


I think if the admin explicitly configures tmpfs as a separate file 
system, then that should be honored -- if there is memory pressure, 
tmpfs+swap is pretty much the worst file system there is.


For example, I have a VM with 32 threads but only 16 GiB of memory 
allocated to it, which is normally sufficient for compiling large 
projects, but occasionally starts thrashing really hard, and /tmp on 
ext4 with very relaxed ordering rules is a good way to make it behave.


(What we're really want is a file system that behaves like 
FILE_ATTRIBUTE_TEMPORARY on Windows, where backing store is allocated, 
and pages are evicted first on memory pressure, but are not flushed 
automatically)



Regarding b/:
The current setup as used in Debian is to only clean /tmp on boot (which 
is pointless with /tmp-on-tmpfs) and never clean up /var/tmp


There is also the tmpreaper package, which does periodic cleanup based 
on atime. I'd expect that to be installed on a lot of long-running 
machines, it is certainly installed on mine.


Also if /var/tmp is no longer cleaned on boot, that is a bug.

I'm not sure if we have software on long running servers which place 
files in /tmp and /var/tmp and expect files to not be deleted during 
runtime, even if not accessed for a long time. This is certainly an 
issue to be aware of and keep an eye on.


IIRC it was an explicit decision not to install tmpreaper by default and 
leave cleanup policies to the local admin, for precisely this reason.


It makes sense to revisit this decision 20 years later, but a lot of the 
original rationale remains valid: it affects only a handful of machines 
(because everyone else reboots often anyway), and we kind of expect 
those machines to be operated by skilled people.


   Simon

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 09:40, Michael Biebl  wrote:
>
> We have two separate issues here:
>
> a/ /tmp-on-tmpfs
> b/ time based clean-up of /tmp and /var/tmp
>
> I think it makes sense to discuss/handle those separately.
>
> Regarding a/:
> tmp.mount as shipped by systemd uses the following mount options:
> "mode=1777,strictatime,nosuid,nodev,size=50%"
>
> In the past there were concerns that those 50% of available RAM wasn't a
> one-size-fits-all solution, especially for (LXC) containers and VMs
>
> One also needs to keep in mind that debian-installer still offers a
> partitioning setup with /tmp on a separate partition. This will be
> created via an entry in /etc/fstab. Such a /tmp entry in /etc/fstab will
> override tmp.mount.
>
> If we go with a/, then I think d-i should be updated to no longer create
> /tmp as a separate partition.
>
>
> Regarding b/:
> The current setup as used in Debian is to only clean /tmp on boot (which
> is pointless with /tmp-on-tmpfs) and never clean up /var/tmp
>
> The tmpfiles rule tmp.conf as shipped by systemd upstream contains:
>
> q /tmp 1777 root root 10d
> q /var/tmp 1777 root root 30d
>
> Files that are older then 10 days or 30 days are automatically cleaned
> up. The age of the files are determined as such:
>
> "The age of a file system entry is determined from its last modification
> timestamp (mtime), its last access timestamp (atime), and (except for
> directories) its last status change timestamp (ctime). By default, any
> of these three (or two) values will prevent cleanup if it is more recent
> than the current time minus the age field."
>
> I'm not sure if we have software on long running servers which place
> files in /tmp and /var/tmp and expect files to not be deleted during
> runtime, even if not accessed for a long time. This is certainly an
> issue to be aware of and keep an eye on.

Defaults are defaults, they are trivially and fully overridable where
needed if needed. Especially container and VM managers these days can
super trivially override them via SMBIOS Type11 strings or
Credentials, ephemerally and without changing the guest image at all.

Re: Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Barak A. Pearlmutter]

> We have two separate issues here:

> a/ /tmp-on-tmpfs
> b/ time based clean-up of /tmp and /var/tmp

> I think it makes sense to discuss/handle those separately.

Agreed.

I also don't see any issue with a/, at worst people will be annoyed
with it for some reason and can then change it back.

> Regarding b/: ...

> The tmpfiles rule tmp.conf as shipped by systemd upstream ...
> Files that are older then 10 days or 30 days are automatically cleaned up.

This seems like a rather dangerous thing to spring on people.

First of all, time can be pretty fluid on user machines.

Someone has a laptop, they do some work and generate big files in
/var/tmp/, maybe they're editing videos or processing medical images
or downloading and pre-processing large datasets for machine learning.
Now they close their laptop and go on vacation for six weeks. When
they come back they open it and ... shazam all the files they were
working on are deleted.

Another scenario. I download a big source tarball, firefox say. I want
to make a few small changes and compile a custom version. I keep it in
/var/tmp/ for obvious reasons. (The patches are stored elsewhere of
course.) Now I download a new patch and apply it and recompile. And
get an error, because *some* of the files in the unpacked tree were
too old and got auto-deleted.

Imagine suspending a computer in the middle of a big build, and when I
un-suspend it a few weeks later the build fails because some files are
now too old and get deleted. Is this really desirable?

To me, the purpose of /var/tmp/ when I have my "user" hat on is: a
place to put files I don't want backed up, particularly large ones,
and which if I run out of disk space is a place to look for stuff to
delete. it's not "a place to put files that the system might
capriciously delete just because."

I do understand the imperative to keep the filesystem clean and to do
routine maintenance and upkeep without user intervention. But changing
something that used to be "stable storage unless your disk crashes"
into "files placed here can mysteriously disappear" seems like a major
potentially-breaking change that can disrupt common work flows.

I can see popping up a warning: "disk space on the filesystem
containing /var/tmp is getting low and /var/tmp contains xxxGb of
files, hit OKAY to delete the xxxGb of oldest files there" or
something like that. But doing it silently is scary.

One compromise would be to enable the /var/tmp/ reaper by default only
on new installations. People can always turn it on if they want. Also,
when files are deleted, I think the user should get warnings all over
their desktop. I'd also suggest that the reaper by default never
delete files from /var/tmp if there is a lot of space available, or if
/var/tmp isn't consuming much space.

--Barak Pearlmutter

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Luca Boccassi]

On Mon, 6 May 2024 at 06:36, Paul Gevers  wrote:
>
> Hi Luca,
>
> On 05-05-2024 10:04 p.m., Luca Boccassi wrote:
>
>  > Hence, I intend to apply these changes in the next src:systemd upload
>  > to unstable, probably next week.
>
> > In case anybody is aware of packages/programs needing an update to cope
> > with these changes, or any other issue, please let me know and I will
> > file bugs.
>
> You filed MR341 against autopkgtest, thanks for that. Can you please
> hold off a bit longer than only one week to get the infrastructure
> updated? I'm confident that every test that has the needs-reboot
> restriction will start failing (which are only 21 tests according to
> codesearch). However, I fear that every test is at risk under common
> circumstances.
>
> I don't want to be rushed into an autopkgtest update and an
> infrastructure update for something that we can plan (there's always
> risk involved and I want to have the time and peace to cope with the
> process). Having said that, maybe we will be ready next week.

Sure, no problem. That MR fixes autopkgtest for my local runs, let me
know if something else is needed.

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Michael Biebl]

Am 05.05.24 um 22:04 schrieb Luca Boccassi:

This will be mentioned in NEWS (and I guess in the release notes when
the time comes), together with the instructions to override for anybody
wanting to keep the old behaviour, which is as trivial as:



..


touch /etc/tmpfiles.d/tmp.conf


This doesn't restore the old/current behaviour, which is to cleanup /tmp 
on boot. For that you would need something like


echo "D /tmp 1777 root root -" > /etc/tmpfiles.d/tmp.conf


OpenPGP_signature.asc
Description: OpenPGP digital signature

Re: Make /tmp/ a tmpfs and cleanup /var/tmp/ on a timer by default [was: Re: systemd: tmpfiles.d not cleaning /var/tmp by default]

[Holger Levsen]

clone 966621 -1
reassign -1 release-notes
thanks

On Mon, May 06, 2024 at 10:40:00AM +0200, Michael Biebl wrote:
> We have two separate issues here:
> 
> a/ /tmp-on-tmpfs
> b/ time based clean-up of /tmp and /var/tmp
> 
> I think it makes sense to discuss/handle those separately.

very much agreed. a.) is mostly "just" a /tmp size issue but b.) can introduce
interesting unforseen breakages for long running stuff.



-- 
cheers,
Holger

 ⢀⣴⠾⠻⢶⣦⠀
 ⣾⠁⢠⠒⠀⣿⡁  holger@(debian|reproducible-builds|layer-acht).org
 ⢿⡄⠘⠷⠚⠋⠀  OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
 ⠈⠳⣄

“I'll tell you what freedom is to me No fear.” (Nina Simone)


signature.asc
Description: PGP signature