Re: Moving /tmp to tmpfs makes it useful
2012/6/1 Goswin von Brederlow wrote: All the complaints about /tmp as tmpfs come down to one simple issue: The size of the tmpfs isn't chosen well. Mounting /tmp to tmpfs not just breaks a lot of apps and reduces system stability, but it actually does nothing else. You get no benefits from /tmp being on tmpfs. That's the complaint: the change makes something bad and nothing good. Even without load it is much faster because fsync() becomes a NOP. Yes, it is. So it's a good idea to use tmpfs for some apps, that heavily use fsync() on files that fit in RAM. But... wait... no app is heavily using fsync() on files in /tmp. So it's useless to put /tmp on tmpfs. It also saves on wear of the disk if the files are small and short lived, like temp files for gcc. I guess you talk about SSD-like disks. Then... 1. short-lived files remain in cache and don't hit the disk. 2. gcc does not use fsync, does it? 3. gcc with -pipe option does not use /tmp 4. if you manually build something, you can add -pipe option to CFLAGS 5. if you build a package, you can add it to global build options 6. this doesn't matter, since pretty much nobody builds apps on SSD disks So this is not even a corner case. :) - There's not enough space, so the system starts swapping, including some apps. Which happens regardless wether tmp is tmpfs or a real filesystem. The more IO there is the more likely some app gets swapped out. No. I checked that. If you have e.g. 50% of free ram then whatever files you read/write other apps won't get swapped, zero swap usage. But if you start writing large files to tmpfs you get swapping. With tmps that instantly frees up all the memory and swap used by the file. With a real FS the file data remains in the dirty cache until such a time as the disk has cought up with writing it all and then it is thrown away. So potentially memory is freed up much later. Actually cache also frees all the memory instantly on delete, but may delay writing to disk. So only a few kilobytes of metadata are not freed. Yes because writing that on disk will only block the thread performing the write, not every thread that tries to allocate memory. Wrong. Not that much. The thread doing the write will just write to cache and not block at all. That is until you run out of cache. Until you run out of dirty_ratio, which is usually less than entire cache. :) No, tmpfs will be swapped out if you don't use a file for a while but something else uses memory, including IO caching. It won't, I tried. I put a few hundreds MB on tmpfs and started reading and copying gigabytes of files, tar/untar archives. I was still getting no swap usage. As long as there's enough RAM they don't get swapped. I'm asking for *popular* apps, that create files in /tmp, *never put large files* there, and become *noticeably* faster with /tmp on tmpfs? gcc, ocamlopt, mc, lintian Which of them becomes faster with /tmp on tmpfs? Can you suggest a use case, that I can test with /tmp on disk and on tmpfs myself and see them becoming faster? -- Serge -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/caovenepfrzmzzppsux8fib9g0ce-gnoum-mdkzn2nzkyxfw...@mail.gmail.com
Re: Moving /tmp to tmpfs makes it useful
On Tue, Jun 05, 2012 at 09:29:46AM +0300, Serge wrote: 2012/6/1 Goswin von Brederlow wrote: All the complaints about /tmp as tmpfs come down to one simple issue: The size of the tmpfs isn't chosen well. Mounting /tmp to tmpfs not just breaks a lot of apps and reduces system stability, but it actually does nothing else. You get no benefits from /tmp being on tmpfs. That's the complaint: the change makes something bad and nothing good. Even without load it is much faster because fsync() becomes a NOP. Yes, it is. So it's a good idea to use tmpfs for some apps, that heavily use fsync() on files that fit in RAM. But... wait... no app is heavily using fsync() on files in /tmp. So it's useless to put /tmp on tmpfs. It takes one application using fsync on any file in / for all files in /tmp to be flushed to disk if it's the same filesystem. It doesn't need to be the application using /tmp doing fsync. Mike -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120605064244.ga15...@glandium.org
Re: Moving /tmp to tmpfs makes it useful
Mike Hommey m...@glandium.org writes: On Tue, Jun 05, 2012 at 09:29:46AM +0300, Serge wrote: 2012/6/1 Goswin von Brederlow wrote: All the complaints about /tmp as tmpfs come down to one simple issue: The size of the tmpfs isn't chosen well. Mounting /tmp to tmpfs not just breaks a lot of apps and reduces system stability, but it actually does nothing else. You get no benefits from /tmp being on tmpfs. That's the complaint: the change makes something bad and nothing good. Even without load it is much faster because fsync() becomes a NOP. Yes, it is. So it's a good idea to use tmpfs for some apps, that heavily use fsync() on files that fit in RAM. But... wait... no app is heavily using fsync() on files in /tmp. So it's useless to put /tmp on tmpfs. It takes one application using fsync on any file in / for all files in /tmp to be flushed to disk if it's the same filesystem. It doesn't need to be the application using /tmp doing fsync. Mike For example runing apt-get upgrade while doing other stuff that uses /tmp. MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87y5o2chhx.fsf@frosties.localnet
Re: Moving /tmp to tmpfs makes it useful
Serge sergem...@gmail.com writes: 2012/6/1 Goswin von Brederlow wrote: All the complaints about /tmp as tmpfs come down to one simple issue: The size of the tmpfs isn't chosen well. Mounting /tmp to tmpfs not just breaks a lot of apps and reduces system stability, but it actually does nothing else. You get no benefits from /tmp being on tmpfs. That's the complaint: the change makes something bad and nothing good. You keep claiming tmpfs compromises system stability. Can you show a kernel oops of a crash caused by tmpfs? It also saves on wear of the disk if the files are small and short lived, like temp files for gcc. I guess you talk about SSD-like disks. Then... 1. short-lived files remain in cache and don't hit the disk. A short-livd file goes to cache, as dirty pages. dirty pages are then written to disk in the background. Are you claiming that filesystem do un-dirty pages when a file is deleted before it was actually written to disk? Do they remove the pages from the write queue when they are already picked up for write out? 2. gcc does not use fsync, does it? As Mike says any fsync on the same filesystem generaly will force write the files in /tmp too. 3. gcc with -pipe option does not use /tmp 4. if you manually build something, you can add -pipe option to CFLAGS 5. if you build a package, you can add it to global build options Not the default and you were talking about inexperienced users. 6. this doesn't matter, since pretty much nobody builds apps on SSD disks So this is not even a corner case. :) I think more than nobody has an SSD disk nowadays and they will only become more popular. At best ignoring SSDs is short sighted. Yes because writing that on disk will only block the thread performing the write, not every thread that tries to allocate memory. Wrong. Not that much. The thread doing the write will just write to cache and not block at all. That is until you run out of cache. Until you run out of dirty_ratio, which is usually less than entire cache. :) Unfortunately that limit seems to simply not work AT ALL. Writing to a slow filesystem like NFS or a USB stick just keeps writing and writing and writing till there is no more ram. With the obvious result of stuff blocking. Also shouldn't/couldn't tmpfs fall under the same ratio (or equivalent setting). No more than dirty_ratio pages should be dirty in tmpfs and then it could start swapping them out even without memory pressure? No, tmpfs will be swapped out if you don't use a file for a while but something else uses memory, including IO caching. It won't, I tried. I put a few hundreds MB on tmpfs and started reading and copying gigabytes of files, tar/untar archives. I was still getting no swap usage. As long as there's enough RAM they don't get swapped. Well, worked here. I know swap usage/behaviour widely varries between kernel versions. I've seen kernels that didn't use swap even after a week but also kernels that used swap after an hour all with the pretty much the same usage pattern on my desktop. I'm asking for *popular* apps, that create files in /tmp, *never put large files* there, and become *noticeably* faster with /tmp on tmpfs? gcc, ocamlopt, mc, lintian Which of them becomes faster with /tmp on tmpfs? Can you suggest a use case, that I can test with /tmp on disk and on tmpfs myself and see them becoming faster? All of them. In mc use the feature to look into tar/rpm/deb files. And try running apt-get upgrade in parallel for extra fsync()s. MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87txyqcgp8.fsf@frosties.localnet
Re: Moving /tmp to tmpfs makes it useful
2012/6/5 Goswin von Brederlow wrote: You keep claiming tmpfs compromises system stability. Can you show a kernel oops of a crash caused by tmpfs? No crashes, system just becomes slow and hard/impossible to use. See the Stefan Lippers-Hollmann email about tmpfs vs disk writes. System that does not respond to mouse clicks I called less stable. I don't mind if you suggest a better name for it. A short-livd file goes to cache, as dirty pages. dirty pages are then written to disk in the background. Are you claiming that filesystem do un-dirty pages when a file is deleted before it was actually written to disk? Yes, I tested that. It's easy to test, run: dd if=/dev/zero of=testfile bs=1M count=20; rm -f testfile then check /proc/diskstats or `iostat -k` for actually written bytes. As Mike says any fsync on the same filesystem generaly will force write the files in /tmp too. It does not (Mike, are you sure?). I wrote a small test to check fsync speed. Certainly fsync becomes slower on a heavily loaded disk. But it becomes about same slower when I READ from disk instead of writing to it. Fsync also does not cause the data to be flushed (there was about 50MB of data in cache, but fsync usually took 0.1s, my disk is not that fast). Can you actually reproduce that on your system? 5. if you build a package, you can add it to global build options Not the default and you were talking about inexperienced users. If the user is experienced enough to build a package, he can write: APPEND CFLAGS -pipe APPEND CXXFLAGS -pipe to the /etc/dpkg/buildflags.conf as well. :) (maybe it should be there by default, btw?) I think more than nobody has an SSD disk nowadays and they will only become more popular. At best ignoring SSDs is short sighted. Ehm... I don't understand you. There're a lot of blind people there, and ignoring them is also not the best idea. But how is that related to /tmp on tmpfs? It does almost no help for SSD disks, because most writes are not done to /tmp. I don't know any people, who care about disk writes AND build packages on SSD disks. But I think you're right, that they will become more and more popular. At the same time they become more reliable, so if we're talking about future — we don't have to worry about SSD at all, because even modern SSD disks are supposed to live for 50+ years. :) [1] Unfortunately that limit seems to simply not work AT ALL. Writing to a slow filesystem like NFS or a USB stick just keeps writing and writing and writing till there is no more ram. With the obvious result of stuff blocking. I don't have NFS to check, but I often write to USB disks, that are much slower than my HDD, and get no swapping/blocking. Also shouldn't/couldn't tmpfs fall under the same ratio (or equivalent setting). No more than dirty_ratio pages should be dirty in tmpfs and then it could start swapping them out even without memory pressure? IMHO, tmpfs is a filesystem in RAM, not a cache for swap filesystem, so dirty pages should be flushed to RAM, not to swap. It won't, I tried. As long as there's enough RAM they don't get swapped. Well, worked here. Maybe you have some unusual sysctl settings, e.g. vm.swappiness? Which of them becomes faster with /tmp on tmpfs? Can you suggest a use case, that I can test with /tmp on disk and on tmpfs myself and see them becoming faster? All of them. In mc use the feature to look into tar/rpm/deb files. And try running apt-get upgrade in parallel for extra fsync()s. I can't reproduce the fsync() problem, see above. Can you? (also, mc does not unpack rpm/deb to /tmp when looking into it :)) [1] http://www.storagesearch.com/ssdmyths-endurance.html -- Serge -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/caoveneo8cqm9p3cksqi6xpg05cczpgt-qzmb56+kmh761xg...@mail.gmail.com
Re: Moving /tmp to tmpfs makes it useful
Serge sergem...@gmail.com writes: I'm asking for *popular* apps, that create files in /tmp, *never put large files* there, and become *noticeably* faster with /tmp on tmpfs? gcc, ocamlopt, mc, lintian MfG Goswin -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87hauvkzl0.fsf@frosties.localnet
Re: Moving /tmp to tmpfs makes it useful
On Thu, May 31, 2012 at 12:46:49AM +0300, Uoti Urpala wrote: Also, nowadays normal filesystems are journaled; using a journal for writes to /tmp damages the SSD for zero benefit. Writing to /tmp will damage a SSD? Are you serious? And writing to /var or /home will not? If SSDs are so easy damageable, that you should never use them for e.g. development because it creates to much write cycles, then they have a serious problem. (Thinking about the firmware bugs with data loss I will stay away from them anyway.) I think no one is saying that tmpfs is always bad and never should be used. But for a default installation /tmp belongs to a disk which will be far bigger than the memory. If a user thinks tmpfs will get him an advantage in his setup, then he can switch. Shade and sweet water! Stephan -- | Stephan Seitz E-Mail: s...@fsing.rootsland.net | | Public Keys: http://fsing.rootsland.net/~stse/keys.html | smime.p7s Description: S/MIME cryptographic signature
Re: Moving /tmp to tmpfs makes it useful
On 2012-05-25 14:49:14 +0100, Will Daniels wrote: On 25/05/12 13:52, Ted Ts'o wrote: So what? If you write to a normal file system, it goes into the page cache, which is pretty much the same as writing into tmpfs. In both cases if you have swap configured, the data will get pushed to disk; That's not at all the same, the page cache is more temporary, it's getting flushed to disk pretty quick if memory is tight (presumably) but in the same situation using tmpfs going to swap is surely going to be more disruptive? Why wouldn't there be a FS option to have a different policy for specific directories, like /tmp? e.g. avoid write-back to disk except when needed (because of lack of RAM). In such a way, the behavior would be similar to tmpfs, without the restriction on the tmpfs size, i.e. this would be more flexible for the user. BTW, I wish there were a document summarizing all these discussions. -- Vincent Lefèvre vinc...@vinc17.net - Web: http://www.vinc17.net/ 100% accessible validated (X)HTML - Blog: http://www.vinc17.net/blog/ Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon) -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120530113010.ga24...@xvii.vinc17.org
Re: Moving /tmp to tmpfs makes it useful
2012/5/30 Vincent Lefevre wrote: So what? If you write to a normal file system, it goes into the page cache, which is pretty much the same as writing into tmpfs. In both cases if you have swap configured, the data will get pushed to disk; That's not at all the same, the page cache is more temporary, it's getting flushed to disk pretty quick if memory is tight (presumably) but in the same situation using tmpfs going to swap is surely going to be more disruptive? Why wouldn't there be a FS option to have a different policy for specific directories, like /tmp? e.g. avoid write-back to disk except when needed (because of lack of RAM). In such a way, the behavior would be similar to tmpfs, without the restriction on the tmpfs size, i.e. this would be more flexible for the user. Well, even being better than tmpfs, this idea still has downsides. Kernel fs cache is smart. It stores the files (not exactly files, but who cares) that are the most useful. When you create tmpfs, or when you set such FS option (keep /tmp files in RAM as long as there's free RAM for them) you eat cache memory. Meaning, you store /tmp files in cache even when they're not used, so kernel cannot use that memory to store some useful files. This increases I/O and makes the system slower. But, anyway, directory-specific options are not impossible. Theoretically one could write a special [FUSE] filesystem just for that case. I was already thinking about doing something like aufs- or unionfs-mount of a small tmpfs (i.e. 10MB) backed up by real disk /tmp. That way small files could remain in RAM while large files that don't fit there would go to real cached disk. I was even thinking about patching unionfs-fuse to support such a case. I just could not find a reason to start doing that. I mean, why would people want that feature? The only case I can think of: people with notebooks having SSD-disks, who want to reduce number of disk writes. And they usually want to do that for the whole disk, not just /tmp. There're a lot of ways to do that (starting from tuning kernel vm/dirty_*_centisecs options, up to a paranoid setup that does no writes at all). And since most writes are not limited to /tmp it's pointless to optimize just a /tmp for it. BTW, I wish there were a document summarizing all these discussions. I'm going to write some summary. Just waiting to hear all the opinions. -- Serge -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/caoveneonuk7xrezmcjr--xfqexyo_tlo1y3wrpqpf737q0w...@mail.gmail.com
Re: Moving /tmp to tmpfs makes it useful
Serge wrote: you eat cache memory. Meaning, you store /tmp files in cache even when they're not used, so kernel cannot use that memory to store some useful files. This increases I/O and makes the system slower. The tmpfs files will be written to swap if they aren't accessed much and the kernel wants to cache other files. I mean, why would people want that feature? The only case I can think of: people with notebooks having SSD-disks, who want to reduce number of disk writes. And they usually want to do that for the whole disk, not just /tmp. There're a lot of ways to do that (starting from tuning kernel The reason normal filesystems write data to disk relatively soon is not that it would be good for performance, but to avoid losing data in a crash. Even if you're willing to accept a somewhat higher risk of data loss on such a notebook, you'd very rarely want to use settings appropriate for /tmp where it's OK to lose any or all writes done since the machine was booted up. Thus your do that for the whole disk comparison is wrong. Also, nowadays normal filesystems are journaled; using a journal for writes to /tmp damages the SSD for zero benefit. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1338414409.21597.13.camel@glyph.nonexistent.invalid
Re: Moving /tmp to tmpfs makes it useful
2012/5/31 Uoti Urpala wrote: you eat cache memory. Meaning, you store /tmp files in cache even when they're not used, so kernel cannot use that memory to store some useful files. This increases I/O and makes the system slower. The tmpfs files will be written to swap if they aren't accessed much and the kernel wants to cache other files. They won't. I tried reading/copying gigabytes of files, tar/untar archives... As long as there're enough memory they don't get swapped. I mean, why would people want that feature? The only case I can think of: people with notebooks having SSD-disks, who want to reduce number of disk writes. And they usually want to do that for the whole disk, not just /tmp. There're a lot of ways to do that (starting from tuning kernel The reason normal filesystems write data to disk relatively soon is not that it would be good for performance, but to avoid losing data in a crash. Or rather to lose less data in a crash. But you are free to modify kernel settings and change the timeouts, that's what kernel settings are for. ;) Even if you're willing to accept a somewhat higher risk of data loss on such a notebook, you'd very rarely want to use settings appropriate for /tmp where it's OK to lose any or all writes done since the machine was booted up. Thus your do that for the whole disk comparison is wrong. The comparison is not about how much data to lose, but about how to reduce writes amount. What's the point in mounting /tmp to tmpfs if it breaks 20% of the apps you usually need, but saves you 1% of disk writes? Why not do something better instead? Also, nowadays normal filesystems are journaled; using a journal for writes to /tmp damages the SSD for zero benefit. Yeah, that's a popular opinion. But how true is it? I mean, how much does the journal increases amount of writes? I don't know, never tested that. Can you tell how many writes are added by journal in your case? Is it 20%, or 10%, or 1% or maybe %0.1? Does it worth worrying? Of course I understand that it's user-specific, and a gentoo world rebuild produces different results from internet browsing. So I'm not asking about some general stats for an average user, just in your case how many writes are added by journal? Anyway, if you care about journal, you should care about the whole disk, since most writes are not related to /tmp. In real applications /tmp is mostly unused, while /home and /var produce most of the write attempts. So worrying *only* about /tmp is pointless. I'm not saying that there's nothing to worry about. It's up to you whether you want to worry about SSD writes or not. I'm saying that by mounting /tmp to tmpfs you solve nothing. Even worse, that may give you a false safety feeling. But if you care, debian already has everything you need, here are a few suggestions that can help you: Option 1: Normal: don't write some things = The idea is simple: you need to find most writing apps and configure them not to do that. To find such apps you can use tools: iostat iotop btrace ... You may also look at audit-related tools. Or you can write your own scripts using /proc/diskstats and /proc/PID/io. Next step depends on apps you use. For example, you may want to use private browsing in firefox. Or if you're heavily using vim you can disable its swap file (`set noswapfile` in .vimrc). If you notice that your (K)DE often writes to /var/tmp, you can mount /var/tmp to tmpfs. :) Of course you would probably want to increase dirty_*_centisecs timeouts to some larger values. In my case most writes were done by browser (so I disabled caching, symlinked history/cache to tmpfs) and syslog (disabled fsync). It takes some time to catch all the apps. You may want to write some script and leave it running in background and collecting stats of running processes, so you could monitor your writes from time to time. Option 2: Paranoid: only write things you need == Assuming that you have a normal non-SSD PC: 0. Split your SSD disk into two partitions meta-root and other. The meta-root partition will contain your system, while all the files you want to save would be on other partition. 1. Create a debian livecd with all the software you need: http://live.debian.net/manual-3.x/html/live-manual.en.html#7 2. Unpack content of the livecd to meta-root partition. 3. Manually install boot-loader (i.e. grub or syslinux) to meta-root 4. Configure boot-loader to boot your livecd 5. Boot and use. :) Whenever you need to write something — write it to other partition. If you need some software that you forgot to put on a livecd, or if you want to upgrade — leave a note to yourself on other partition, later rebuild livecd and replace files on meta-root partition with a new build. This is harder to configure, but allows you to control every disk write happening in your system. Also this gives you a flexible undo: if something
Re: Moving /tmp to tmpfs makes it useful
On 26 May 2012 19:20, Carlos Alberto Lopez Perez clo...@igalia.com wrote: *Important*: use df -h /tmp NOT du -hs /tmp, since the flash player deletes the file entry from /tmp as soon as it gets the inode allocated. I believe this a measure to prevent piracy (people ripping the video from /tmp) I think you're too willing to assume bad faith. If I were writing similar software, that's exactly what I'd do: create a temporary file, which will be in /tmp except in the marginal corner-case that the user has set another $TMPDIR, then unlink it and continue using it. This means that you don't need to worry about deleting it afterwards, even if your application crashes or is killed, and has no obvious downsides. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/cahb+spdrxfpd1e6dovwfg8pttbezduqgchskcwb0l+oc3p3...@mail.gmail.com
Re: Moving /tmp to tmpfs makes it useful
Salvo Tomaselli tipos...@tiscali.it writes: This is indeed a valid point. But that’s no regression; /tmp has always been for small short-lived files, and /var/tmp for those that are not one of them or not both. You just made up this difference. No he didn't, it's common practice from unix-days-of-yore (though it was /usr/tmp or whatever back then...). -miles -- Christian, n. One who follows the teachings of Christ so long as they are not inconsistent with a life of sin. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87wr3wal0m@catnip.gol.com
Re: Moving /tmp to tmpfs makes it useful
On 28/05/12 08:15, Miles Bader wrote: Salvo Tomaselli tipos...@tiscali.it writes: This is indeed a valid point. But that’s no regression; /tmp has always been for small short-lived files, and /var/tmp for those that are not one of them or not both. You just made up this difference. No he didn't, it's common practice from unix-days-of-yore (though it was /usr/tmp or whatever back then...). -miles common practice? sorry I don't know about any standard which such name. Can you link here any document which says that its recommended the use of /tmp for small files? -- ~~~ Carlos Alberto Lopez Perez http://neutrino.es Igalia - Free Software Engineeringhttp://www.igalia.com ~~~ signature.asc Description: OpenPGP digital signature
Re: Moving /tmp to tmpfs makes it useful
On 05/27/2012 02:55 AM, Mike Hommey wrote: I hope it isn't, because it's poor security. It's easy to read the file from /proc/$pid/fd/$num Mike If you were to trust Adobe in terms of security, then you'd be the first person I know to do so. Jokes set apart, I believe that they just changed the behavior to make it a bit more difficult for newbies, knowing it's still possible to go around this hack (just a wild guess). Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fc2273d.6070...@debian.org
Re: Moving /tmp to tmpfs makes it useful
On 05/27/2012 02:35 AM, François Bottin wrote: My point is that I can't stand that the major point to not use tmpfs is caused by a closed source software. If you think it is faulty, then do not use it! By the way, I can't reproduce your problem. That's probably due to the fact that there is no non-free software on my machines... Flash is *not* the major point, it's only one of them. We gave you *many* other examples. Shall I repeat again? - MySQL - cd burner software - midnight commander - image viewers - web browsers and mail clients - archive managers - ... These are all very commonly used software on both the server side (eg: MySQL, possibly mc too, but you may find others) and the desktop (all others on the list above). Please stop making us repeat what was on the first message of this tread. This is starting to be quite annoying, and this isn't constructive at all. Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fc22895.4030...@debian.org
Re: Moving /tmp to tmpfs makes it useful
Joey Hess dixit: If programs that write large files to /tmp are changed to usr /var/tmp, then over time a system will accumulate orphaned large tmp files in /var/tmp. Nothing will come along and clean them up. This is indeed a valid point. But that’s no regression; /tmp has always been for small short-lived files, and /var/tmp for those that are not one of them or not both. The question whether there needs to be a way to store large temporary files that are cleaned up automatically (yes, please), when (Ben Hutchings fsck experience comes to mind) and how (?) should be orthogonal to the /tmp as tmpfs discussion (but be done). For what it’s worth, I’ve run into limits when having /tmp as a filesystem more often than when it was in RAM. bye, //mirabilos -- ch you introduced a merge commit│mika % g rebase -i HEAD^^ mika sorry, no idea and rebasing just fscked │mika Segmentation ch should have cloned into a clean repo │ fault (core dumped) ch if I rebase that now, it's really ugh │mika:#grml wuahh -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/pine.bsm.4.64l.1205271759070.24...@herc.mirbsd.org
Re: Moving /tmp to tmpfs makes it useful
This is indeed a valid point. But that’s no regression; /tmp has always been for small short-lived files, and /var/tmp for those that are not one of them or not both. You just made up this difference. Nothing nowhere says anything about the size of the files. The only thing mentioned is the frequency of the cleanup. Bye -- Salvo Tomaselli -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/201205272015.47774.tipos...@tiscali.it
Re: Moving /tmp to tmpfs makes it useful
On 27/05/12 20:01, Thorsten Glaser wrote: This is indeed a valid point. But that’s no regression; /tmp has always been for small short-lived files, and /var/tmp for those that are not one of them or not both. I am still waiting for someone to tell me where is said that /tmp should be for small files. Neither the FHS [1] or the IEEE standard P1003.2 (POSIX, part 2) [2] or any other standard (that I know about) says that /tmp should be for small files So... where is said that files on /tmp should be small? Please, enlighten me Regards! [1] http://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html [2] http://www.educat.hu-berlin.de/doc/Posix-1003.2.txt -- ~~~ Carlos Alberto Lopez Perez http://neutrino.es Igalia - Free Software Engineeringhttp://www.igalia.com ~~~ signature.asc Description: OpenPGP digital signature
Re: Moving /tmp to tmpfs makes it useful
On Sat, May 26, 2012 at 03:24:02AM +0800, Thomas Goirand wrote: /tmp 8,0G 60M 8,0G1% /tmp Does this count as large files? As a lot of small-only files? Exactly how is this a practical explanation and example? :/ Are you saying that in *your case* /tmp is almost unused? I'm saying that 60M is a substantial amount which still may be not called large. -- WBR, wRAR signature.asc Description: Digital signature
Re: Moving /tmp to tmpfs makes it useful
On Fri, 25 May 2012 21:56:55 -0400 Ted Ts'o ty...@mit.edu wrote: The major difference is that tmpfs pages only get written out to swap when the system is under memory pressure. In contrast, pages which are backed by a filesystem will start being written to disk after 30 seconds _or_ if the system is under memory pressure. I still think the major difference is that page cache - ext2 can be quota'd while tmpfs - swap cannot (AFAIK; has that changed in the past couple of years?). This may be good or bad depending on your use case. And if you consider how much memory most desktop/laptops have, and how often people **really** are downloading multi-gigabyte files to /tmp (note that browsers tend to default downloads to ~/Downloads) Browsers write files to ~/Downloads (or ~/Desktop) *when the user selects Save As*. On Iceweasel and Chrome at least, if you click a link to a content-type the browser cannot natively display, it downloads it to /tmp while it's waiting for you to tell it what to do with it. These files can be arbitrarily large. Weldon -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120526073853.d0868690.wel...@b.rontosaur.us
Re: Moving /tmp to tmpfs makes it useful
2012/5/26 Clint Byrum: On laptops and other power sensitive devices, this is pretty critical. Hypothetical: I have 2GB of RAM, and I want to watch a 50MB video file on a connection that will take, say, 10 minutes to cache the whole thing (and its a 10 minute video). With a regular filesystem hosting /tmp, Every 30 seconds I will wake up the hard disk, and write data to it. There's a much better solution to that: /proc/sys/vm/laptop_mode and laptop-mode-tools. If you `echo 1 /proc/sys/vm/laptop_mode` disk will spin up only when reading. laptop-mode-tools usually also increase dirty_*_centisecs allowing to not write cache for minutes. So you get all the benefits of tmpfs without its problems, like heavy swapping or size limited by memory. :) I doubt most spinning disks will go to sleep in 30 seconds Mine spins down in 5 seconds. -- Serge -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAOVenEpJzU+kqpKy5-g7XNA8RH99jOn7aarw43Gfnf-rMO=v...@mail.gmail.com
Re: Moving /tmp to tmpfs makes it useful
On 05/26/2012 04:40 PM, Andrey Rahmatullin wrote: On Sat, May 26, 2012 at 03:24:02AM +0800, Thomas Goirand wrote: /tmp 8,0G 60M 8,0G1% /tmp Does this count as large files? As a lot of small-only files? Exactly how is this a practical explanation and example? :/ Are you saying that in *your case* /tmp is almost unused? I'm saying that 60M is a substantial amount which still may be not called large. Try playing a 2h video with flash, and see that 60 MB isn't enough... Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fc0ed09.3080...@debian.org
Re: Moving /tmp to tmpfs makes it useful
On Sat, May 26, 2012 at 10:47:37PM +0800, Thomas Goirand wrote: /tmp 8,0G 60M 8,0G1% /tmp Does this count as large files? As a lot of small-only files? Exactly how is this a practical explanation and example? :/ Are you saying that in *your case* /tmp is almost unused? I'm saying that 60M is a substantial amount which still may be not called large. Try playing a 2h video with flash, and see that 60 MB isn't enough... Enough for what? 60 Mb is used space, not total space. And how is that relevant to my question anyway? -- WBR, wRAR signature.asc Description: Digital signature
Re: Moving /tmp to tmpfs makes it useful
On 05/26/2012 01:33 PM, Clint Byrum wrote: On laptops and other power sensitive devices, this is pretty critical. Hypothetical: I have 2GB of RAM, and I want to watch a 50MB video file on a connection that will take, say, 10 minutes to cache the whole thing (and its a 10 minute video). With a regular filesystem hosting /tmp, Every 30 seconds I will wake up the hard disk, and write data to it. I doubt most spinning disks will go to sleep in 30 seconds, so this is more than 10 minutes solid of hard disk spinning. With tmpfs, there is no memory pressure, so my disk never even spins up to write anything to it. If I do run into memory pressure, yes, I need to use swap at that point. But at that point I've got a lot more than just the disk draining power. Now, another hypothetical. Same machine but with 1 GB of RAM (please, don't tell me that's unusual...), a 600 MB video file, with a video that is 2 hours long. You have Firefox started (to watch the video), and many pluggins and many tabs open, and it's taking 600 MB of RAM (that's nothing really unusual, unfortunately). After a while, your HDD starts spinning like hell, because it's swapping... Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fc0ee82.6030...@debian.org
Re: Moving /tmp to tmpfs makes it useful
On 05/26/2012 04:47 PM, Thomas Goirand wrote: Try playing a 2h video with flash, and see that 60 MB isn't enough... If Adobe Flash is broken, then fix it! -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fc0f35a.5090...@free.fr
Re: Moving /tmp to tmpfs makes it useful
On 05/26/2012 11:14 PM, François Bottin wrote: On 05/26/2012 04:47 PM, Thomas Goirand wrote: Try playing a 2h video with flash, and see that 60 MB isn't enough... If Adobe Flash is broken, then fix it! I will let you ask the sources from Adobe, create an alternative *that works*. I will also let you fix all the other apps that we mentioned that have the same kind of issues. But *IN THE MEAN WHILE*, until you are done with this huge and important work, let's not change the default to tmpfs. Cheers, Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fc10e23.7050...@debian.org
Re: Moving /tmp to tmpfs makes it useful
Thomas Goirand z...@debian.org writes: I will let you ask the sources from Adobe, create an alternative *that works*. I will also let you fix all the other apps that we mentioned that have the same kind of issues. But *IN THE MEAN WHILE*, until you are done with this huge and important work, let's not change the default to tmpfs. I find some of the assertions in this thread confusing. I've been using tmpfs /tmp on my laptop for quite some time and have watched hour-long movies via the Adobe Flash player and have never noticed any unexpected consumption of space in /tmp. Are you sure that video streaming via Adobe's Flash player works the way that you seem to think it works? -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/878vgec1bv@windlord.stanford.edu
Re: Moving /tmp to tmpfs makes it useful
On 26/05/12 19:13, Russ Allbery wrote: Thomas Goirand z...@debian.org writes: I will let you ask the sources from Adobe, create an alternative *that works*. I will also let you fix all the other apps that we mentioned that have the same kind of issues. But *IN THE MEAN WHILE*, until you are done with this huge and important work, let's not change the default to tmpfs. I find some of the assertions in this thread confusing. I've been using tmpfs /tmp on my laptop for quite some time and have watched hour-long movies via the Adobe Flash player and have never noticed any unexpected consumption of space in /tmp. Are you sure that video streaming via Adobe's Flash player works the way that you seem to think it works? http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=666096 -- ~~~ Carlos Alberto Lopez Perez http://neutrino.es Igalia - Free Software Engineeringhttp://www.igalia.com ~~~ signature.asc Description: OpenPGP digital signature
Re: Moving /tmp to tmpfs makes it useful
Carlos Alberto Lopez Perez clo...@igalia.com writes: On 26/05/12 19:13, Russ Allbery wrote: I find some of the assertions in this thread confusing. I've been using tmpfs /tmp on my laptop for quite some time and have watched hour-long movies via the Adobe Flash player and have never noticed any unexpected consumption of space in /tmp. Are you sure that video streaming via Adobe's Flash player works the way that you seem to think it works? http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=666096 That bug contains little actual information, not even what software was being used. -- Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/ -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87txz2alef@windlord.stanford.edu
Re: Moving /tmp to tmpfs makes it useful
On 26/05/12 19:43, Russ Allbery wrote:
Carlos Alberto Lopez Perez clo...@igalia.com writes:
On 26/05/12 19:13, Russ Allbery wrote:
I find some of the assertions in this thread confusing. I've been
using tmpfs /tmp on my laptop for quite some time and have watched
hour-long movies via the Adobe Flash player and have never noticed any
unexpected consumption of space in /tmp.
Are you sure that video streaming via Adobe's Flash player works the
way that you seem to think it works?
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=666096
That bug contains little actual information, not even what software was
being used.
Ok... so you want a prove?
* Open with iceweasel http://vimeo.com/13726978
* Ensure that is playing with flashplayer (right click on the movie
should show the typical dialog that includes about flashplayer
* Now launch:
watch df -h /tmp/
* And see how the space used grows.
*Important*: use df -h /tmp NOT du -hs /tmp, since the flash player
deletes the file entry from /tmp as soon as it gets the inode allocated.
I believe this a measure to prevent piracy (people ripping the video
from /tmp)
See:
ls -l /proc/$(ps aux|grep libf[l]ashplayer.so|awk '{print $2}')/fd
And see how there is an entry pointing to /tmp/FlashXX that appears
as deleted but you can still access it. Just open /proc/pid/fd/number
with vlc for example
Regards!
--
~~~
Carlos Alberto Lopez Perez http://neutrino.es
Igalia - Free Software Engineeringhttp://www.igalia.com
~~~
signature.asc
Description: OpenPGP digital signature
Re: Moving /tmp to tmpfs makes it useful
Carlos Alberto Lopez Perez clo...@igalia.com writes:
On 26/05/12 19:43, Russ Allbery wrote:
Carlos Alberto Lopez Perez clo...@igalia.com writes:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=666096
That bug contains little actual information, not even what software was
being used.
Ok... so you want a prove?
I want the bug report to actually be useful, yes. :) It's always good to
have a reproducible test case.
* Open with iceweasel http://vimeo.com/13726978
* Ensure that is playing with flashplayer (right click on the movie
should show the typical dialog that includes about flashplayer
* Now launch:
watch df -h /tmp/
* And see how the space used grows.
*Important*: use df -h /tmp NOT du -hs /tmp, since the flash player
deletes the file entry from /tmp as soon as it gets the inode allocated.
I believe this a measure to prevent piracy (people ripping the video
from /tmp)
See:
ls -l /proc/$(ps aux|grep libf[l]ashplayer.so|awk '{print $2}')/fd
And see how there is an entry pointing to /tmp/FlashXX that appears
as deleted but you can still access it. Just open /proc/pid/fd/number
with vlc for example
Thank you! Those instructions are very good. I'm copying them to the bug
report so that the bug report contains clear information.
When I did this (on a laptop with 2GB of memory and 2.5GB of swap and a
405MB tmpfs /tmp), the movie exhausted all space in /tmp. It then kept
playing without any trouble, and the system never started swapping or had
any other performance issues. However, /tmp was completely full, and
creating any other files in /tmp resulted in errors (which I know can
cause all sorts of problems for other programs that assume they can always
create small files in /tmp).
I didn't continue to play the movie through to the end, so I don't know if
the movie playback would encounter problems at the point at which the
buffered movie exhausted /tmp or if the Flash player would have coped by
windowing the streaming.
--
Russ Allbery (r...@debian.org) http://www.eyrie.org/~eagle/
--
To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Archive: http://lists.debian.org/87bolaaj12@windlord.stanford.edu
Re: Moving /tmp to tmpfs makes it useful
On 05/26/2012 07:08 PM, Thomas Goirand wrote: On 05/26/2012 11:14 PM, François Bottin wrote: On 05/26/2012 04:47 PM, Thomas Goirand wrote: Try playing a 2h video with flash, and see that 60 MB isn't enough... If Adobe Flash is broken, then fix it! I will let you ask the sources from Adobe, create an alternative *that works*. I will also let you fix all the other apps that we mentioned that have the same kind of issues. But *IN THE MEAN WHILE*, until you are done with this huge and important work, let's not change the default to tmpfs. My point is that I can't stand that the major point to not use tmpfs is caused by a closed source software. If you think it is faulty, then do not use it! By the way, I can't reproduce your problem. That's probably due to the fact that there is no non-free software on my machines... -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fc1226a.5090...@free.fr
Re: Moving /tmp to tmpfs makes it useful
On Sat, May 26, 2012 at 08:20:46PM +0200, Carlos Alberto Lopez Perez wrote: On 26/05/12 19:43, Russ Allbery wrote: Carlos Alberto Lopez Perez clo...@igalia.com writes: On 26/05/12 19:13, Russ Allbery wrote: I find some of the assertions in this thread confusing. I've been using tmpfs /tmp on my laptop for quite some time and have watched hour-long movies via the Adobe Flash player and have never noticed any unexpected consumption of space in /tmp. Are you sure that video streaming via Adobe's Flash player works the way that you seem to think it works? http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=666096 That bug contains little actual information, not even what software was being used. Ok... so you want a prove? * Open with iceweasel http://vimeo.com/13726978 * Ensure that is playing with flashplayer (right click on the movie should show the typical dialog that includes about flashplayer * Now launch: watch df -h /tmp/ * And see how the space used grows. *Important*: use df -h /tmp NOT du -hs /tmp, since the flash player deletes the file entry from /tmp as soon as it gets the inode allocated. I believe this a measure to prevent piracy (people ripping the video from /tmp) I hope it isn't, because it's poor security. It's easy to read the file from /proc/$pid/fd/$num Mike -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120526185519.ga28...@glandium.org
Re: Moving /tmp to tmpfs makes it useful
Ted Ts'o wrote: If you're worried about installations which don't have much memory (i.e., the 512mb netbook), then swap is absolutely mandatory, I would think! Not really. I have no legitimate programs that use more than 50% of my 1 gb. I have an SSD. So why enable swap? If chromium goes crazy, the OOM killer does what I want it to do. And if you consider how much memory most desktop/laptops have, and how often people **really** are downloading multi-gigabyte files to /tmp (note that browsers tend to default downloads to ~/Downloads), I really think the people who are agitating against tmpfs are really making a much more theoretical argument than one that seems likely to be hit by an unsophisticated user --- and a sophistcated user can easily decide whether to use /tmp on disk or not. I read the debian-user list and have forwarded a half-dozen cases of users experiencing problems with tmpfs /tmp to the BTS. -- see shy jo signature.asc Description: Digital signature
Re: Moving /tmp to tmpfs makes it useful
Henrique de Moraes Holschuh dixit: On Fri, 25 May 2012, Thomas Goirand wrote: for small files, and in that case, it's faster. In reality, it's not that much faster, thanks to Linux caching of the filesystem, Under heavy filesystem IO load, yes it is. By several orders of magnitude. Not just that. It’s faster because the files don’t even get written to disc if they’re kept for, say, five minutes, and aren’t just short-lived. If you’ve got a slow hard disc (say low-end spec or old system), RAM’s still cheaper for tmpfs: no matter whether tmpfs or a real on-disc filesystem, buffer cache would still get it, AND the writes would be very slow (especially for short-lived files, _if_ the system is otherwise busy and would not wait before writing them to disc), and the system could concentrate on getting the files we want to disc instead of the temporary ones. And then there’s the CF card usecase. Or SSD, nowadays, I guess (my IBM X40 just got two CF cards to replace the HDD). There, you’d absolutely want to minimise disc writes. And just the chance that it could not end up there (except as part of swap) is better then the chance it could end up there (plus journal writes). In fact it is the other way. We have /var/tmp for the large file since about forever, and important platforms that have /tmp in memory since the early 2000's (Solaris) Right. /tmp always had size limitations, it used to be no bigger than a few MiB, and people were ALWAYS tought to use /var/tmp for files that are large, should not be removed across reboots or from a cronjob pruning /tmp files older than 7 days, or a combination of these. (Uglities like /usr/tmp notwithstanding, although I’ve mostly seen that as compat symlink to /var/tmp later.) I’m absolutely for /tmp as tmpfs by default. Even, no, especially on low-end systems. Heck, be lucky it’s tmpfs and not Classical BSDs’ mfs (basically a variable-size 4.2FFS ramdisc, not just an object store in the buffer cache). Every tool either supports setting TMPDIR=/var/tmp before running it or is buggy. Go fix these instead, and then just run them with that if you need them to process files you don’t want in tmpfs. bye, //mirabilos -- [...] if maybe ext3fs wasn't a better pick, or jfs, or maybe reiserfs, oh but what about xfs, and if only i had waited until reiser4 was ready... in the be- ginning, there was ffs, and in the middle, there was ffs, and at the end, there was still ffs, and the sys admins knew it was good. :) -- Ted Unangst über *fs -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/pine.bsm.4.64l.1205251057460.4...@herc.mirbsd.org
Re: Moving /tmp to tmpfs makes it useful
2012/5/25 Thorsten Glaser wrote: It’s faster because the files don’t even get written to disc if they’re kept for, say, five minutes, and aren’t just short-lived. Theoretically, yes. But I'm not asking to forbid everybody to use tmpfs. We're talking about defaults and about the most used test cases. Can you name some popular real-world program that write only small files and become noticeably faster when /tmp is on tmpfs? Every tool either supports setting TMPDIR=/var/tmp before running it or is buggy. Go fix these instead Do I understand you right? You suggest every tool that need large tmp files to use /var/tmp instead? Ok, then... what popular tools should use /tmp now? And then there’s the CF card usecase. Or SSD, nowadays, I guess (my IBM X40 just got two CF cards to replace the HDD). There, you’d absolutely want to minimise disc writes. /tmp on tmpfs won't help in that case. You do not reduce number of disk writes, you just move them to other directories. As you suggested, all the programs will still use files i.e. in /var/tmp. Or they'll write to swap if tmpfs is large enough. Or they'll just break, if it's small. But you'll get the same number of SSD writes (or maybe even more, because of other applications being heavily swapped as well). Again, I'm not asking to drop this feature. I'm asking to have it disabled *by default* but supported by debian installer, so really smart people, that know what may be broken by it, but really need it, could enable it. -- Serge -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/CAOVenEp+mr3yNi=4obcg2qtztfjuaj_svcus1y-7ij5mnhj...@mail.gmail.com
Re: Moving /tmp to tmpfs makes it useful
On Fri, May 25, 2012 at 03:50:31PM +0300, Serge wrote: /tmp on tmpfs won't help in that case. You do not reduce number of disk writes, you just move them to other directories. As you suggested, all the programs will still use files i.e. in /var/tmp. Or they'll write to swap if tmpfs is large enough. Or they'll just break, if it's small. Why do you assume tmpfs contents is always written to the swap? -- WBR, wRAR signature.asc Description: Digital signature
Re: Moving /tmp to tmpfs makes it useful
On 25/05/12 13:50, Serge wrote: Again, I'm not asking to drop this feature. I'm asking to have it disabled *by default* but supported by debian installer, so really smart people, that know what may be broken by it, but really need it, could enable it. +1 On 25/05/12 13:52, Ted Ts'o wrote: So what? If you write to a normal file system, it goes into the page cache, which is pretty much the same as writing into tmpfs. In both cases if you have swap configured, the data will get pushed to disk; That's not at all the same, the page cache is more temporary, it's getting flushed to disk pretty quick if memory is tight (presumably) but in the same situation using tmpfs going to swap is surely going to be more disruptive? I won't pretend to know the details half as well as other commentators but it seems only logical that you'd end up pushing more memory from other running processes onto disk as well as (or instead of) the tmpfs memory, which is going to have to get reloaded at some point. And anyway, not everybody uses swap, in which case this default is not entirely viable. I, for one, had no idea this had become default for Debian and I think it's likely to be one of those things that jumps out to bite people who weren't expecting it at some inconvenient moment. I'm sure the project veterans and more attentive readers of this list are tired of recurring arguments like this, but usually if something is recurring it is for a reason. Given my general no swap preference, I'm glad this has come up again so that I'm aware of it this time. The tmpfs setup seems far more appropriate as a performance tweak for admins than as a default. Where there is plenty of RAM, buffer cache makes the difference largely negligible. But where there isn't an abundance of RAM, it could needlessly cause problems (especially without swap). Not big problems perhaps, and not likely to many, but that could still be a few thousand people with a project like Debain, so I just don't see the issue with leaving /tmp on disk, why complicate the matter? -Will -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fbf8dda.9070...@willdaniels.co.uk
Re: Moving /tmp to tmpfs makes it useful
2012/5/25 Andrey Rahmatullin wrote: Why do you assume tmpfs contents is always written to the swap? Most programs, using /tmp for temporary files, may write *large* files there. So most programs fill tmpfs with large files. So tmpfs grows and swaps out (probably swapping out other applications as well). I really can't think of any popular application that write a lot of small-only files and can benefit from /tmp being on tmpfs. And I know a lot of *popular* apps that will have troubles because of that. If default settings cause trouble in a lot of cases and cause no benefits, than it's a bad default, and it should be fixed. :) -- Serge -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/caoveneqkgq7pud1xenervfxwzrh3qxk_xd9eweuz7nkjdt2...@mail.gmail.com
Re: Moving /tmp to tmpfs makes it useful
On Fri, May 25, 2012 at 05:08:37PM +0300, Serge wrote: Why do you assume tmpfs contents is always written to the swap? Most programs, using /tmp for temporary files, may write *large* files there. So most programs fill tmpfs with large files. So tmpfs grows and swaps out (probably swapping out other applications as well). I really can't think of any popular application that write a lot of small-only files and can benefit from /tmp being on tmpfs. And I know a lot of *popular* apps that will have troubles because of that. /tmp 8,0G 60M 8,0G1% /tmp Does this count as large files? As a lot of small-only files? -- WBR, wRAR signature.asc Description: Digital signature
Re: Moving /tmp to tmpfs makes it useful
Serge dixit: cases. Can you name some popular real-world program that write only small files and become noticeably faster when /tmp is on tmpfs? Most shell scripts using (here documents). mc, out of all things, as long as the temporary files (e.g. of a tarball) fit inside it. And countless others. The majority, in any case. What you want is to optimise for a corner case. Every tool either supports setting TMPDIR=/var/tmp before running it or is buggy. Go fix these instead Do I understand you right? You suggest every tool that need large tmp files to use /var/tmp instead? I do not suggest it. I say that every tool that doesn’t honour when the user sets TMPDIR=/var/tmp is broken (but that’s nothing new, as it has always been like that), and that the user should set that in some corner cases like yours. Ok, then... what popular tools should use /tmp now? /tmp (on tmpfs) should still be the default, as that’s the common case, and one that can be optimised for well. /tmp on tmpfs won't help in that case. You do not reduce number of disk writes, you just move them to other directories. As you suggested, all No. You reduce the number of disc writes quite a bit, as all current filesystems use journalling and write more additional metadata than paging would ever use. (And, from the other mail, the read accesses could be fixed (for the slow HDD case) or don’t matter that much (for the CF/SSD case).) the programs will still use files i.e. in /var/tmp. Or they'll write to swap if tmpfs is large enough. Or they'll just break, if it's small. Ever heard of statfs? But you'll get the same number of SSD writes (or maybe even more, because of other applications being heavily swapped as well). No. The common case wouldn’t swap them a lot, besides read-only things (.text and .rodata, plus some bits) would not need to be written anyway. So you’d still get less. Again, I'm not asking to drop this feature. I'm asking to have it disabled *by default* but supported by debian installer, so really smart people, that know what may be broken by it, but really need it, could enable it. And I’m proving that it should be *enabled by default*, so that really smart people like you, who need it disabled in corner cases, can do that. And I’m suggesting that file managers that extract archives apply some heuristics (size of the archive vs. size and current usage of /tmp) to determine whether they should use /tmp or /var/tmp. Most programs, using /tmp for temporary files, may write *large* files No. The absolute majority writes *small* files there. there. So most programs fill tmpfs with large files. So tmpfs grows and swaps out (probably swapping out other applications as well). You can always limit the size of your tmpfs? I think in Debian it is already sized to a quite small (IMHO) portion of the main memory by default. I really can't think of any popular application that write a lot of small-only files and can benefit from /tmp being on tmpfs. And I $GUI_BROWSER_DU_JOUR with XTaran’s unburden-home-dir, for example. bye, //mirabilos -- Using Lynx is like wearing a really good pair of shades: cuts out the glare and harmful UV (ultra-vanity), and you feel so-o-o COOL. -- Henry Nelson, March 1999 -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/pine.bsm.4.64l.1205251615020.20...@herc.mirbsd.org
Re: Moving /tmp to tmpfs makes it useful
On 05/25/2012 10:20 PM, Andrey Rahmatullin wrote: /tmp 8,0G 60M 8,0G1% /tmp Does this count as large files? As a lot of small-only files? Exactly how is this a practical explanation and example? :/ Are you saying that in *your case* /tmp is almost unused? That doesn't even remotely help to write this. Serge made a much better point with common apps. Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fbfdc52.20...@debian.org
Re: Moving /tmp to tmpfs makes it useful
On 05/26/2012 12:21 AM, Thorsten Glaser wrote: What you want is to optimise for a corner case. Ah... So Mysql, flash videos, mc, image editors, cd burners, and all the (very common) examples he gave are just corner cases in your book? We must have very different readings... Thomas -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/4fbfdd03.3050...@debian.org
Re: Moving /tmp to tmpfs makes it useful
Serge sergem...@gmail.com writes: Every tool either supports setting TMPDIR=/var/tmp before running it or is buggy. Go fix these instead Do I understand you right? You suggest every tool that need large tmp files to use /var/tmp instead? That's not a new suggestion, it's been standard practice for nigh-on _decades_... -miles -- Do not taunt Happy Fun Ball. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/87zk8vdbnx@catnip.gol.com
Re: Moving /tmp to tmpfs makes it useful
2012/5/25 Thorsten Glaser wrote: Ok, we have guesses, let's do some TESTS... Can you name some popular real-world program that write only small files and become noticeably faster when /tmp is on tmpfs? Most shell scripts using (here documents). mc, out of all things, as long as the temporary files (e.g. of a tarball) fit inside it. And countless others. The majority, in any case. What you want is to optimise for a corner case. Do they become faster? My tests show they don't (see below). I'm asking for *popular* apps, that create files in /tmp, *never put large files* there, and become *noticeably* faster with /tmp on tmpfs? And remember that there're popular apps that break because of that, so to make sure it's still a good thing all 3 conditions should be met. We're talking about defaults, not about corner cases after all... I say that every tool that doesn’t honour when the user sets TMPDIR=/var/tmp is broken (but that’s nothing new, as it has always been like that), and that the user should set that in some corner cases like yours. Browsers, filemanagers, flash video, cd burners... Do you basically suggest every debian user to think whether it's a corner case or not and manually select TMPDIR *every time* they start browser? ;) You reduce the number of disc writes quite a bit, as all current filesystems use journalling and write more additional metadata than paging would ever use. I believe you're not talking about some small short-lived temporary files, because they never actually get to disk, kernel filesystem cache is smart enough for that (unless you've changed dirty_writeback_centisecs and dirty_expire_centisecs to some low values). So you expect to reduce number of disk writes for *large* files quite a bit. I'm curious how much is that? I wanted to test: # time cp * /mnt/temp/ # mount tmpfs /mnt/temp -t tmpfs -o size=10G # time cp * /mnt/temp/ watching `iostat -k 1` but failed. My 2GB RAM was filled pretty quickly and the system almost died. Anybody feels the power to compare time and number of kB_wrtn by swap and extfs for the file that's at least twice as large as physical RAM? And I’m suggesting that file managers that extract archives apply some heuristics (size of the archive vs. size and current usage of /tmp) to determine whether they should use /tmp or /var/tmp. So you suggest every file/archive manager developers to change their code (with a not so trivial change) and ignore FHS because debian decided to change some variable from no to yes by default? And what's this all for? What's the benefit? It does not make them faster: # mkdir /mnt/temp # mount tmpfs /mnt/temp -t tmpfs -o size=10G # cd /mnt/temp # time tar xf ~/linux-3.4.tar.bz2 real0m27.100s user0m24.327s sys 0m2.100s # cd .. ; umount temp ; cd temp # time tar xf ~/linux-3.4.tar.bz2 real0m28.072s user0m22.103s sys 0m4.767s It varies from 26 to 31 seconds between runs, but the result is obvious — no noticeable difference. User won't notice the speed, but will notice problems when system becomes heavily swapping or when some apps break because of not enough space in /tmp. ... apply some heuristics ... Just curious, what heuristics? I've attached a small archive to this email, can you estimate its unpacked size without unpacking it? ;) I really can't think of any popular application that write a lot of small-only files and can benefit from /tmp being on tmpfs. $GUI_BROWSER_DU_JOUR with XTaran’s unburden-home-dir, for example. `unburden-home-dir` is a bad example: (a) It's a real corner case (it's the first time I've heard about it) (b) It can store cache in any directory, not just /tmp (c) Those files may be really large, so it may also break or make your system unstable because of tmpfs. And, finally, (d) it still does not benefit from /tmp being on tmpfs. It's not making anything noticeably faster because of tmpfs, is it? PS: there's a major difference between most programs will still work and most programs will become faster. You're telling that in many cases apps will still work somehow with /tmp being on tmpfs. And YOU'RE RIGHT, they will. And if they won't there're workarouns like TMPDIR. That's true. What I'm saying is that most apps will work *better* with /tmp being on real disk, system will use less RAM and be more stable, user won't have to use any workarounds and *won't lose anything*. Thanks for reading this HUGE message. -- Serge archive.bz2.gz Description: GNU Zip compressed data
Re: Moving /tmp to tmpfs makes it useful
Serge sergem...@gmail.com writes: I'm asking for *popular* apps, that create files in /tmp, *never put large files* there, and become *noticeably* faster with /tmp on tmpfs? Is that even the issue, for the most part? My impression is that using tmpfs is just logistically simpler and safer -- it can be mounted very early, doesn't muck with the (possibly small or r/o) root fs, makes it simple to limit temp space without a separate partition, etc. -miles -- 「寒いね」と話しかければ「寒いね」と答える人のいるあったかさ [俵万智]
Re: Moving /tmp to tmpfs makes it useful
On Fri, May 25, 2012 at 02:49:14PM +0100, Will Daniels wrote: On 25/05/12 13:52, Ted Ts'o wrote: So what? If you write to a normal file system, it goes into the page cache, which is pretty much the same as writing into tmpfs. In both cases if you have swap configured, the data will get pushed to disk; That's not at all the same, the page cache is more temporary, it's getting flushed to disk pretty quick if memory is tight (presumably) but in the same situation using tmpfs going to swap is surely going to be more disruptive? There will be some, but really, not that much difference between going from tmpfs to swap compared to files written to a filesystem (in both cases the data is stored in the page cache, whether it's a tmpfs file or an ext2/3/4 or xfs or btrfs file) in many cases. The major difference is that tmpfs pages only get written out to swap when the system is under memory pressure. In contrast, pages which are backed by a filesystem will start being written to disk after 30 seconds _or_ if the system is under memory pressure. So if you have a super-fast networking connection, it won't matter; the download fill the memory very quickly, at which point it will get written to swap and/or the file location on disk at the same rate, since you'll be able to download faster than you can save to disk, so the network connection will get throttled due to TCP/IP backpressure to roughly the same rate as you are writing to the HDD. If you have a slow networking connection, it's possible that the 30 second writeback timer will kick off before you start filling the memory --- but in that case, it's not going to be that disrupting in the tmpfs case, either. You'll hit memory pressure, and at that point you'll start writing to disk perhaps a bit later than the 30 second writeback timer. But at the same time, the download is coming in slowly enough that you're probably not overwhelming the speed at which you can write to the HDD or SSD. The place where it will make a difference is if you have a very large amount of memory, *and* you are downloading a really big file to /tmp (substantially bigger than your physical memory), *and* your effective end-to-end download speed is faster than your HDD write speed, but slow enough that it takes substantially longer than 30 seconds to download enough to fill your free physical memory. But that's actually a pretty narrow window. And anyway, not everybody uses swap, in which case this default is not entirely viable. I, for one, had no idea this had become default for Debian and I think it's likely to be one of those things that jumps out to bite people who weren't expecting it at some inconvenient moment. Well, it's all about defaults, right? It's easy enough to set up a swap partition, or even just a swap file by default in the installer. You can set up a swap file on the fly, so it's not that hard to deal with it after the fact. I'm sure the project veterans and more attentive readers of this list are tired of recurring arguments like this, but usually if something is recurring it is for a reason. Given my general no swap preference, I'm glad this has come up again so that I'm aware of it this time. The tmpfs setup seems far more appropriate as a performance tweak for admins than as a default. Where there is plenty of RAM, buffer cache makes the difference largely negligible. But where there isn't an abundance of RAM, it could needlessly cause problems (especially without swap). If you're worried about installations which don't have much memory (i.e., the 512mb netbook), then swap is absolutely mandatory, I would think! And if you consider how much memory most desktop/laptops have, and how often people **really** are downloading multi-gigabyte files to /tmp (note that browsers tend to default downloads to ~/Downloads), I really think the people who are agitating against tmpfs are really making a much more theoretical argument than one that seems likely to be hit by an unsophisticated user --- and a sophistcated user can easily decide whether to use /tmp on disk or not. - Ted -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/20120526015655.ga5...@thunk.org
Re: Moving /tmp to tmpfs makes it useful
2012/5/26 Miles Bader wrote: I'm asking for *popular* apps, that create files in /tmp, *never put large files* there, and become *noticeably* faster with /tmp on tmpfs? Is that even the issue, for the most part? My impression is that using tmpfs is just logistically simpler and safer -- it can be mounted very early, doesn't muck with the (possibly small or r/o) root fs, makes it simple to limit temp space without a separate partition, etc. We're talking about *default* settings. That what makes it an issue. It won't be an issue if most debian installations were on a small read-only root and most of them wanted /tmp to be tmpfs because of that. But it's not. I doubt major part of debians use r/o root and even if they do, not all of them would like to use tmpfs for that case. Most servers dedicate separate partitions to /tmp. I had it mount-bound to /home/tmp in that case. There're many solutions that are fast, use fewer memory, support quotas... Why tmpfs? What makes things worse: there're popular programs and popular usecases that break because of that. It may slow down or even freeze the system because of heavy swapping. So there must be a strong good reason surpassing these problems. I assume people supporting /tmp on tmpfs actually use it that way. What I don't understand is why they're arguing that it's not that bad instead of just saying why it's good? Does it makes something faster? What? How many seconds faster? Does it reduces disks operations? In what cases and how much? If so many people tested RAMTMP and found no good reasons for it, then maybe let's change it back to no by default? That's what testing is for anyway? -- Serge -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/caovenep+aeoxvtyomjksxrsdasno3j80w4byzsb0esvcant...@mail.gmail.com
Re: Moving /tmp to tmpfs makes it useful
Excerpts from Ted Ts'o's message of 2012-05-25 18:56:55 -0700: On Fri, May 25, 2012 at 02:49:14PM +0100, Will Daniels wrote: On 25/05/12 13:52, Ted Ts'o wrote: So what? If you write to a normal file system, it goes into the page cache, which is pretty much the same as writing into tmpfs. In both cases if you have swap configured, the data will get pushed to disk; That's not at all the same, the page cache is more temporary, it's getting flushed to disk pretty quick if memory is tight (presumably) but in the same situation using tmpfs going to swap is surely going to be more disruptive? There will be some, but really, not that much difference between going from tmpfs to swap compared to files written to a filesystem (in both cases the data is stored in the page cache, whether it's a tmpfs file or an ext2/3/4 or xfs or btrfs file) in many cases. The major difference is that tmpfs pages only get written out to swap when the system is under memory pressure. In contrast, pages which are backed by a filesystem will start being written to disk after 30 seconds _or_ if the system is under memory pressure. On laptops and other power sensitive devices, this is pretty critical. Hypothetical: I have 2GB of RAM, and I want to watch a 50MB video file on a connection that will take, say, 10 minutes to cache the whole thing (and its a 10 minute video). With a regular filesystem hosting /tmp, Every 30 seconds I will wake up the hard disk, and write data to it. I doubt most spinning disks will go to sleep in 30 seconds, so this is more than 10 minutes solid of hard disk spinning. With tmpfs, there is no memory pressure, so my disk never even spins up to write anything to it. If I do run into memory pressure, yes, I need to use swap at that point. But at that point I've got a lot more than just the disk draining power. -- To UNSUBSCRIBE, email to debian-devel-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/1338010047-sup-1...@fewbar.com
