Re: [ubuntu-cloud] RFC on Cloud Images: Make /tmp a tmpfs

[Dustin Kirkland]

On Thu, Jan 21, 2016 at 4:14 AM, Martin Pitt  wrote:
> Robie Basak [2016-01-21  9:56 +]:
>> I think we need a single canonical (small c) answer. Let's say that I'm
>> an upstream maintainer of a project that currently uses large amounts of
>> space in /tmp (say debdiff on kernel sources, for example). A user files
>> a bug that his machine now explodes when he uses my program. How should
>> I fix the bug?
>
> How is that any different than what we have now? systems with more
> memory than space on the root partition do exist [1], systems with
> /tmp on tmpfs do exist. We are *not* going from "/tmp has indefinite
> space" to "/tmp has little space", we are just changing the limit (not
> necessarily downwards even!) to a differently fuzzy definition.

Martin is absolutely correct!

Ubuntu's AWS images which are EBS backed -- the VAST majority of
Ubuntu currently running in the world -- all have a root volume of 7.8
GB.  Here's a fresh instance:

ubuntu@ip-172-30-0-30:~⟫ df -h /
Filesystem  Size  Used
Avail Use% Mounted on
/dev/disk/by-uuid/0a76513a-37fc-43df-9833-34f8f9598ada  7.8G  790M  6.6G  11% /

With Ubuntu 14.04 LTS, /tmp is still on that root volume, giving me
~6.6 GB of usable space there:

ubuntu@ip-172-30-0-30:~⟫ df -h /tmp/
Filesystem  Size  Used
Avail Use% Mounted on
/dev/disk/by-uuid/0a76513a-37fc-43df-9833-34f8f9598ada  7.8G  790M  6.6G  11% /

However, the instance that I just launched has 16 GB of memory:

ubuntu@ip-172-30-0-30:~⟫ free
 total   used   free sharedbuffers cached
Mem:  16433132 380392   16052740428   8468 209584
-/+ buffers/cache: 162340   16270792
Swap:0  0  0

Let's remount /tmp on tmpfs:

ubuntu@ip-172-30-0-30:~⟫ df -h /tmp/
Filesystem  Size  Used Avail Use% Mounted on
tmpfs   7.9G 0  7.9G   0% /tmp

How about that...  My /tmp space actually went up!

In fact, 27 of the 39 current AWS instance types have >15GB of memory.

Oh, and EBS writes -- are SLOW!

ubuntu@ip-172-30-0-30:~⟫ dd if=/dev/zero of=/tmp/zero bs=2G count=1
2147479552 bytes (2.1 GB) copied, 1.35841 s, 1.6 GB/s

ubuntu@ip-172-30-0-30:~⟫ dd if=/dev/zero of=/home/ubuntu/zero bs=2G
count=1

  2147479552 bytes
(2.1 GB) copied, 2.3478 s, 915 MB/s

ubuntu@ip-172-30-0-30:~⟫ dd if=/dev/zero of=/tmp/zero bs=2G count=1
oflag=dsync

2147479552 bytes
(2.1 GB) copied, 1.34014 s, 1.6 GB/s

ubuntu@ip-172-30-0-30:~⟫ dd if=/dev/zero of=/home/ubuntu/zero bs=2G
count=1 oflag=dsync

  2147479552 bytes (2.1
GB) copied, 23.1934 s, 92.6 MB/s

So on these instance types, /tmp on tmpfs provides both *more*
storage, and *faster* storage!

tmpfsffs,
Dustin

> If you program dumps vast amounts of data into /tmp, that is a problem
> somewhere no matter what you do. /var/tmp/ has traditionally had a
> better chance of carrying large files (but of course also not
> guaranteed).
>
> Martin
>
>
> [1] We've even had bug reports where people filled the root partition
> and then weren't able to log in any more because /tmp/ could not be
> written into. We detected this on boot and mounted a small tmpfs over
> /tmp/ to mitigate this.
>
> --
> Martin Pitt| http://www.piware.de
> Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
>
> --
> ubuntu-devel mailing list
> ubuntu-de...@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
>

-- 
Ubuntu-cloud mailing list
Ubuntu-cloud@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-cloud

Re: [ubuntu-cloud] RFC on Cloud Images: Make /tmp a tmpfs

[Dustin Kirkland]

On Thu, Jan 21, 2016 at 5:38 AM, Martin Pitt  wrote:
> Dimitri John Ledkov [2016-01-21 11:31 +]:
>> Is it just me, or did the RFC asks about cloud-images alone, and we
>> are diverging to discussing all the things, and all the form factors,
>> and all the installation types.
>
> That's true, sorry for diverging this.
>
>> And, e.g. imho the default should be off for cloud-images as pointed
>> out earlier. But others are advocating otherwise.
>
> FTR, I agree. I don't think it makes much sense for VMs to do this,
> unless of course they have a r/o root (but then you don't have a
> choice anyway, so this isn't part of this discussion).

Respectfully, Martin, the data and research disagrees.

I/O to EBS, Azure, or any other physical disks in the cloud is really
quite painfully slow.

-- 
Ubuntu-cloud mailing list
Ubuntu-cloud@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-cloud

Re: [ubuntu-cloud] RFC on Cloud Images: Make /tmp a tmpfs

[Dustin Kirkland]

On Sat, Jan 16, 2016 at 7:49 PM, Clint Byrum  wrote:
> Excerpts from Dustin Kirkland's message of 2016-01-16 04:25:58 -0800:
>> On Fri, Jan 15, 2016 at 2:25 AM, Seth Arnold  
>> wrote:
>> > On Thu, Jan 14, 2016 at 12:27:58PM +0200, Dustin Kirkland wrote:
>> >> Moreover, just 'sudo apt-get install swapspace' and watch as swapfiles
>> >> are created/deleted as needed.  If your root disk is lvm-encrypted,
>> >> then obviously such swap files are encrypted, too.
>> >
>> > I've been severely skeptical of the swapspace package:
>> >
>> > - Swap is used when the system is already under pressure; a few hundred
>> >   megs is great and probably for the best but if the system is actively
>> >   pushing beyond that then it's being pushed too hard.
>> >
>> > - If the swap space is going to be allocated on the fly, that means the
>> >   disk blocks have to zeroed on the fly, when the system is under
>> >   pressure, rather than at some leisurely time beforehand.
>> >
>> > - If the swap space is allocated on a filesystem, it's probably being
>> >   allocated from a fragmented filesystem that's 90% full rather than a
>> >   nice contiguous block of space as it would with a swap partition.
>> >
>> > - Accessing further into a file may involve loading multiple indirect
>> >   blocks from disk into unswappable kernel memory. A swap partition does
>> >   not require indirection blocks.
>> >
>> > - If the swap space allocated from a filesystem pushes the filesystem to
>> >   95% full (or whatever is left after accounting for reserved blocks),
>> >   programs will error and almost nothing handles "disk full" errors
>> >   gracefully. Swap partitions do not cause surprise gigabyte losses in
>> >   free space.
>> >
>> > - Swap files can't be allocated from btrfs filesystems and probably
>> >   shouldn't be allocated from zfs filesystems either. (Swap on zvols,
>> >   maybe.)
>> >
>> > Perhaps the swapspace package uses some tasteful tunables to mitigate
>> > against my concerns but the end result is that it contributes extra load,
>> > extra IO pressure, and extra uncertainty at a time when the system is
>> > already experiencing too much load, too much IO pressure, and too much
>> > uncertainty.
>> >
>> > The risks and downsides of swapspace feel like a lot compared to the
>> > slight hassle of having the installer make a swap partition.
>>
>> I count 4 "if's", 3 "probably's", 2 "should/would's", and 1 "maybe" in
>> that reply :-)
>>
>> Perhaps try it out?
>>
>> I've been running it and /tmp on tmpfs for several years (since before
>> ~precise) on my desktop on an encrypted LVM partition.  My machine has
>> a lot of memory (16GB), though I do push it hard), and have never
>> noticed a swapspace-related problem.  I've also used this combination
>> on hundreds of servers, and several production systems.
>>
>
> The 'if' and 'probably' are missing in your anecdotal evidence though.
> If you use the servers the way you have, it will probably work fine.
> Also we're talking about cloud instances, not "servers", which have
> quite different use and performance profiles.
>
> I'd like to see even some rudimentary experiments done with realistic
> workloads before saying this is a better idea than leaving things as
> they are. We've all speculated and provided anecdotal evidence enough to
> warrant such an investigation for those who speculate it will be a
> worthwhile change.

Sure, done!  You can find a detailed statistical analysis, as well as
the raw data for your download and treatment at:

http://blog.dustinkirkland.com/2016/01/data-driven-analysis-tmp-on-tmpfs.html

Based on a statistical analysis of 502 physical and virtual servers
running production and test services at Canonical (including
databases, websites, OpenStack, ubuntu.com, launchpad.net, et al.),
96.6% of them could fit all of the data they currently have in /tmp,
entirely in half of the free memory available in the system.  That
ratio goes up to 99.2% of the systems surveyed (i.e., all but 4) when
one takes into account both free available memory and available swap.
The remaining 4 systems are are currently using [101 GB, 42 GB, 13 GB,
and 10 GB] of swap, respectively, and are themselves somewhat special
cases.

Moreover, Ubuntu is hardly the first Linux/UNIX distribution that has
considered putting /tmp on tmpfs by default.  Solaris has used a tmpfs
since 1994.  Fedora moved to /tmp on tmpfs in 2012, as did ArchLinux.
Things seem to be working okay there...

As a recap, the benefits of /tmp on tmpfs are:
 - Performance: reads, writes, and seeks are insanely fast in a tmpfs;
as fast as accessing RAM (I tested 1.4GB/s writes and 1.1GB/s reads
to/from tmpfs)
 - Security: data leaks to disk are prevented (especially when swap is
disabled), and since /tmp is its own mount point, we should add the
nosuid and nodev options (and motivated sysadmins could optionally add
noexec, if they desire)
 - Energy efficiency: disk wake-ups 

Re: [ubuntu-cloud] RFC on Cloud Images: Make /tmp a tmpfs

[Clint Byrum]

Excerpts from Dustin Kirkland's message of 2016-01-19 20:27:51 -0800:
> On Sat, Jan 16, 2016 at 7:49 PM, Clint Byrum  wrote:
> > Excerpts from Dustin Kirkland's message of 2016-01-16 04:25:58 -0800:
> >> On Fri, Jan 15, 2016 at 2:25 AM, Seth Arnold  
> >> wrote:
> >> > On Thu, Jan 14, 2016 at 12:27:58PM +0200, Dustin Kirkland wrote:
> >> >> Moreover, just 'sudo apt-get install swapspace' and watch as swapfiles
> >> >> are created/deleted as needed.  If your root disk is lvm-encrypted,
> >> >> then obviously such swap files are encrypted, too.
> >> >
> >> > I've been severely skeptical of the swapspace package:
> >> >
> >> > - Swap is used when the system is already under pressure; a few hundred
> >> >   megs is great and probably for the best but if the system is actively
> >> >   pushing beyond that then it's being pushed too hard.
> >> >
> >> > - If the swap space is going to be allocated on the fly, that means the
> >> >   disk blocks have to zeroed on the fly, when the system is under
> >> >   pressure, rather than at some leisurely time beforehand.
> >> >
> >> > - If the swap space is allocated on a filesystem, it's probably being
> >> >   allocated from a fragmented filesystem that's 90% full rather than a
> >> >   nice contiguous block of space as it would with a swap partition.
> >> >
> >> > - Accessing further into a file may involve loading multiple indirect
> >> >   blocks from disk into unswappable kernel memory. A swap partition does
> >> >   not require indirection blocks.
> >> >
> >> > - If the swap space allocated from a filesystem pushes the filesystem to
> >> >   95% full (or whatever is left after accounting for reserved blocks),
> >> >   programs will error and almost nothing handles "disk full" errors
> >> >   gracefully. Swap partitions do not cause surprise gigabyte losses in
> >> >   free space.
> >> >
> >> > - Swap files can't be allocated from btrfs filesystems and probably
> >> >   shouldn't be allocated from zfs filesystems either. (Swap on zvols,
> >> >   maybe.)
> >> >
> >> > Perhaps the swapspace package uses some tasteful tunables to mitigate
> >> > against my concerns but the end result is that it contributes extra load,
> >> > extra IO pressure, and extra uncertainty at a time when the system is
> >> > already experiencing too much load, too much IO pressure, and too much
> >> > uncertainty.
> >> >
> >> > The risks and downsides of swapspace feel like a lot compared to the
> >> > slight hassle of having the installer make a swap partition.
> >>
> >> I count 4 "if's", 3 "probably's", 2 "should/would's", and 1 "maybe" in
> >> that reply :-)
> >>
> >> Perhaps try it out?
> >>
> >> I've been running it and /tmp on tmpfs for several years (since before
> >> ~precise) on my desktop on an encrypted LVM partition.  My machine has
> >> a lot of memory (16GB), though I do push it hard), and have never
> >> noticed a swapspace-related problem.  I've also used this combination
> >> on hundreds of servers, and several production systems.
> >>
> >
> > The 'if' and 'probably' are missing in your anecdotal evidence though.
> > If you use the servers the way you have, it will probably work fine.
> > Also we're talking about cloud instances, not "servers", which have
> > quite different use and performance profiles.
> >
> > I'd like to see even some rudimentary experiments done with realistic
> > workloads before saying this is a better idea than leaving things as
> > they are. We've all speculated and provided anecdotal evidence enough to
> > warrant such an investigation for those who speculate it will be a
> > worthwhile change.
> 
> Sure, done!  You can find a detailed statistical analysis, as well as
> the raw data for your download and treatment at:
> 
> http://blog.dustinkirkland.com/2016/01/data-driven-analysis-tmp-on-tmpfs.html
> 
> Based on a statistical analysis of 502 physical and virtual servers
> running production and test services at Canonical (including
> databases, websites, OpenStack, ubuntu.com, launchpad.net, et al.),
> 96.6% of them could fit all of the data they currently have in /tmp,
> entirely in half of the free memory available in the system.  That
> ratio goes up to 99.2% of the systems surveyed (i.e., all but 4) when
> one takes into account both free available memory and available swap.
> The remaining 4 systems are are currently using [101 GB, 42 GB, 13 GB,
> and 10 GB] of swap, respectively, and are themselves somewhat special
> cases.
> 

This is a very cool window into actual running servers, and I appreciate
the effort that has gone into this. This shows that most of the server
workloads will work fine with /tmp on tmpfs. I actually would love to
have a program which ran through a set of disk and memory metrics from
something like influxdb and basically told me "these should all be using
tmpfs you dummy!".

However, I do want to point out that I find the sample size very small.
A snapshot of a single point in 

Re: [ubuntu-cloud] RFC on Cloud Images: Make /tmp a tmpfs

[Clint Byrum]

Excerpts from Dustin Kirkland's message of 2016-01-16 04:25:58 -0800:
> On Fri, Jan 15, 2016 at 2:25 AM, Seth Arnold  
> wrote:
> > On Thu, Jan 14, 2016 at 12:27:58PM +0200, Dustin Kirkland wrote:
> >> Moreover, just 'sudo apt-get install swapspace' and watch as swapfiles
> >> are created/deleted as needed.  If your root disk is lvm-encrypted,
> >> then obviously such swap files are encrypted, too.
> >
> > I've been severely skeptical of the swapspace package:
> >
> > - Swap is used when the system is already under pressure; a few hundred
> >   megs is great and probably for the best but if the system is actively
> >   pushing beyond that then it's being pushed too hard.
> >
> > - If the swap space is going to be allocated on the fly, that means the
> >   disk blocks have to zeroed on the fly, when the system is under
> >   pressure, rather than at some leisurely time beforehand.
> >
> > - If the swap space is allocated on a filesystem, it's probably being
> >   allocated from a fragmented filesystem that's 90% full rather than a
> >   nice contiguous block of space as it would with a swap partition.
> >
> > - Accessing further into a file may involve loading multiple indirect
> >   blocks from disk into unswappable kernel memory. A swap partition does
> >   not require indirection blocks.
> >
> > - If the swap space allocated from a filesystem pushes the filesystem to
> >   95% full (or whatever is left after accounting for reserved blocks),
> >   programs will error and almost nothing handles "disk full" errors
> >   gracefully. Swap partitions do not cause surprise gigabyte losses in
> >   free space.
> >
> > - Swap files can't be allocated from btrfs filesystems and probably
> >   shouldn't be allocated from zfs filesystems either. (Swap on zvols,
> >   maybe.)
> >
> > Perhaps the swapspace package uses some tasteful tunables to mitigate
> > against my concerns but the end result is that it contributes extra load,
> > extra IO pressure, and extra uncertainty at a time when the system is
> > already experiencing too much load, too much IO pressure, and too much
> > uncertainty.
> >
> > The risks and downsides of swapspace feel like a lot compared to the
> > slight hassle of having the installer make a swap partition.
> 
> I count 4 "if's", 3 "probably's", 2 "should/would's", and 1 "maybe" in
> that reply :-)
> 
> Perhaps try it out?
> 
> I've been running it and /tmp on tmpfs for several years (since before
> ~precise) on my desktop on an encrypted LVM partition.  My machine has
> a lot of memory (16GB), though I do push it hard), and have never
> noticed a swapspace-related problem.  I've also used this combination
> on hundreds of servers, and several production systems.
> 

The 'if' and 'probably' are missing in your anecdotal evidence though.
If you use the servers the way you have, it will probably work fine.
Also we're talking about cloud instances, not "servers", which have
quite different use and performance profiles.

I'd like to see even some rudimentary experiments done with realistic
workloads before saying this is a better idea than leaving things as
they are. We've all speculated and provided anecdotal evidence enough to
warrant such an investigation for those who speculate it will be a
worthwhile change.

-- 
Ubuntu-cloud mailing list
Ubuntu-cloud@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-cloud

Re: [ubuntu-cloud] RFC on Cloud Images: Make /tmp a tmpfs

[Dustin Kirkland]

On Thu, Jan 14, 2016 at 12:05 AM, Seth Arnold  wrote:
> On Wed, Jan 13, 2016 at 11:00:16PM +0100, Martin Pitt wrote:
>> In a perfect world we'd have some clever tmpfs file system which would
>> use RAM as available and start overflowing onto a disk partition
>> (which could be LUKS with a random key) when necessary.. But even
>
> In fact this is what happens, 'unused' data from tmpfs heads to swap. So
> just configure swap space on your systems.

Moreover, just 'sudo apt-get install swapspace' and watch as swapfiles
are created/deleted as needed.  If your root disk is lvm-encrypted,
then obviously such swap files are encrypted, too.

-- 
Ubuntu-cloud mailing list
Ubuntu-cloud@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-cloud

Re: [ubuntu-cloud] RFC on Cloud Images: Make /tmp a tmpfs

[Dustin Kirkland]

On Thu, Jan 14, 2016 at 1:20 PM, Robie Basak  wrote:
> On Thu, Jan 14, 2016 at 12:27:58PM +0200, Dustin Kirkland wrote:
>> Moreover, just 'sudo apt-get install swapspace' and watch as swapfiles
>> are created/deleted as needed.  If your root disk is lvm-encrypted,
>> then obviously such swap files are encrypted, too.
>
> I didn't know about this package, thanks.

:-)  You're welcome.

> Should we MIR it and make it available (maybe even installed by default,
> whether enabled or disabled) on the cloud images at the same time? If
> this becomes our standard answer for "but /tmp runs out of space in
> tmpfs" then that feels appropriate to me.

+1!  MIR swapspace, add it to the images, and get rid of static swap
partitions, please :-)

-- 
Ubuntu-cloud mailing list
Ubuntu-cloud@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-cloud

Re: [ubuntu-cloud] RFC on Cloud Images: Make /tmp a tmpfs

[Clint Byrum]

Excerpts from Dustin Kirkland's message of 2016-01-14 02:27:58 -0800:
> On Thu, Jan 14, 2016 at 12:05 AM, Seth Arnold  
> wrote:
> > On Wed, Jan 13, 2016 at 11:00:16PM +0100, Martin Pitt wrote:
> >> In a perfect world we'd have some clever tmpfs file system which would
> >> use RAM as available and start overflowing onto a disk partition
> >> (which could be LUKS with a random key) when necessary.. But even
> >
> > In fact this is what happens, 'unused' data from tmpfs heads to swap. So
> > just configure swap space on your systems.
> 
> Moreover, just 'sudo apt-get install swapspace' and watch as swapfiles
> are created/deleted as needed.  If your root disk is lvm-encrypted,
> then obviously such swap files are encrypted, too.
> 

This is a neat idea. However, at that point, I'm not sure what this
actually gets you over just letting /tmp be on the filesystem, which will
have a much more limited effect. With a regular /tmp on a filesystem,
you mostly have non-unlinked named temporary files for the purpose of
sharing small amounts of data between processes. That use is definitely
a win for any tmpfs use case, as metadata for these is unlikely to get
paged out.  For the other typical use, you have one program spooling
to an unlinked file. This file will only go to disk when the number of
dirty buffers gets too high, and this flushing can be re-ordered for the
elevator, and will be heavily optimized by the VFS layer. IMO this is a
loss for tmpfs, because effectively you don't get through that optimized
code path, since swap files are never treated as a closed, unlinked file,
but instead a file which must be synced to immediately as soon as a block
has been written, and thus blows holes in the elevator queuing since it
has an immediate need.

Also, since swap cannot be on sparse files, I assume a number of files
must be created at install time, and if more are needed, large amounts
of 0-filled files must be created on the fly. Also, swapoff _must_ move
any used pages out of a swap space before removing it, causing that data
to be re-read, and likely soon thereafter re-written to disk _twice_
if the system is still low on available RAM at the time of swapoff.

Can anyone point to an actual measurement that actually proves the effects
over time when tmpfs is used for /tmp? It feels like we've reasoned
through a few things, but perhaps somebody has done experiments and
published a paper or two that we can refer to for details.

-- 
Ubuntu-cloud mailing list
Ubuntu-cloud@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-cloud

Re: [ubuntu-cloud] RFC on Cloud Images: Make /tmp a tmpfs

[Steve Langasek]

On Thu, Jan 14, 2016 at 03:54:30PM +0200, Dustin Kirkland wrote:
> > As a data point, I used to have my /tmp on tmpfs while I still had a
> > spinning disk, in order to address the power usage issues of disk flushing.
> > I found it to be a least-bad option which led to serious degradation of
> > desktop interactivity in the face of even moderate memory usage (at the
> > time, with 4GB RAM), and not because of excessive /tmp usage.

> > And as others in this thread have noted, this same problem can occur in
> > cloud instances.

> Definitely.  /tmp on tmpfs saves energy when you have a spinning HDD,
> and extends the life of your SSD by reducing the number of NAND flash
> writes!

Sorry, I seem to have not made my point clear.  I said this was a *least
bad* option when running on an HDD.  I ditched it soon after switching to an
SSD, because it had a horrible impact on desktop interactivity.  It might
extend the life of the SSD, but it was certainly shortening mine.

The correct fix for writing to /tmp causing excessive flushes to disk is to
*fix the thing that's causing flushing to disk*.  Trying to dodge by moving
/tmp into tmpfs just shuffles the problem around.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature
-- 
Ubuntu-cloud mailing list
Ubuntu-cloud@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-cloud

Re: [ubuntu-cloud] RFC on Cloud Images: Make /tmp a tmpfs

[Clint Byrum]

Excerpts from Ben Howard's message of 2016-01-13 04:26:13 -0800:
> All,
> 
> On the Ubuntu Cloud Images, we have a request to make /tmp a tmpfs. The
> rationale, from the bug:
>  * Performance - much faster read/write access to data in /tmp
>  * Security - sensitive data would be cleared from memory on boot,
>rather than written (leaked) to disk -- important for encryption
>scenarios
> 
> Since the Ubuntu Cloud Images are used by a wide number of users, I
> wanted to gather feedback and gather consensus on whether or not we
> should make this change.

The two arguments in the bug are:

a) It is more secure
b) It is faster

(a) is only half-correct. _IF_ the system has 0 swap configured, then
tmpfs will never be on disk. But if there is swap, then it will very
likely end up on disk. The answer to this is LUKS, not tmpfs.

(b) is overly general. For some things, this is absolutely true. For
others, the opposite is true. Some tmp usage will overrun the available
RAM and end up using swap directly, which does not have all of the tuning
of the VFS layer, and thus will perform in a much less predictable
manner. Plus, some programs use temporary files specifically because
they know they're about to do something that will be overly expensive
to guarantee RAM speed on, like sorting a file of arbitrary size. The
choice to make in-ram things slower is generally made when the system
owner chooses how much swap to make available. So making those operations
which the developer has chosen to allow to be spooled onto disk faster,
isn't actually a big win. However, failing those operations because of
a lack of space in /tmp is quite frustrating and could be very costly if
they must be retried.

So, the current paradigm is actually the more conservative, more
generally acceptable approach. Making /tmp a tmpfs is an optimization
_for some use cases_, which I think should be fine, but doesn't seem
worth taking a risk with all cloud image use cases.

-- 
Ubuntu-cloud mailing list
Ubuntu-cloud@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-cloud

Re: [ubuntu-cloud] RFC on Cloud Images: Make /tmp a tmpfs

[Robert Collins]

On 14 January 2016 at 12:05, Clint Byrum  wrote:
> Excerpts from Ben Howard's message of 2016-01-13 04:26:13 -0800:
>> All,
>>
>> On the Ubuntu Cloud Images, we have a request to make /tmp a tmpfs. The
>> rationale, from the bug:
>>  * Performance - much faster read/write access to data in /tmp
>>  * Security - sensitive data would be cleared from memory on boot,
>>rather than written (leaked) to disk -- important for encryption
>>scenarios
>>
>> Since the Ubuntu Cloud Images are used by a wide number of users, I
>> wanted to gather feedback and gather consensus on whether or not we
>> should make this change.
>
> The two arguments in the bug are:
>
> a) It is more secure
> b) It is faster
>
> (a) is only half-correct. _IF_ the system has 0 swap configured, then
> tmpfs will never be on disk. But if there is swap, then it will very
> likely end up on disk. The answer to this is LUKS, not tmpfs.

To add more confusion, this is for VMs and container workloads. The
host machines may suspend VM's or swap container contents out at any
time without regard to the configuration of the container/VM contents
itself.
..
> So, the current paradigm is actually the more conservative, more
> generally acceptable approach. Making /tmp a tmpfs is an optimization
> _for some use cases_, which I think should be fine, but doesn't seem
> worth taking a risk with all cloud image use cases.

There is a view that apps that need stuff in memory should just, well,
use memory. So yeah - lets make this configurable, but not default i.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

-- 
Ubuntu-cloud mailing list
Ubuntu-cloud@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-cloud

Re: [ubuntu-cloud] RFC on Cloud Images: Make /tmp a tmpfs

[Steve Langasek]

On Wed, Jan 13, 2016 at 11:00:16PM +0100, Martin Pitt wrote:
> Ben Howard [2016-01-13 14:26 +0200]:
> > On the Ubuntu Cloud Images, we have a request to make /tmp a tmpfs. The
> > rationale, from the bug:
> >  * Performance - much faster read/write access to data in /tmp
> >  * Security - sensitive data would be cleared from memory on boot,
> >rather than written (leaked) to disk -- important for encryption
> >scenarios

> > Since the Ubuntu Cloud Images are used by a wide number of users, I
> > wanted to gather feedback and gather consensus on whether or not we
> > should make this change.

> I really wish we would do this in general for new installs, at least
> as the first thing after releasing 16.04 LTS. I also do this on my
> boxes, not only for the reasons above [1], but also because it is much
> more power efficient -- as I literally work in /tmp a lot of my time
> the disk doesn't need to spin up often.

> The main reason AFAIK why we didn't yet do that was the concern that
> there is some broken software out there which potentially dumps really
> large files into /tmp (yes firefox, I'm looking at YOU!). These would
> need to be fixed to go to /var/tmp. This is a chicken-and-egg problem,
> though: We won't find out what's broken until we actually enable it on
> real-life installations. This problem applies to cloud image use cases
> just as much as desktop or "classic" servers.

> My gut feeling is that we should do it if there is ≥ 4 GB RAM, so that
> /tmp as at least 2 GB of space (That should be a rather simple
> installer/cloud-init decision?). We don't want to do this on small
> embedded devices with 512 MB of RAM or so, but there is absolutely no
> reason to not do it on beefy servers or laptops.

As a data point, I used to have my /tmp on tmpfs while I still had a
spinning disk, in order to address the power usage issues of disk flushing.
I found it to be a least-bad option which led to serious degradation of
desktop interactivity in the face of even moderate memory usage (at the
time, with 4GB RAM), and not because of excessive /tmp usage.

And as others in this thread have noted, this same problem can occur in
cloud instances.

-- 
Steve Langasek   Give me a lever long enough and a Free OS
Debian Developer   to set it on, and I can move the world.
Ubuntu Developerhttp://www.debian.org/
slanga...@ubuntu.com vor...@debian.org


signature.asc
Description: Digital signature
-- 
Ubuntu-cloud mailing list
Ubuntu-cloud@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/ubuntu-cloud