Re: [gentoo-user] aligning SSD partitions

2012-09-06 Thread Dale
Neil Bothwick wrote:
 On Wed, 05 Sep 2012 12:54:51 -0500, Dale wrote:

 I might also add, I see no speed improvements in putting portages
 work directory on tmpfs.  I have tested this a few times and the
 difference in compile times is just not there.  
 Probably because with 16GB everything stays cached anyway.  
 I cleared the cache between the compiles.  This is the command I
 use:

 echo 3  /proc/sys/vm/drop_caches  
 But you are still using the RAM as disk cache during the emerge, the
 data doesn't stay around long enough to need to get written to disk
 with so much RAM for cache.  
 Indeed. Try setting the mount to write-through to see the difference.
 When I run that command, it clears all the cache.  It is the same as if
 I rebooted.  Certainly you are not thinking that cache survives a
 reboot?
 You clear the cache between the two emerge runs, not during them.

If I recall correctly the process, I cleared the cache, ran emerge with
the portage work directory on disk.  Then cleared cache again and run on
tmpfs.  If you think that the cache would make any difference for the
second run then it would be faster just because of that *while using
tmpfs* since that was the second run.  Thing is, it wasn't faster.  In
some tests it was actually slower. 

I'm trying to catch on to why you think that clearing the cache means it
is still there.  That's the whole point of clearing cache is that it is
gone.  When I was checking on drop_caches, my understanding was that
clearing that was the same as a reboot.  This is from kernel.org

drop_caches

Writing to this will cause the kernel to drop clean caches, dentries and
inodes from memory, causing that memory to become free.

To free pagecache:
echo 1  /proc/sys/vm/drop_caches
To free dentries and inodes:
echo 2  /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes:
echo 3  /proc/sys/vm/drop_caches

According to that, the 3 option clears all cache.  One site I found that
on even recommended running sync first just in case something in ram was
not yet written to disk. 


 If you are talking about ram on the drive itself, well, when it is on
 tmpfs, it is not on the drive to be cached.  That's the whole point of
 tmpfs is to get the slow drive out of the way.  By the way, there are
 others that ran tests with the same results.  It just doesn't speed up
 anything since drives are so much faster nowadays. 
 Drives are still orders of magnitude slower than RAM, that's why using
 swap is so slow. What appears to be happening here is that because
 files are written and then read again in short succession, they are still
 in the kernel's disk cache, so the speed of the disk is irrelevant. Bear
 in mind that tmpfs is basically a cached disk without the disk, so you
 are effectively comparing the same thing twice.



I agree that they are slower but for whatever reason, it doesn't seem to
matter as much as one thought.  I was expecting a huge difference in
this with using tmpfs being much faster.  Thing is, that is NOT what I
got.  Theory meets reality and it was not what I expected or what others
expected either.  Putting portages work directory on tmpfs doesn't make
much if any difference in emerge times. 

Dale

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how 
you interpreted my words!




Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Neil Bothwick
On Tue, 4 Sep 2012 20:42:56 -0400, Philip Webb wrote:

 What is the best line for  /etc/fstab ?  The only example I have is :
 
   'tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0'
 
 This doesn't seem to limit the size in any way.

man mount explains it all, but the option you want is size, which
defaults to 50%. I use 80% which is what gives the somewhat odd size of
13GB. This is based on physical RAM, but tmpfs will use swap if there is
not enough memory.


-- 
Neil Bothwick

Angular Momentum Makes The World Go 'Round


signature.asc
Description: PGP signature


Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Philip Webb
120905 Neil Bothwick wrote:
 On Tue, 4 Sep 2012 20:42:56 -0400, Philip Webb wrote:
 What is the best line for  /etc/fstab ?  The only example I have is :
   'tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0'
 This doesn't seem to limit the size in any way.
 'man mount' explains it all ...

Well, it outlines it (smile).

 ... but the option you want is size, which defaults to 50 % .

That looks ok : I assume that's the maximum,
ie it doesn't take up that much memory unless it's needed.

 I use 80 % , which is what gives the somewhat odd size of 13 GB .
  tmpfs  will use swap if there is not enough memory.

Thanks for the advice + others' comments.

BTW the problem of System Rescue stalling while trying to install itself
is avoided by adding the boot option 'skipmount=/dev/sda3' (or as needed).

-- 
,,
SUPPORT ___//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT`-O--O---'   purslowatchassdotutorontodotca




Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Peter Humphrey
On Wednesday 05 September 2012 10:02:49 Philip Webb wrote:
 120905 Neil Bothwick wrote:
  On Tue, 4 Sep 2012 20:42:56 -0400, Philip Webb wrote:
  What is the best line for  /etc/fstab ?  The only example I have is 
:
'tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0'
  
  This doesn't seem to limit the size in any way.
  
  'man mount' explains it all ...
 
 Well, it outlines it (smile).
 
  ... but the option you want is size, which defaults to 50 % .
 
 That looks ok : I assume that's the maximum,
 ie it doesn't take up that much memory unless it's needed.

The kernel only uses as much tmpfs as it needs at any given time. If it 
needs more than has been specified, it starts rolling less active parts 
out to swap. So if you don't have a lot of memory, you can still specify 
more tmpfs than you have RAM and everything will just work.

The only reason I specify a large tmpfs is to be able to compile Libre 
Office. At other times it just isn't used.

-- 
Rgds
Peter



Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Neil Bothwick
On Wed, 5 Sep 2012 05:02:49 -0400, Philip Webb wrote:

  'man mount' explains it all ...  
 
 Well, it outlines it (smile).

:-)

I'll rephrase that:

'man mount' explains it all, for small values of all.

  ... but the option you want is size, which defaults to 50 % .  
 
 That looks ok : I assume that's the maximum,
 ie it doesn't take up that much memory unless it's needed.

Exactly, it uses what it needs, size sets the upper limit.


-- 
Neil Bothwick

With 7 billion people on earth chances are slim it will ever be *your*
day.


signature.asc
Description: PGP signature


Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Dale
Peter Humphrey wrote:
 On Wednesday 05 September 2012 10:02:49 Philip Webb wrote:
 120905 Neil Bothwick wrote:
 On Tue, 4 Sep 2012 20:42:56 -0400, Philip Webb wrote:
 What is the best line for  /etc/fstab ?  The only example I have is 
 :
   'tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0'

 This doesn't seem to limit the size in any way.
 'man mount' explains it all ...
 Well, it outlines it (smile).

 ... but the option you want is size, which defaults to 50 % .
 That looks ok : I assume that's the maximum,
 ie it doesn't take up that much memory unless it's needed.
 The kernel only uses as much tmpfs as it needs at any given time. If it 
 needs more than has been specified, it starts rolling less active parts 
 out to swap. So if you don't have a lot of memory, you can still specify 
 more tmpfs than you have RAM and everything will just work.

 The only reason I specify a large tmpfs is to be able to compile Libre 
 Office. At other times it just isn't used.



I let mine default to half of ram but when I need to compile LOo, then I
have to manually increase it.  I guess 8Gbs isn't enough.  Come to think
of it, I updated LOo the other day and it didn't complain about it being
less than 8Gbs.  I guess it doesn't need as much as it used to.  Code
clean up??

I might also add, I see no speed improvements in putting portages work
directory on tmpfs.  I have tested this a few times and the difference
in compile times is just not there. 

Dale

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how 
you interpreted my words!




Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Peter Humphrey
On Wednesday 05 September 2012 12:07:13 Dale wrote:

 I might also add, I see no speed improvements in putting portages
 work directory on tmpfs.  I have tested this a few times and the
 difference in compile times is just not there.

Yes, I'd forgotten that. I just haven't got round to changing it. It 
works, so I haven't fixed it.  :-)

-- 
Rgds
Peter



Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Dale
Peter Humphrey wrote:
 On Wednesday 05 September 2012 12:07:13 Dale wrote:

 I might also add, I see no speed improvements in putting portages
 work directory on tmpfs.  I have tested this a few times and the
 difference in compile times is just not there.
 Yes, I'd forgotten that. I just haven't got round to changing it. It 
 works, so I haven't fixed it.  :-)


I leave mine on too.  I figure the ram will not suffer wear and tear
like a hard drive will.  Compiling is a good bit of writing, deleting
and such and I just think doing that in ram, since it is temporary
anyway, is a better deal in the long run.  I would think that would be a
good idea when using a SSD. 

Off topic a bit.  I been using ext4 for my file system since I moved
drives around.  It has a defrag tool that is just plain wonderful.  For
anyone that uses ext4, try the command e4defrag.  It's simple to use. 
To check it, e4defrag -c device, mount point or even a specific file or
directory.  If you want it to actually defrag it, just remove the -c
option.  I find that after a big update, like KDE, it helps to defrag
/usr.  I defrag my /home on occasion too.  I'm still downloading a LOT
of TV shows and movies.  That goodness for LVM.

LOL 

Dale

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how 
you interpreted my words!




Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Neil Bothwick
On Wed, 05 Sep 2012 06:07:13 -0500, Dale wrote:

 I might also add, I see no speed improvements in putting portages work
 directory on tmpfs.  I have tested this a few times and the difference
 in compile times is just not there. 

Probably because with 16GB everything stays cached anyway.


-- 
Neil Bothwick

If the cops arrest a mime, do they tell her she has the right to remain
silent?


signature.asc
Description: PGP signature


Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Dale
Neil Bothwick wrote:
 On Wed, 05 Sep 2012 06:07:13 -0500, Dale wrote:

 I might also add, I see no speed improvements in putting portages work
 directory on tmpfs.  I have tested this a few times and the difference
 in compile times is just not there. 
 Probably because with 16GB everything stays cached anyway.




I cleared the cache between the compiles.  This is the command I use:

echo 3  /proc/sys/vm/drop_caches

That clears all cache from what I read and it also shows up as cleared
in top too.  It's amazing how much cache can build up over time.  Right
now, just about all my ram, 16Gbs worth, is used.  I check it with
gkrellm.  It seems no matter how much ram you have, it will use it for
something. 

Still, no difference.  There is also a thread on the forums on this
too.  I think I posted my results on there but it was a while back.  I
think I ran that while in single user mode too. 

Dale

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how 
you interpreted my words!




Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Adam Carter
 I might also add, I see no speed improvements in putting portages work
 directory on tmpfs.  I have tested this a few times and the difference
 in compile times is just not there.

 Probably because with 16GB everything stays cached anyway.

Would it still be useful to use tmpfs if you wanted to keep drive
writes down on an SSD? I imagine even tho its cached it would get
written to disk after a timer expires, so while it doesn't affect
performance it does reduce the drive life.



Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Peter Humphrey
On Wednesday 05 September 2012 13:02:01 Dale wrote:

 I find that after a big update, like KDE, it helps to defrag /usr.

Interesting. I've just run sudo e4defrag -c /usr and got a fragmentation 
of zero. That's after upgrading KDE last week.

Then I ran it on all the nine ext4 partitions here and only two had 
nonzero fragmentations; one was 1 and the other 2.

Looks like I can forget about it on this box.

-- 
Rgds
Peter



Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Neil Bothwick
On Wed, 05 Sep 2012 07:52:45 -0500, Dale wrote:

  I might also add, I see no speed improvements in putting portages
  work directory on tmpfs.  I have tested this a few times and the
  difference in compile times is just not there.   
  Probably because with 16GB everything stays cached anyway.

 I cleared the cache between the compiles.  This is the command I use:
 
 echo 3  /proc/sys/vm/drop_caches

But you are still using the RAM as disk cache during the emerge, the data
doesn't stay around long enough to need to get written to disk with so
much RAM for cache.


-- 
Neil Bothwick

There are no stupid questions, just too many inquisitive idiots.


signature.asc
Description: PGP signature


Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Michael Mol
On Wed, Sep 5, 2012 at 11:17 AM, Neil Bothwick n...@digimed.co.uk wrote:
 On Wed, 05 Sep 2012 07:52:45 -0500, Dale wrote:

  I might also add, I see no speed improvements in putting portages
  work directory on tmpfs.  I have tested this a few times and the
  difference in compile times is just not there.
  Probably because with 16GB everything stays cached anyway.

 I cleared the cache between the compiles.  This is the command I use:

 echo 3  /proc/sys/vm/drop_caches

 But you are still using the RAM as disk cache during the emerge, the data
 doesn't stay around long enough to need to get written to disk with so
 much RAM for cache.

Indeed. Try setting the mount to write-through to see the difference.


-- 
:wq



Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Florian Philipp
Am 05.09.2012 14:55, schrieb Adam Carter:
 I might also add, I see no speed improvements in putting portages work
 directory on tmpfs.  I have tested this a few times and the difference
 in compile times is just not there.

 Probably because with 16GB everything stays cached anyway.
 
 Would it still be useful to use tmpfs if you wanted to keep drive
 writes down on an SSD? I imagine even tho its cached it would get
 written to disk after a timer expires, so while it doesn't affect
 performance it does reduce the drive life.
 

Yes. For ext{3,4}, this timer is controlled by the commit=x mount flag
(default: 5 seconds).

IIRC, app-laptop/laptop-mode-tools sets this to 30 so I guess it is safe
to use large values (besides the obvious risk of loosing data on system
crashes).

Regards,
Florian Philipp



signature.asc
Description: OpenPGP digital signature


Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Volker Armin Hemmann
Am Mittwoch, 5. September 2012, 09:23:58 schrieb Neil Bothwick:
 On Tue, 4 Sep 2012 20:42:56 -0400, Philip Webb wrote:
  What is the best line for  /etc/fstab ?  The only example I have is :
'tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0'
  
  This doesn't seem to limit the size in any way.
 
 man mount explains it all, but the option you want is size, which
 defaults to 50%. I use 80% which is what gives the somewhat odd size of
 13GB. This is based on physical RAM, but tmpfs will use swap if there is
 not enough memory.

swap destroys your performance and interactivity so throughly - that is 
something you really want to avoid.

-- 
#163933



Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Neil Bothwick
On Wed, 05 Sep 2012 18:30:24 +0200, Volker Armin Hemmann wrote:

  man mount explains it all, but the option you want is size, which
  defaults to 50%. I use 80% which is what gives the somewhat odd size
  of 13GB. This is based on physical RAM, but tmpfs will use swap if
  there is not enough memory.  
 
 swap destroys your performance and interactivity so throughly - that is 
 something you really want to avoid.

Absolutely, but it beats the hell out of an OOM. I prefer to have a
system with plenty of swap and using none of it.


-- 
Neil Bothwick

The quickest way to a man's heart is through his sternum.


signature.asc
Description: PGP signature


Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Dale
Michael Mol wrote:
 On Wed, Sep 5, 2012 at 11:17 AM, Neil Bothwick n...@digimed.co.uk wrote:
 On Wed, 05 Sep 2012 07:52:45 -0500, Dale wrote:

 I might also add, I see no speed improvements in putting portages
 work directory on tmpfs.  I have tested this a few times and the
 difference in compile times is just not there.
 Probably because with 16GB everything stays cached anyway.
 I cleared the cache between the compiles.  This is the command I use:

 echo 3  /proc/sys/vm/drop_caches
 But you are still using the RAM as disk cache during the emerge, the data
 doesn't stay around long enough to need to get written to disk with so
 much RAM for cache.
 Indeed. Try setting the mount to write-through to see the difference.



When I run that command, it clears all the cache.  It is the same as if
I rebooted.  Certainly you are not thinking that cache survives a reboot?

If you are talking about ram on the drive itself, well, when it is on
tmpfs, it is not on the drive to be cached.  That's the whole point of
tmpfs is to get the slow drive out of the way.  By the way, there are
others that ran tests with the same results.  It just doesn't speed up
anything since drives are so much faster nowadays. 

Dale

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how 
you interpreted my words!




Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Dale
Peter Humphrey wrote:
 On Wednesday 05 September 2012 13:02:01 Dale wrote:

 I find that after a big update, like KDE, it helps to defrag /usr.
 Interesting. I've just run sudo e4defrag -c /usr and got a fragmentation 
 of zero. That's after upgrading KDE last week.

 Then I ran it on all the nine ext4 partitions here and only two had 
 nonzero fragmentations; one was 1 and the other 2.

 Looks like I can forget about it on this box.


I have to say that here, it is not a whole lot of fragmentation but it
does seem a bit faster afterwards.  I guess it depends on what is
fragmented and such.  I sometimes wonder if it defrags itself.  Even
when I watch the fsck when booting, all the ext4 partitions have a very
small percentage of fragmentation.  My /boot which is ext2 is fragmented
as heck.  lol  I'm not worried about it tho.  ;-)  When I was using
reiserfs, it was always a good bit of fragmentation. 

Just thought it was worth a mention since this is the first time I saw a
Linux defrag tool. 

Dale

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how 
you interpreted my words!




Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Paul Hartman
On Wed, Sep 5, 2012 at 1:02 PM, Dale rdalek1...@gmail.com wrote:
 Peter Humphrey wrote:
 On Wednesday 05 September 2012 13:02:01 Dale wrote:

 I find that after a big update, like KDE, it helps to defrag /usr.
 Interesting. I've just run sudo e4defrag -c /usr and got a fragmentation
 of zero. That's after upgrading KDE last week.

 Then I ran it on all the nine ext4 partitions here and only two had
 nonzero fragmentations; one was 1 and the other 2.

 Looks like I can forget about it on this box.


 I have to say that here, it is not a whole lot of fragmentation but it
 does seem a bit faster afterwards.  I guess it depends on what is
 fragmented and such.  I sometimes wonder if it defrags itself.  Even
 when I watch the fsck when booting, all the ext4 partitions have a very
 small percentage of fragmentation.  My /boot which is ext2 is fragmented
 as heck.  lol  I'm not worried about it tho.  ;-)  When I was using
 reiserfs, it was always a good bit of fragmentation.

 Just thought it was worth a mention since this is the first time I saw a
 Linux defrag tool.

I think almost all linux defrag tools/techniques deal with file
fragmentation only, that is to say one file with more than 1 extent,
but don't deal with filesystem fragmentation (1 small files
scattered all over the drive, rather than written contiguously). So
I'm not surprised that Peter did not see fragmentation after
installing KDE.

AFAIK almost all that modern defrag tools do is just copy the file,
allocating the whole file at once in the copy process, and if that new
copy has fewer extents than the old copy, it fills in the data, then
removes the original file. The concept is not entirely dissimilar to
the old backup, format, restore defrag process.

Over the years I have used a poor-man's version of that concept to
defrag files. Just move it to another drive (or -- even better -- a
ramdrive/tmpfs), then move it back to disk (with a tool that performs
preallocation).

There is a userland defrag tool that does exactly this, on any
filesystem. It is called shake.

Typically I only see fragmentation on large files that were copied
from a slow source (over the network/internet), or bittorrent clients
that do not preallocate space, etc. Any kind of streaming file that
was written, huge multi-gigabyte video recording files, that kind of
stuff. But the key to avoiding file fragmentation is preallocation...



Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Dale
Paul Hartman wrote:
 On Wed, Sep 5, 2012 at 1:02 PM, Dale rdalek1...@gmail.com wrote:

 I have to say that here, it is not a whole lot of fragmentation but it
 does seem a bit faster afterwards.  I guess it depends on what is
 fragmented and such.  I sometimes wonder if it defrags itself.  Even
 when I watch the fsck when booting, all the ext4 partitions have a very
 small percentage of fragmentation.  My /boot which is ext2 is fragmented
 as heck.  lol  I'm not worried about it tho.  ;-)  When I was using
 reiserfs, it was always a good bit of fragmentation.

 Just thought it was worth a mention since this is the first time I saw a
 Linux defrag tool.
 I think almost all linux defrag tools/techniques deal with file
 fragmentation only, that is to say one file with more than 1 extent,
 but don't deal with filesystem fragmentation (1 small files
 scattered all over the drive, rather than written contiguously). So
 I'm not surprised that Peter did not see fragmentation after
 installing KDE.

 AFAIK almost all that modern defrag tools do is just copy the file,
 allocating the whole file at once in the copy process, and if that new
 copy has fewer extents than the old copy, it fills in the data, then
 removes the original file. The concept is not entirely dissimilar to
 the old backup, format, restore defrag process.

 Over the years I have used a poor-man's version of that concept to
 defrag files. Just move it to another drive (or -- even better -- a
 ramdrive/tmpfs), then move it back to disk (with a tool that performs
 preallocation).

 There is a userland defrag tool that does exactly this, on any
 filesystem. It is called shake.

 Typically I only see fragmentation on large files that were copied
 from a slow source (over the network/internet), or bittorrent clients
 that do not preallocate space, etc. Any kind of streaming file that
 was written, huge multi-gigabyte video recording files, that kind of
 stuff. But the key to avoiding file fragmentation is preallocation...



I used shake before but it just didn't seem to work right for me.  I
found a script that does something and it seems to work for the most
part but still not great or anything.  I just like the way ext4 works. 
Heck, I liked it before I found the defrag tool.  I've had this install
for a while and it has never had much fragmentation even before the
tool.  So, I find it funny that they make a tool that really isn't
needed very much.  :/

Dale

:-)  :-) 

-- 
I am only responsible for what I said ... Not for what you understood or how 
you interpreted my words!




Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Paul Hartman
On Wed, Sep 5, 2012 at 3:46 PM, Dale rdalek1...@gmail.com wrote:
 Paul Hartman wrote:
 On Wed, Sep 5, 2012 at 1:02 PM, Dale rdalek1...@gmail.com wrote:

 I have to say that here, it is not a whole lot of fragmentation but it
 does seem a bit faster afterwards.  I guess it depends on what is
 fragmented and such.  I sometimes wonder if it defrags itself.  Even
 when I watch the fsck when booting, all the ext4 partitions have a very
 small percentage of fragmentation.  My /boot which is ext2 is fragmented
 as heck.  lol  I'm not worried about it tho.  ;-)  When I was using
 reiserfs, it was always a good bit of fragmentation.

 Just thought it was worth a mention since this is the first time I saw a
 Linux defrag tool.
 I think almost all linux defrag tools/techniques deal with file
 fragmentation only, that is to say one file with more than 1 extent,
 but don't deal with filesystem fragmentation (1 small files
 scattered all over the drive, rather than written contiguously). So
 I'm not surprised that Peter did not see fragmentation after
 installing KDE.

 AFAIK almost all that modern defrag tools do is just copy the file,
 allocating the whole file at once in the copy process, and if that new
 copy has fewer extents than the old copy, it fills in the data, then
 removes the original file. The concept is not entirely dissimilar to
 the old backup, format, restore defrag process.

 Over the years I have used a poor-man's version of that concept to
 defrag files. Just move it to another drive (or -- even better -- a
 ramdrive/tmpfs), then move it back to disk (with a tool that performs
 preallocation).

 There is a userland defrag tool that does exactly this, on any
 filesystem. It is called shake.

 Typically I only see fragmentation on large files that were copied
 from a slow source (over the network/internet), or bittorrent clients
 that do not preallocate space, etc. Any kind of streaming file that
 was written, huge multi-gigabyte video recording files, that kind of
 stuff. But the key to avoiding file fragmentation is preallocation...



 I used shake before but it just didn't seem to work right for me.  I
 found a script that does something and it seems to work for the most
 part but still not great or anything.  I just like the way ext4 works.
 Heck, I liked it before I found the defrag tool.  I've had this install
 for a while and it has never had much fragmentation even before the
 tool.  So, I find it funny that they make a tool that really isn't
 needed very much.  :/

I think shake's default options might require extended attributes
enabled in your mounted fs. It also has some thresholds for file size
and age that cause it to skip certain files, unless you tell it
otherwise. I haven't used it in quite a long time, to be honest. :)



Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Peter Humphrey
On Wednesday 05 September 2012 21:46:59 Dale wrote:

 So, I find it funny that they make a tool that really isn't needed very
 much.  :/

Call it belt-and-braces if you like. (That's UK braces, which I think 
are US suspenders.)

-- 
Rgds
Peter



Re: [gentoo-user] aligning SSD partitions

2012-09-05 Thread Neil Bothwick
On Wed, 05 Sep 2012 12:54:51 -0500, Dale wrote:

  I might also add, I see no speed improvements in putting portages
  work directory on tmpfs.  I have tested this a few times and the
  difference in compile times is just not there.  
  Probably because with 16GB everything stays cached anyway.  
  I cleared the cache between the compiles.  This is the command I
  use:
 
  echo 3  /proc/sys/vm/drop_caches  
  But you are still using the RAM as disk cache during the emerge, the
  data doesn't stay around long enough to need to get written to disk
  with so much RAM for cache.  
  Indeed. Try setting the mount to write-through to see the difference.

 When I run that command, it clears all the cache.  It is the same as if
 I rebooted.  Certainly you are not thinking that cache survives a
 reboot?

You clear the cache between the two emerge runs, not during them.

 If you are talking about ram on the drive itself, well, when it is on
 tmpfs, it is not on the drive to be cached.  That's the whole point of
 tmpfs is to get the slow drive out of the way.  By the way, there are
 others that ran tests with the same results.  It just doesn't speed up
 anything since drives are so much faster nowadays. 

Drives are still orders of magnitude slower than RAM, that's why using
swap is so slow. What appears to be happening here is that because
files are written and then read again in short succession, they are still
in the kernel's disk cache, so the speed of the disk is irrelevant. Bear
in mind that tmpfs is basically a cached disk without the disk, so you
are effectively comparing the same thing twice.


-- 
Neil Bothwick

Theory is when you know everything, but nothing works.
Reality is when everything works, but you don't know why.
However, usually theory and reality are mixed together :
Nothing works, and nobody knows why not.


signature.asc
Description: PGP signature


Re: [gentoo-user] aligning SSD partitions

2012-09-04 Thread Neil Bothwick
On Tue, 4 Sep 2012 03:20:03 -0400, Philip Webb wrote:

 I plan to partition + format the SSD in my new machine very soon.
 The Arch wiki has an article on the subject, which says
 that recent versions of Fdisk will safely align SSD partitions.
 
 Is this correct ?

Yes, but all of this was covered in some detail a few days ago.

 My partition scheme is
 
   1  boot 0,6   0,06  /boot
   2  root  30   3,55  / including opt usr var
   3  swap   4  -- swap
   5  home  30   3,3   /home
   6  portage   15   3,43  /usr/portage (distfiles 2,3)
   7  z 41   1,5   /z
  total121  19,45

If it's a new machine, use a GPT rather than DOS partition table.

I prefer to use a small ext2 filesystem for PORTDIR for speed and set
DISTDIR somewhere else (a directory in /z would make sense on your system
as the contents of DISTDIR are fairly temporary)

 I use  /z/tmp/  for Portage's temporary disk space.
 Other items, eg  /usr/local/  /usr/src/  wb on the HDD.
 
 I plan to put  /tmp/  on a ram disk.

If you have enough RAM, use tmpfs for /tmp and set PORTAGE_TMPDIR=/tmp.


-- 
Neil Bothwick

And then Adam said, What's a headache?


signature.asc
Description: PGP signature


Re: [gentoo-user] aligning SSD partitions

2012-09-04 Thread Volker Armin Hemmann
Am Dienstag, 4. September 2012, 08:47:12 schrieb Neil Bothwick:
 On Tue, 4 Sep 2012 03:20:03 -0400, Philip Webb wrote:
  I plan to partition + format the SSD in my new machine very soon.
  The Arch wiki has an article on the subject, which says
  that recent versions of Fdisk will safely align SSD partitions.
  
  Is this correct ?
 
 Yes, but all of this was covered in some detail a few days ago.
 
  My partition scheme is
  
1  boot 0,6   0,06  /boot
2  root  30   3,55  / including opt usr var
3  swap   4  -- swap
5  home  30   3,3   /home
6  portage   15   3,43  /usr/portage (distfiles 2,3)
7  z 41   1,5   /z

   total121  19,45
 
 If it's a new machine, use a GPT rather than DOS partition table.
 
 I prefer to use a small ext2 filesystem for PORTDIR for speed and set
 DISTDIR somewhere else (a directory in /z would make sense on your system
 as the contents of DISTDIR are fairly temporary)
 
  I use  /z/tmp/  for Portage's temporary disk space.
  Other items, eg  /usr/local/  /usr/src/  wb on the HDD.
  
  I plan to put  /tmp/  on a ram disk.
 
 If you have enough RAM, use tmpfs for /tmp and set PORTAGE_TMPDIR=/tmp.

hell no!

don't do that!

tmpfs for /tmp is fine
and
tmpfs for PORTAGE_TMPDIR is fine too. Like /var/tmp/portage.

But don't put PORTAGE_TMPDIR into /tmp. Not good. Bad idea. Really. 

IF PORTAGE_TMPDIR fills up - no biggy, emerge dies, that's it. But /tmp filled 
up? Suddenly you will have lots of strange problems... don't do it. Spare 
yourself some headaches.

-- 
#163933



Re: [gentoo-user] aligning SSD partitions

2012-09-04 Thread Neil Bothwick
On Tue, 04 Sep 2012 22:31:23 +0200, Volker Armin Hemmann wrote:

 IF PORTAGE_TMPDIR fills up - no biggy, emerge dies, that's it. But /tmp
 filled up? Suddenly you will have lots of strange problems... don't do
 it. Spare yourself some headaches.

Good point.


-- 
Neil Bothwick

Q: What's the proper plural of a 'Net-connected Windows machine?
A: A Botnet


signature.asc
Description: PGP signature


Re: [gentoo-user] aligning SSD partitions

2012-09-04 Thread Neil Bothwick
On Tue, 04 Sep 2012 22:31:23 +0200, Volker Armin Hemmann wrote:

 IF PORTAGE_TMPDIR fills up - no biggy, emerge dies, that's it. But /tmp
 filled up? Suddenly you will have lots of strange problems... don't do
 it. Spare yourself some headaches.

Good point, maybe I should have mentioned I have a 13GB /tmp.


-- 
Neil Bothwick

When cryptography is outlawed, bayl bhgynjf jvyy unir cevinpl.


signature.asc
Description: PGP signature


Re: [gentoo-user] aligning SSD partitions

2012-09-04 Thread Peter Humphrey
On Tuesday 04 September 2012 22:00:48 Neil Bothwick wrote:
 On Tue, 04 Sep 2012 22:31:23 +0200, Volker Armin Hemmann wrote:
  IF PORTAGE_TMPDIR fills up - no biggy, emerge dies, that's it. But
  /tmp filled up? Suddenly you will have lots of strange problems...
  don't do it. Spare yourself some headaches.
 
 Good point, maybe I should have mentioned I have a 13GB /tmp.

Interesting. Also having 16GB RAM I've limited /tmp to 10GB. I wonder 
whether 13GB would offer any advantage. Unlikely, as the only time it 
gets used in earnest is when compiling Firefox, OO and the like. Maybe I 
should just remove the restriction and let the kernel optimise its own 
use of swap and tmpfs.

This box spends well over 90% of its cycles on BOINC projects, which 
crunch large numbers of numbers but don't take up a lot of space (by 
modern standards).

-- 
Rgds
Peter



Re: [gentoo-user] aligning SSD partitions

2012-09-04 Thread Philip Webb
120904 Peter Humphrey wrote:
 On Tuesday 04 September 2012 22:00:48 Neil Bothwick wrote:
 On Tue, 04 Sep 2012 22:31:23 +0200, Volker Armin Hemmann wrote:
 If PORTAGE_TMPDIR fills up no biggy, emerge dies, that's it.
 But /tmp filled up? Suddenly you will have lots of strange problems.
 Don't do it. Spare yourself some headaches.
 Good point, maybe I should have mentioned I have a 13GB /tmp.

At the moment, after a few hours catching up with the news with FF,
my memory usage is :

   total used free shared buffers cached
  Mem:  3960  915 3044  0  63408
  -/+ buffers/cache:  443 3516
  Swap: 38200 3820

Even compiling LO, it doesn't spill into swap anymore.
I assume having PORTAGE_TMPDIR on SSD wb noticeably faster than on HDD,
but how much faster still would it be to have it in memory ?
Memory is cheap  I could buy another  4 GB , it there were a reason.

 Also having 16GB RAM I've limited /tmp to 10GB.
 I wonder whether 13GB would offer any advantage.
 Unlikely, as the only time it gets used in earnest
 is when compiling Firefox, OO and the like.
 Maybe I should just remove the restriction
 and let the kernel optimise its own use of swap and tmpfs.
 This box spends well over 90% of its cycles on BOINC projects,
 which crunch large numbers of numbers but don't take up a lot of space.

What is the best line for  /etc/fstab ?  The only example I have is :

  'tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0'

This doesn't seem to limit the size in any way.

-- 
,,
SUPPORT ___//___,   Philip Webb
ELECTRIC   /] [] [] [] [] []|   Cities Centre, University of Toronto
TRANSIT`-O--O---'   purslowatchassdotutorontodotca