Eric Iverson-3 wrote:
>
> What matters is the ratio of the size of RAM to the size of active data on
> the page file AND the nature of the access. Popluating a huge array with
> i.x
> where x is much larger that RAM size will page but the thrash won't be too
> bad because of the sequential nature of the disk access. Totally random
> access to bytes across an array significantly larger that RAM will thrash
> to
> a near standstill and would soon encourage another solution to the
> problem.
>
After all I wrote about this in other posts one would think
that I'd agree with you, but I think that i.x is just as prone
to page thrashing as anything.
Let me demonstrate on a 32bit GNU/Linux (aka Debian) computer
with 1,5GB RAM and about 2.7GB swap (see below for the exact figure):
$ ulimit -v # usually irrelevant on desktop computers
unlimited
NO SWAPPING: ======================================
$ /usr/bin/time -f "%e sec, %R minor %F major swaps" ~/j602/bin/jconsole
<<END
> i=:i.5000000x
> i=:>:i
> END
12.84 sec, 187181 minor 0 major swaps
typical behavior of top command:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4352 ******** 25 0 699m 670m 1304 R 99.0 44.1 0:07.21 jconsole
SOME MAJOR SWAPPING, BUT SCALABLE: =======================
$ /usr/bin/time -f "%e sec, %R minor %F major swaps" ~/j602/bin/jconsole
<<END
> i=:i.10000000x
> i=:>:i
> END
26.44 sec, 372846 minor 31 major swaps
Timing goes up linearly with array size as expected.
Few major swaps do not change this fact appreciably.
Typical output of top command:
Mem: 1556892k total, 1416748k used, 140144k free, 980k buffers
Swap: 2650684k total, 42200k used, 2608484k free, 16612k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4356 ******** 25 0 1393m 1.3g 1304 R 99.9 88.1 0:15.05 jconsole
THRASHING: =========================================
$ /usr/bin/time -f "%e sec, %R minor %F major swaps" ~/j602/bin/jconsole
<<END
> i=:i.20000000x
> i=:>:i
> END
Command terminated by signal 15
1654.95 sec, 1710494 minor 238198 major swaps
Couldn't wait any longer... Compuer is using disk as if it were RAM.
Typical output of top command:
Mem: 1556892k total, 1525440k used, 31452k free, 104k buffers
Swap: 2650684k total, 1483240k used, 1167444k free, 1760k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
126 root 10 -5 0 0 0 D 9.2 0.0 2:31.54 kswapd0
4406 ****** 18 0 2781m 1.4g 176 D 1.5 95.5 0:43.97 jconsole
CPU is idle, all of the time is spent swapping, and we didn't even use
up all the swap space.
MAJOR SWAPPING WITHOUT THRASHING: =====================
$ /usr/bin/time -f "%e sec, %R minor %F major swaps" ~/j602/bin/jconsole
<<END
> i=:i.10000000x
> i=:>:i
> j=:i.10000000x NB. Swapping should take place here.
> j=:j+10000000x
> END
171.29 sec, 860934 minor 25603 major swaps
It's more than 3 times as slow as it would have been with
a sufficient amount of RAM, but at least it works, of course
with a major hassle with partitioning of data according
to the size of available RAM.
--
View this message in context:
http://www.nabble.com/Large-mapped-files-and-VAS-again-tp20870863s24193p20928040.html
Sent from the J Programming mailing list archive at Nabble.com.
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
