On Sun, Jun 20, 2010 at 11:41 PM, David Mertens <[email protected]>wrote:
> On Sun, Jun 20, 2010 at 7:51 PM, Chris Marshall <[email protected]> wrote: > >> On 6/20/2010 8:42 PM, P Kishor wrote: >> > >> > Figured it out. Stare at it hard and long enough, and sense begins to >> > emerge. I was getting confused by the use of ':' for the whole >> > dimension, but am now getting into the rhythm of the slice syntax. It >> > is not that bad, and it certainly doesn't need a nicer slicing (at >> > least until I run into confusion with it). >> >> Hi Puneet- >> >> I don't know what your solution was but you can use help 'vars' in >> the pdl2 shell (or just help vars in perldl) to see the dimensions >> of all your piddles. That may help sort things out. >> >> >> Btw, FlexRaw write is very fast. However, FlexRaw read is still slow. >> >> Faster than Storable, but still appreciably slow before it responds. >> >> I think that may be your laptop and memory issues. The IO is >> most likely comparable in speed (actually reading a hard disk >> is faster than writing it). However, on write, you already have >> all your data in memory. On read, you'll need to swap out >> enough things in memory to make room for a large data set. >> That could explain your start-up times. >> >> >> > By the way, the above still holds. writeflex is very fast, and even a >> > very large piddle is written out to the disk quickly (in spite of the >> > errors that I get in Core.pm). But, reading that piddle back from the >> > disk is slow, certainly not fast enough to be part of a program that >> > would want to load several such piddles in rapid fire succession. >> >> You might try the memmap options. Perhaps that will work around >> your swap issues. >> > > Actually, if you're after speed and you have only a handful of large > piddles, you could try using FastRaw. The differences between FastRaw and > FlexRaw is that FastRaw can only store one piddle per file and FlexRaw > handles both Fortran raw writes as well as C raw writes. This makes > FlexRaw's file-reading logic much more complicated than FastRaw's logic. > > FlexRaw and FastRaw both default to using C's data writing. This means that > data written to a new file using FlexRaw can be read by FastRaw, and > vice-versa. To see if FlexRaw's logic is causing the slowdown, try reading > your piddle (if it's one piddle per file) using FastRaw. > > David > > -- > Sent via my carrier pigeon. > Also, FastRaw's memory mapping should not suffer from the problems that FlexRaw's memory mapping exhibits, I think. -- Sent via my carrier pigeon.
_______________________________________________ Perldl mailing list [email protected] http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
