On Sun, Jun 20, 2010 at 7:51 PM, Chris Marshall <[email protected]> wrote:
> On 6/20/2010 8:42 PM, P Kishor wrote: > > > > Figured it out. Stare at it hard and long enough, and sense begins to > > emerge. I was getting confused by the use of ':' for the whole > > dimension, but am now getting into the rhythm of the slice syntax. It > > is not that bad, and it certainly doesn't need a nicer slicing (at > > least until I run into confusion with it). > > Hi Puneet- > > I don't know what your solution was but you can use help 'vars' in > the pdl2 shell (or just help vars in perldl) to see the dimensions > of all your piddles. That may help sort things out. > > >> Btw, FlexRaw write is very fast. However, FlexRaw read is still slow. > >> Faster than Storable, but still appreciably slow before it responds. > > I think that may be your laptop and memory issues. The IO is > most likely comparable in speed (actually reading a hard disk > is faster than writing it). However, on write, you already have > all your data in memory. On read, you'll need to swap out > enough things in memory to make room for a large data set. > That could explain your start-up times. > > > > By the way, the above still holds. writeflex is very fast, and even a > > very large piddle is written out to the disk quickly (in spite of the > > errors that I get in Core.pm). But, reading that piddle back from the > > disk is slow, certainly not fast enough to be part of a program that > > would want to load several such piddles in rapid fire succession. > > You might try the memmap options. Perhaps that will work around > your swap issues. > Actually, if you're after speed and you have only a handful of large piddles, you could try using FastRaw. The differences between FastRaw and FlexRaw is that FastRaw can only store one piddle per file and FlexRaw handles both Fortran raw writes as well as C raw writes. This makes FlexRaw's file-reading logic much more complicated than FastRaw's logic. FlexRaw and FastRaw both default to using C's data writing. This means that data written to a new file using FlexRaw can be read by FastRaw, and vice-versa. To see if FlexRaw's logic is causing the slowdown, try reading your piddle (if it's one piddle per file) using FastRaw. David -- Sent via my carrier pigeon.
_______________________________________________ Perldl mailing list [email protected] http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
