Hello,
since everybody seems really helpful here, I'm going to ask for some
help with this:
I'm reading imaging data that looks like the following:
(header+2*x,y,z, ...)
In other words, there is a line of complex data (two floats) preceded by
a header (128 bytes long, 47 fields of different data type (bitmaps,
ushort, long ...).
Up until now, I have a loop reading first the header with read (then
using unpack), following by readflex to fetch a line). This requires two
file accesses per line, and a perl loop. For 12 GB of data that takes a
while.
In the header are fields that tell the indices in the big data matrix
where this should go to. So I use something like this to populate a big
piddle:
data(,x,header(field_y),header(field_z), ...)+=line
The + is there because some lines are acquired multiple times, the
lookup of the header is necessary since they can come in arbitrary order
and some are missing completely (intentionally).
Now I have the following issues:
1. The data piddle will be bigger than 2GB (see my previous request),
each dimension will max out around 10^5!
2. Do you see an effective pdlish way to avoid a loop?
ad 1) For the moment, I can/have to split data into multiple pidlles,
not nice but doable. Since the data is going to be processed further
(FFT, filtering ..) I have to find the best place to split, which can be
hard since the structure of the data varies a lot depending on the
experiment.
Thanks
Ingo
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
