Here is a large data structure --
my $pdl = pdl (
(
1 .. 10,
[ [1 .. 33], x $d ], # d arrays
[ [1 .. 57], x $l ], # l arrays
[ [1 .. 9 ], x $m ], # m arrays
)
);
$d is BETWEEN 0 and 5
$l is BETWEEN 1 and 10
$m is 7300, but could be as high as 18,000 or 20,000
my $size = howbig($pdl->get_datatype);
print "size of pdl is: $size\n";
print $pdl->info("Type: %T Dim: %-15D State: %S"), "\n";
my $n = $pdl->nelem;
print "There are $n elements in the piddle\n";
I get the following --
size of pdl is: 8
Type: Double Dim: D [57,7300,13] State: P
There are 54093000 elements in the piddle
Makes sense so far, but what does that "size of pdl is: 8" mean?
Surely, that is not the number of bytes being used by this data
structure? By my calculations, the data structure weighs in at about
450 KB packed as a Storable object. By the way... in the pseudo code
above, I have shown the number of elements in the arrays, not the
actual values. So, for example, in each of the 'd' arrays, there are
33 elements, but only about 4 or 5 of them are INTEGERS, the rest
being REAL numbers. This is useful to get a sense of the size of the
data structure.
Now, this data structure is the data for computation that is applied
to a large array, say, 1000 x 1000 or even 1500 x 1800, so between a
million to a couple of million or more elements, on a cell by cell
basis. Imagine applying f(d) to the array where d is data structure,
with f(d) being applied to each cell individually.
Curious to test the limits of my machine and PDL, I tried to create a
piddle that held 1000_000 such structures, I got a 'bus error'. At an
array with 100 elements, I got a segmentation fault. At an array with
10 elements, it worked.
I am seeking some suggestions on how to work with such data using PDL.
Many thanks,
--
Puneet Kishor http://www.punkish.org/
Nelson Institute for Environmental Studies http://www.nelson.wisc.edu/
Carbon Model http://carbonmodel.org/
Open Source Geospatial Foundation http://www.osgeo.org/
Sent from: College Park MD United States.
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
