Yet another alternative is PDL::IO::FastRaw's mapfraw function. It uses
disk space as the primary storage for the data, and automatically maps
required chunks of data into memory via the OS's memory mapping
capability. It would allow you to use piddles larger than your memory,
but there's a performance hit. For example:
use PDL;
use PDL::IO::FastRaw;
# create a gigantic piddle on disk
$PDL::BIGPDL = 1;
my $bigpdl = mapfraw( 'bigpdl', { Dims => [ 15_000, 15_000 ],
Datatype => double,
Creat => 1
},
);
$bigpdl .= sequence( $bigpdl->dims );
On Tue, Sep 9, 2014 at 9:41 AM, Chris Marshall <[email protected]>
wrote:
> On Tue, Sep 9, 2014 at 9:28 AM, Chris Marshall <[email protected]>
> wrote:
> > On Tue, Sep 9, 2014 at 5:30 AM, Ronak Agrawal <[email protected]>
> wrote:
> >>
> >> ...snip...
> >>
> >> I am using Virtual Machine with 1 GB Memory and therby I am getting Out
> of
> >> Memory for $a = sequnece 15000 * 15000
> >> Is there any way to tackle it...Currently i need to compute a Spare
> matrix
> >> of 12000 * 8000 and find A'A ( A transpose A )
> >
> > If you have very small memory, you'll need to do something
> > like a disk-to-disk computation.
>
> Take a look a PDL::DiskCache where you could use readflex/writeflex
> to read/write the blocks of the matrix and then write a blocked matrix
> multiply on the PDL::DiskCache object with the IO handled transparently.
>
> --Chris
>
> _______________________________________________
> Perldl mailing list
> [email protected]
> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
>
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
