Hi, Chris:
Thanks for your reply.
I'm using PDL v2.4.11 on a 64bit Scientific Linux machine (Linux
version 2.6.32-220.4.1.el6.x86_64). The PDL should be the most recent version.
2**31=2,147,483,648 and my matrix has about 4000 * 50000 = 200,000,000
elements, storing number values, with absolute value less than 1000 in general
and with 2 digit accuracy (such as 123.45). So, I should be fine if I only do
some simple manipulations, such as sum/subtraction etc., right?
Thanks!
Nan
-----Original Message-----
From: Chris Marshall [mailto:[email protected]]
Sent: Tuesday, October 30, 2012 12:44 PM
To: Nan Yu
Cc: [email protected]
Subject: Re: [Perldl] how to work with large 2D matrix
Hi Nan-
I see Craig answered your specific question. A couple of thoughts.
You don't say what version of PDL you are using (or your
platform) but the current release has support for memory mapped IO (see docs
for PDL::IO::FlexRaw).
There is also support for allocating piddles (the PDL data objects) having more
than 2**31 elements in them. Some of the internal indexing in PDL is in 32bit
signed integers which means some operations might fail due to counter overflow
or wrap-around.
If you stick to single dimensions less than 2**31 elements and work with slices
smaller than than size it is believed to allow one to work around the
limitations. I would advised you to stay below that size in your work until
you have experience with the processing flow. After that, we're always looking
for help with debugging large-piddle support! :-)
--Chris
On Tue, Oct 30, 2012 at 12:21 AM, Nan Yu <[email protected]> wrote:
>
> I'm working with a large 2D matrix: about 4000 x 50000.
>
> I used to use perl 2D array to do the job (when working with smaller
> dimensions) but the memory became an issue when the dimension grows
> into this size. So I decided to use PDL.
>
> After I finished my code and started testing it, I found out that just
> to initiate the 2D matrix takes a long time.
>
> I start with an 1D array of 4000 elements and then use "$matrix =
> $matrix->glue(1,$oneDarray)" within a loop (from 1 to 50000) to create
> the 2D matrix. The first 5000-10000 were reasonable quick (within 10
> mins) then the speed slows down a lot.
>
> I'm wondering if I used a wrong way to do the initiate of the matrix.
>
> Could someone please tell me how to work with the large 2D matrix?
>
> Thanks!
> Nan Yu
Please note that my email address has changed to [email protected]
________________________________
*** The information contained in this communication may be confidential, is
intended only for the use of the recipient(s) named above, and may be legally
privileged. If the reader of this message is not the intended recipient, you
are hereby notified that any dissemination, distribution, or copying of this
communication, or any of its contents, is strictly prohibited. If you have
received this communication in error, please return it to the sender
immediately and delete the original message and any copies of it. If you have
any questions concerning this message, please contact the sender. ***
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
