To create the matrix I am calling the following snippet for 15000 times
with array reference as parameter
{
$p_arrayref = $_[0];
my $p_new = pdl ( [@$p_arrayref] );
$matrix = $matrix->glue(1,$p_new);
}
Since the brute force method $c = $a x $b will take 200 hours, can you
guide me with better approach..
On Fri, Sep 5, 2014 at 9:19 PM, Craig DeForest <[email protected]>
wrote:
> If your matrix is not necessarily sparse, you will have to process it all
> through memory. PDL is optimized for problems that fit in your machine's
> RAM limit. 15000x15000 floats is 900 MB, which should fit within most
> machines. (15000x15000 double-precision values is 1.8 GB, which should
> also be OK). You'll need to set the global variable $PDL::BIGPDL to 1 to
> let Perl know you plan to work with arrays that large.
>
> My laptop computer has 16GB of RAM. This works fine:
>
> use PDL;
> $a = random(15000,15000); # generate 15000x15000 array of random numbers
> $b = random(15000,15000); # generate another one
>
> If you're running out of memory you may be trying to do something silly
> like read all the numbers in as Perl scalars...?
>
> On the other hand, this may take a while:
>
> $c = $a x $b; # brute-force matrix multiply -- ~200 hours to complete
>
> The reason is that the final step requires (8 * 15000 * 3 * 15000 * 15000)
> memory
> accesses.
>
> Finding eigenvalues of a 15000x15000 matrix is a nontrivial process. PDL
> has an eigenvalue solver ("eigens") but it is a general purpose tool for
> small matrices, it would take considerably longer than the age of the
> Universe to find the eigenvalues of a 15000x15000 nonsparse matrix -- so
> your project might be a little late if you use that.
>
> Working with large matrices is its own computational subject. PDL makes a
> nice framework for it, but for any serious operations you can't just use
> the kind of general purpose tools that work fine on (say) a 10x10 matrix.
>
>
>
> On Sep 5, 2014, at 9:12 AM, Ronak Agrawal <[email protected]> wrote:
>
> Thank You Sir for the early response.
>
> I am new to Perl and have been assigned project on Topic Modeling where I
> have to search, browse and find information from large archives of texts.
>
> Matrix operation is one of the operation and as per requirement my matrix
> may be sparse or dense. Is it possible for you help me with both the cases.
>
> More to that can you tell me some good methods to handle large data in
> Perl.
>
> Once again thank you for the response
>
>
> On Fri, Sep 5, 2014 at 7:36 PM, Craig DeForest <[email protected]>
> wrote:
>
>> Glad to help. First, a few questions. Is the matrix sparse? (i.e. are
>> less than, say 10^-3 of the elements nonzero?) How close to tridiagonal is
>> it?
>>
>>
>> On Sep 5, 2014, at 6:27 AM, Ronak Agrawal <[email protected]> wrote:
>>
>> *Hi*
>>
>> *I am doing a project in Topic Modelling which involves large matrix
>> operations.*
>>
>> *I have a sql database from where I have to generate 15000 x 15000 matix -
>> transform and obtain A'A.Later I have to find Eigen Values and Eigen
>> Vectors.*
>>
>> *Can you suggest me ways to do this in Perl.I get "Out of Memory" while
>> storing the matrix in memory.*
>>
>> *Your input will help in handling big data and therby making my project
>> success*
>>
>> Thank You
>>
>> Ronak
>>
>> _______________________________________________
>> Perldl mailing list
>> [email protected]
>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
>>
>>
>>
>
>
_______________________________________________
Perldl mailing list
[email protected]
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
