Performance was also helped by the fact that the original inner product matmult algorithm was replaced by a tile based one which leads to better cache reuse,
,,,and you have Craig to thank for the implementation. :-) --Chris On Sun, Sep 7, 2014 at 7:03 AM, David Mertens <[email protected]> wrote: > It's also quite likely that Intel has heuristics to pre-fetch data, and > they're helping out. Maybe. > > David > > > On Sat, Sep 6, 2014 at 10:28 PM, Craig DeForest <[email protected]> > wrote: >> >> Cool! Maybe there are more cache hits than I expected (which was none)... >> >> (Mobile) >> >> >> > On Sep 6, 2014, at 1:54 PM, Chris Marshall <[email protected]> >> > wrote: >> > >> > On my PC (2.8GHZ i7) it takes about an hour for the multiply >> > just using $a x $b as Craig shows. I haven't tried using the >> > autothreading support to see how that changes things. >> > >> > As discussed already, GPU acceleration could allow for much >> > faster computation. For a start Nvidia has a cuBLAS library >> > which implements matrix multiply which could be used to >> > optimize the performance. >> > >> > --Chris >> > >> > >> > On Fri, Sep 5, 2014 at 11:49 AM, Craig DeForest >> > <[email protected]> wrote: >> >> If your matrix is not necessarily sparse, you will have to process it >> >> all >> >> through memory. PDL is optimized for problems that fit in your >> >> machine's >> >> RAM limit. 15000x15000 floats is 900 MB, which should fit within most >> >> machines. (15000x15000 double-precision values is 1.8 GB, which should >> >> also >> >> be OK). You'll need to set the global variable $PDL::BIGPDL to 1 to >> >> let >> >> Perl know you plan to work with arrays that large. >> >> >> >> My laptop computer has 16GB of RAM. This works fine: >> >> >> >> use PDL; >> >> $a = random(15000,15000); # generate 15000x15000 array of random >> >> numbers >> >> $b = random(15000,15000); # generate another one >> >> >> >> If you're running out of memory you may be trying to do something silly >> >> like >> >> read all the numbers in as Perl scalars...? >> >> >> >> On the other hand, this may take a while: >> >> >> >> $c = $a x $b; # brute-force matrix multiply -- ~200 hours to complete >> >> >> >> The reason is that the final step requires (8 * 15000 * 3 * 15000 * >> >> 15000) >> >> memory >> >> accesses. >> >> >> >> Finding eigenvalues of a 15000x15000 matrix is a nontrivial process. >> >> PDL has >> >> an eigenvalue solver ("eigens") but it is a general purpose tool for >> >> small >> >> matrices, it would take considerably longer than the age of the >> >> Universe to >> >> find the eigenvalues of a 15000x15000 nonsparse matrix -- so your >> >> project >> >> might be a little late if you use that. >> >> >> >> Working with large matrices is its own computational subject. PDL >> >> makes a >> >> nice framework for it, but for any serious operations you can't just >> >> use the >> >> kind of general purpose tools that work fine on (say) a 10x10 matrix. >> >> >> >> >> >> >> >> On Sep 5, 2014, at 9:12 AM, Ronak Agrawal <[email protected]> wrote: >> >> >> >> Thank You Sir for the early response. >> >> >> >> I am new to Perl and have been assigned project on Topic Modeling where >> >> I >> >> have to search, browse and find information from large archives of >> >> texts. >> >> >> >> Matrix operation is one of the operation and as per requirement my >> >> matrix >> >> may be sparse or dense. Is it possible for you help me with both the >> >> cases. >> >> >> >> More to that can you tell me some good methods to handle large data in >> >> Perl. >> >> >> >> Once again thank you for the response >> >> >> >> >> >> On Fri, Sep 5, 2014 at 7:36 PM, Craig DeForest >> >> <[email protected]> >> >> wrote: >> >>> >> >>> Glad to help. First, a few questions. Is the matrix sparse? (i.e. >> >>> are >> >>> less than, say 10^-3 of the elements nonzero?) How close to >> >>> tridiagonal is >> >>> it? >> >>> >> >>> >> >>> On Sep 5, 2014, at 6:27 AM, Ronak Agrawal <[email protected]> >> >>> wrote: >> >>> >> >>> Hi >> >>> >> >>> I am doing a project in Topic Modelling which involves large matrix >> >>> operations. >> >>> >> >>> I have a sql database from where I have to generate 15000 x 15000 >> >>> matix - >> >>> transform and obtain A'A.Later I have to find Eigen Values and Eigen >> >>> Vectors. >> >>> >> >>> Can you suggest me ways to do this in Perl.I get "Out of Memory" while >> >>> storing the matrix in memory. >> >>> >> >>> Your input will help in handling big data and therby making my project >> >>> success >> >>> >> >>> Thank You >> >>> >> >>> Ronak >> >>> >> >>> _______________________________________________ >> >>> Perldl mailing list >> >>> [email protected] >> >>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl >> >> >> >> >> >> >> >> _______________________________________________ >> >> Perldl mailing list >> >> [email protected] >> >> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl >> > >> >> _______________________________________________ >> Perldl mailing list >> [email protected] >> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl > > > > > -- > "Debugging is twice as hard as writing the code in the first place. > Therefore, if you write the code as cleverly as possible, you are, > by definition, not smart enough to debug it." -- Brian Kernighan _______________________________________________ Perldl mailing list [email protected] http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
