$PDL::BIGPDL is a PDL package variable that you set to enable pdl allocations bigger than 1GiB in size. Just set it to 1 before you need to create huge piddles:
$PDL::BIGPDL = 1; $huge = pdl(300*1000*1000); Steve Cicala wrote: > Thanks Chris--The output is attached. > > As auxiliary question related to working with lots of data that the program > can't hold all it once: If I have around 100,000 individual files containing > a single column, all of the same length, for I/O speed is it better to > convert all of the files to load using PDL::IO::FastRaw, or is it better to > read it in from .txt each time using rcols()? I've notice that pdls loaded > using rcols aren't taking up any memory (state VC in help vars). > > Looking at Craig's output, it could be that I'm not calling $PDL::BIGPDL > properly. When working in a script, is > > $c=$PDL::BIGPDL=$a==$b->dummy > > the way I allow $c to be >1Gb? > > Many Thanks. > > Steve > > On Sun, Sep 14, 2008 at 2:40 PM, Chris Marshall <[EMAIL PROTECTED]> wrote: > > >> Could you please run 'perldl -V' and send the output. >> I could not reproduce the problem as I had out of memory >> errors at that point. >> >> Maybe someone with a large memory 64bit OS and >> PDL could take a look further to debug the internals. >> >> As a work around, just loop over your large $a values >> by blocks small enough to run using slicing to extract >> the values of interest for each step. >> >> --Chris >> >> Steve Cicala wrote: >> >> >>> Hi--I am running perldl on a unix server with 32Gb of memory, but am >>> having >>> a problem with the following operation:Here's a small version that works: >>> perldl> $a=sequence(5) >>> >>> perldl> $b=pdl[1,4,0] >>> >>> perldl> p $c=$a==$b->dummy >>> >>> [ >>> [0 1 0 0 0] >>> [0 0 0 0 1] >>> [1 0 0 0 0] >>> ] >>> (I am using this to collect indicies in a that correspond with elements in >>> b: >>> $d=which(maximum (($a==$b->dummy)->xchg(0,1))!=0) >>> ). >>> >>> Now, when I try to use this operation for large numbers (i.e. $a is >>> 480000x1 >>> and $b is 500x1) I get: >>> >>> 'multielement piddle in a conditional expression' >>> >>> --And I get this error whether calculating $c on its own, or just sticking >>> the expression that generates $c into the one that generates $d. >>> >>> I have also tried: >>> >>> $c=$PDL::BIGPDL=$a==$b->dummy >>> >>> and get the same error. >>> >>> ------------------------------------------------------------------------ >>> >>> _______________________________________________ >>> Perldl mailing list >>> [email protected] >>> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl >>> ------------------------------------------------------------------------ >>> >>> No virus found in this incoming message. >>> Checked by AVG. Version: 7.5.526 / Virus Database: 270.6.21/1671 - Release >>> Date: 9/14/2008 7:16 AM >>> >>> >>> >> > > > ------------------------------------------------------------------------ > > No virus found in this incoming message. > Checked by AVG. > Version: 7.5.526 / Virus Database: 270.6.21/1671 - Release Date: 9/14/2008 > 7:16 AM > _______________________________________________ Perldl mailing list [email protected] http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
