I couldn't reproduce the problem if I disable the 64bit index support either. Perhaps the problem is in the perl version. Maybe someone will be able to reproduce the problem when you have a specific test case---including me.
--Chris On Fri, Nov 21, 2014 at 9:34 AM, Chris Marshall <devel.chm...@gmail.com> wrote: > Hi Ken- > > I am unable to generate the error with PDL-2.007 > either. My system has 8GiB of memory and the > PDL build is using the 64bit index support. > > What are the specs of your linux box and could > you please send the output of the 'perldl -V' > command. If you built PDL from sources, then > the build log should indicate whether 64bit index > support was enabled. If you $Config{ivsize} is > 8 then you should have 64bit support as well. > > --Chris > > > On Fri, Nov 21, 2014 at 8:02 AM, Chris Marshall <devel.chm...@gmail.com> > wrote: >> Ken- >> >> If you could make a short script that generates the >> problem along with the output/error messages, that >> would help. >> >> Do you have $PDL::BIGPDL set? Might try with >> that set to 1. >> >> I'll try the problem code on PDL-2.007 to see if that >> is the reason for the differences. >> >> --Chris >> >> >> >> On Thu, Nov 20, 2014 at 6:18 PM, LYONS, KENNETH B (KENNETH) >> <k...@research.att.com> wrote: >>> Chris >>> I'm running perl 5.8.8 on a rather old linux system. I installed the perl >>> modules rather recently from the PDL site, so I'd expect they are up to >>> date with whatever is there. From the names of the files, I'd say it's >>> 2.007. >>> >>> I've tried a variety of ways of using the inplace method, and none of them >>> produced a perl error akin to what you got below. The errors were coming >>> out of the PDL module itself, complaining about the size of the piddle >>> being over 1GB. Given the dimensions of the piddle that is being >>> calculated (around 200 MB), that shouldn't have happened--unless it's using >>> doubles, which would make it ~1.6 GB. Like I said, I got around the >>> problem in kind of a hack, by just slicing things up 20K rows at a >>> time--but I'd really like to find a way to do it right! >>> >>> Among the things I tried were these: >>> $sigs->xchg(0,1) *= $present; >>> $sigs->xchg(0,1)->inplace->mult($present,0); >>> PDL::Ops::mult(inplace $sigs->xchg(0,1), $present, 0); >>> $sigs->xchg(0,1)->inplace *= present; >>> None of which got around the error. >>> >>> Below is what finally worked (but only by occupying more memory than it >>> should): >>> >>> ($psize) = $present->dims; >>> $STEPSIZE = 20000; >>> for ($p = 0; $p < $psize; $p += $STEPSIZE) { # note: it's known that >>> $present and $sigs have the same size! >>> my $start = $p; >>> my $end = $start+$STEPSIZE-1; >>> $end = $psize-1 if $end >= $psize; >>> $sigs->xchg(0,1)->slice("$start:$end,:,:") *= >>> $present->slice("$start:$end"); >>> } >>> >>> Like I said, it's a bit of a hack. But it does wind up doing the >>> appropriate filtering on the $sigs matrix. >>> >>> Ken >>> >>> p.s. I don't know if it makes a difference, per se, but you are evidently >>> operating in an interactive environment, not an actual perl script. I'm >>> using this to automate thru a very large body of data, eventually be run >>> automatically on a daily basis, so it's written as a script that calls the >>> PDL modules. The error I refer to above was appearing in the error output >>> of the perl command. >>> >>> KL >>> >>> ---------------------------------------------------------------------------- >>> >>> Below is the remainder of the thread that was mostly sidebar: >>> ------------------------------------- >>> Hi Ken- >>> >>> You could sync up with the message I forwarded >>> to perldl by replying with this message to that >>> thread. The main reason for keeping the discussion >>> on the list is so that others can benefit from the >>> discussion and/or offer other points of view/facts/... >>> >>> I tried the following in pdl2 and was not able to generate >>> an error. You are right that all byte args shouldn't be >>> expanded to double intermediates. I'm using PDL-2.007_03 >>> on cygwin64/win7 and the *= works fine but I get an error >>> with the inplace construct (not the same as yours) >>> >>> pdl> $sigs = (10*random(40,150000,26))->floor->byte >>> pdl> $present = (20*random(150000))->floor->byte >>> pdl> $ns = $sigs->copy >>> pdl> ?vars >>> PDL variables in package main:: >>> >>> Name Type Dimension Flow State Mem >>> ---------------------------------------------------------------- >>> $ns Byte D [40,150000,26] P 148.77MB >>> $present Byte D [150000] P 0.14MB >>> $sigs Byte D [40,150000,26] P 148.77MB >>> pdl> $sigs->xchg(0,1) *= $present # works >>> pdl> $sigs = $ns->copy >>> pdl> $sigs->xchg(0,1)->inplace *= $present >>> Runtime error: Can't modify non-lvalue subroutine call at (eval 484) line >>> 5. >>> >>> What is your os/platform specs and what version of PDL are >>> you using? >>> >>> --Chris >>> >>> On Thu, Nov 20, 2014 at 2:47 PM, LYONS, KENNETH B (KENNETH) >>> <k...@research.att.com> wrote: >>>> (Didn't understand your first line, as there was no cc on this message? I >>>> pretty much automatically avoid ever using reply-all, but I guess in this >>>> case that's how it's supposed to work, right? How do I cc it to get the >>>> thread to match up?) >>>> >>>> Actually, all the pdls involved are byte type. I was assuming when I saw >>>> the errors occurring that it was somehow generating a double intermediate, >>>> because it should have had plenty of room if it stayed as byte. >>>> >>>> The specific code was as follows: >>>> >>>> # sigs is byte, with dimensions about 40 x 150000 x 26 >>>> # present is byte, with dimension of 150000 >>>> $sigs->xchg(0,1)->inplace *= $present; >>>> >>>> I had tried numerous ways of using inplace in that line, and none of them >>>> avoided the complaint that it had run out of memory (although the memory >>>> usage prior to that command was about 10%). So if it's not generating a >>>> double intermediate, I don't see why it would run out of memory (it >>>> shouldn't have exceeded about 20% or so). I finally got it to work by >>>> splitting the structures up into slices of about 20K rows each, and doing >>>> the calculation that way. >>>> >>>> Other approaches? >>>> >>>> Ken >>>> >>>> >>>> >>>> >>>> -----Original Message----- >>>> From: Chris Marshall [mailto:devel.chm...@gmail.com] >>>> Sent: Wednesday, November 19, 2014 4:23 PM >>>> To: LYONS, KENNETH B (KENNETH) >>>> Subject: Re: [Perldl] matching vectors inside a PDL >>>> >>>> <re-cc-ing the perldl list> >>>> >>>> Thanks for the background. If you hit a snag, feel free to >>>> post to the perldl list. We're usually able to help for >>>> specific problems especially if accompanied by code >>>> demonstrating the problem: >>>> >>>> If I have a big byte piddle $a and I multiply it in-place >>>> my PDL session crashes because of a huge intermediate >>>> temp: >>>> >>>> pdl> $a = (10*random(100))->floor->byte; >>>> >>>> pdl> $a->inplace->mult(5,0); >>>> Error message here or crash >>>> >>>> Without a specific example, I would guess that the problem >>>> is the piddle you are multiplying by (or perl scalar) is of >>>> type double which would result in an intermediate temp of >>>> double type which would then collapse down to a byte piddle >>>> again at the end. If both arguments to multiply are of byte >>>> type, you can avoid the big double intermediate temp. E.g. >>>> >>>> pdl> p pdl(5)->type >>>> double >>>> pdl> p byte(5)->type >>>> byte >>>> >>>> Improved type support is planned for the PDL3 work. My >>>> initial ideas for bitfield support can be seen here: >>>> >>>> >>>> http://mailman.jach.hawaii.edu/pipermail/pdl-porters/2013-December/006132.html >>>> >>>> Hope this helps, >>>> Chris >>>> >>>> On Wed, Nov 19, 2014 at 1:42 PM, LYONS, KENNETH B (KENNETH) >>>> <k...@research.att.com> wrote: >>>>> Chris >>>>> >>>>> In answer to your question: my path in was as follows: I wanted to find >>>>> a way to implement an LP on a medium-size problem (<~10K variables), and >>>>> the rest of my code was in perl--so I went looking for an LP >>>>> implementation in perl. I was expecting to find a C-compiled module >>>>> that would do an LP specifically. I found some instances of that sort of >>>>> thing, but I also ran across one using PDL. It didn't do quite what I >>>>> wanted, but when I saw the PDL site, it was obvious this was something I >>>>> needed to know about. I wound up writing my own simplex implementation >>>>> in PDL to do specifically what I needed, and that worked great--and I was >>>>> pretty blown away at the speed. So then I started looking into how I >>>>> could back up and get the datasets I was dealing with implemented as PDLs >>>>> to start with. So I've got a good bit of code now using PDL, not just >>>>> the little simplex program (which was only a few dozen lines--that was >>>>> pretty easy to implement in PDL.) >>>>> >>>>> I continue to have issues with the documentation, though. Just as one >>>>> example from today: the mult function seems to claim that you can get it >>>>> to operate in-place. And for me that was important, because I'm dealing >>>>> with a large dataset (of byte variables). But, not only does "mult" by >>>>> itself cause an error because it isn't exported, but when I try to use it >>>>> as PDL::Ops::Mult(inplace ...), or PDL::Mult(inplace ...), or as >>>>> $piddle->inplace->mult(...), it completely fails to avoid generating a >>>>> large intermediate. That was clobbering my program, repetitively, so I >>>>> finally punted and decided to break that step up into segments with only >>>>> a few hundred thousand elements to multiply in each (using slices), and >>>>> that got me around the problem. But there was nothing in the >>>>> documentation that seemed to suggest that would be necessary. It also >>>>> seemed, although I didn't document this carefully, that changing the >>>>> default PDL type didn't have any impact on the size of that temporary >>>>> intermediate (I think it was using double no matter what I did--whereas >>>>> using byte would have been fine.) >>>>> >>>>> I'd love it, in this context, if there were a PDL type of "bit" by the >>>>> way, since that's actually what this problem is using--it's a 3D binary >>>>> matrix, of ones and zeroes, with up to ~3*10^^9 elements. When the >>>>> number of elements goes above ~200M, when I'm using bytes, I have to do >>>>> things to break it up and process one segment at a time, and it would be >>>>> nice if that weren't necessary--but there is evidently no implementation >>>>> of a "bit" type in PDL. >>>>> >>>>> Ken >>>>> >>>>> >>> >>> -----Original Message----- >>> From: Chris Marshall [mailto:devel.chm...@gmail.com] >>> Sent: Saturday, November 15, 2014 11:42 AM >>> To: Derek Lamb >>> Cc: LYONS, KENNETH B (KENNETH); perldl >>> Subject: Re: [Perldl] matching vectors inside a PDL >>> >>> Hi Ken and welcome to the PDL community! >>> >>>> On Nov 14, 2014, at 1:33 PM, LYONS, KENNETH B (KENNETH) >>>> <k...@research.att.com> wrote: >>>> >>>> Yes, most of this I knew, but thanks. It’s because of that behavior of > >>>> and <, that you mentioned, that I thought that ‘==’ would compare element >>>> by >>>> element instead of on the whole vector. >>>> >>>> Have you ever tried, for example, to search the documentation for, say, the >>>> function “list”? it gives you every occurrence of the word “list” in the >>>> documents (which, needless to say, is rather voluminous, and the first few >>>> hundred entries have nothing to do with the function!) there should be >>>> some >>>> analog of the “man” command in unix that gives you information about the >>>> *function* without all the other garbage. I think it’s just doing >>>> something >>>> akin to a grep thru the documents. >>> >>> In addition to the help/? and apropos/?? in the PDL shells >>> (pdl2 and perldl) there is the command line version pdldoc >>> which can be used starting with 'pdldoc pdldoc' to get the usage. >>> >>> When I do 'pdldoc -a list' I get 45 lines of output all >>> of whose descriptions seem relevant to a general search >>> for something having to do with 'list' including the 'list' >>> command itself. This type of problem is not specific to >>> PDL as searching the docs for any complicated system >>> or program does tend to produce a large number of >>> incomprehensible and not particularly useful results. >>> >>> It is definitely desired to have smarter and more useful >>> documentation searches. The ability to add keywords >>> would be nice and is on the feature request list, I believe. >>> >>> If you are on unix-ish system, you might try something >>> like 'pdldoc -a list | grep --color list' to make the output >>> more visually comprehensible. (We should probably add >>> that to the PDL shell output) >>> >>>> It’s horribly designed in that regard. The software itself is great, and >>>> I’m very happy with the results, but finding the simplest little thing in >>>> the docs can be a total pain! >>> >>> I understand frustration. What would really help PDL >>> development would be to know how you got to using >>> PDL without being introduced to the concepts that would >>> have made the learning curve less steep. (At least you >>> found the mailing list and used it. :-) >>> >>> Happy PDL-ing! >>> Chris _______________________________________________ Perldl mailing list Perldl@jach.hawaii.edu http://mailman.jach.hawaii.edu/mailman/listinfo/perldl