Hi Ken-
I am unable to generate the error with PDL-2.007
either. My system has 8GiB of memory and the
PDL build is using the 64bit index support.
What are the specs of your linux box and could
you please send the output of the 'perldl -V'
command. If you built PDL from sources, then
the build log should indicate whether 64bit index
support was enabled. If you $Config{ivsize} is
8 then you should have 64bit support as well.
--Chris
On Fri, Nov 21, 2014 at 8:02 AM, Chris Marshall <devel.chm...@gmail.com> wrote:
> Ken-
>
> If you could make a short script that generates the
> problem along with the output/error messages, that
> would help.
>
> Do you have $PDL::BIGPDL set? Might try with
> that set to 1.
>
> I'll try the problem code on PDL-2.007 to see if that
> is the reason for the differences.
>
> --Chris
>
>
>
> On Thu, Nov 20, 2014 at 6:18 PM, LYONS, KENNETH B (KENNETH)
> <k...@research.att.com> wrote:
>> Chris
>> I'm running perl 5.8.8 on a rather old linux system. I installed the perl
>> modules rather recently from the PDL site, so I'd expect they are up to date
>> with whatever is there. From the names of the files, I'd say it's 2.007.
>>
>> I've tried a variety of ways of using the inplace method, and none of them
>> produced a perl error akin to what you got below. The errors were coming
>> out of the PDL module itself, complaining about the size of the piddle being
>> over 1GB. Given the dimensions of the piddle that is being calculated
>> (around 200 MB), that shouldn't have happened--unless it's using doubles,
>> which would make it ~1.6 GB. Like I said, I got around the problem in kind
>> of a hack, by just slicing things up 20K rows at a time--but I'd really like
>> to find a way to do it right!
>>
>> Among the things I tried were these:
>> $sigs->xchg(0,1) *= $present;
>> $sigs->xchg(0,1)->inplace->mult($present,0);
>> PDL::Ops::mult(inplace $sigs->xchg(0,1), $present, 0);
>> $sigs->xchg(0,1)->inplace *= present;
>> None of which got around the error.
>>
>> Below is what finally worked (but only by occupying more memory than it
>> should):
>>
>> ($psize) = $present->dims;
>> $STEPSIZE = 20000;
>> for ($p = 0; $p < $psize; $p += $STEPSIZE) { # note: it's known that
>> $present and $sigs have the same size!
>> my $start = $p;
>> my $end = $start+$STEPSIZE-1;
>> $end = $psize-1 if $end >= $psize;
>> $sigs->xchg(0,1)->slice("$start:$end,:,:") *=
>> $present->slice("$start:$end");
>> }
>>
>> Like I said, it's a bit of a hack. But it does wind up doing the
>> appropriate filtering on the $sigs matrix.
>>
>> Ken
>>
>> p.s. I don't know if it makes a difference, per se, but you are evidently
>> operating in an interactive environment, not an actual perl script. I'm
>> using this to automate thru a very large body of data, eventually be run
>> automatically on a daily basis, so it's written as a script that calls the
>> PDL modules. The error I refer to above was appearing in the error output
>> of the perl command.
>>
>> KL
>>
>> ----------------------------------------------------------------------------
>>
>> Below is the remainder of the thread that was mostly sidebar:
>> -------------------------------------
>> Hi Ken-
>>
>> You could sync up with the message I forwarded
>> to perldl by replying with this message to that
>> thread. The main reason for keeping the discussion
>> on the list is so that others can benefit from the
>> discussion and/or offer other points of view/facts/...
>>
>> I tried the following in pdl2 and was not able to generate
>> an error. You are right that all byte args shouldn't be
>> expanded to double intermediates. I'm using PDL-2.007_03
>> on cygwin64/win7 and the *= works fine but I get an error
>> with the inplace construct (not the same as yours)
>>
>> pdl> $sigs = (10*random(40,150000,26))->floor->byte
>> pdl> $present = (20*random(150000))->floor->byte
>> pdl> $ns = $sigs->copy
>> pdl> ?vars
>> PDL variables in package main::
>>
>> Name Type Dimension Flow State Mem
>> ----------------------------------------------------------------
>> $ns Byte D [40,150000,26] P 148.77MB
>> $present Byte D [150000] P 0.14MB
>> $sigs Byte D [40,150000,26] P 148.77MB
>> pdl> $sigs->xchg(0,1) *= $present # works
>> pdl> $sigs = $ns->copy
>> pdl> $sigs->xchg(0,1)->inplace *= $present
>> Runtime error: Can't modify non-lvalue subroutine call at (eval 484) line
>> 5.
>>
>> What is your os/platform specs and what version of PDL are
>> you using?
>>
>> --Chris
>>
>> On Thu, Nov 20, 2014 at 2:47 PM, LYONS, KENNETH B (KENNETH)
>> <k...@research.att.com> wrote:
>>> (Didn't understand your first line, as there was no cc on this message? I
>>> pretty much automatically avoid ever using reply-all, but I guess in this
>>> case that's how it's supposed to work, right? How do I cc it to get the
>>> thread to match up?)
>>>
>>> Actually, all the pdls involved are byte type. I was assuming when I saw
>>> the errors occurring that it was somehow generating a double intermediate,
>>> because it should have had plenty of room if it stayed as byte.
>>>
>>> The specific code was as follows:
>>>
>>> # sigs is byte, with dimensions about 40 x 150000 x 26
>>> # present is byte, with dimension of 150000
>>> $sigs->xchg(0,1)->inplace *= $present;
>>>
>>> I had tried numerous ways of using inplace in that line, and none of them
>>> avoided the complaint that it had run out of memory (although the memory
>>> usage prior to that command was about 10%). So if it's not generating a
>>> double intermediate, I don't see why it would run out of memory (it
>>> shouldn't have exceeded about 20% or so). I finally got it to work by
>>> splitting the structures up into slices of about 20K rows each, and doing
>>> the calculation that way.
>>>
>>> Other approaches?
>>>
>>> Ken
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Chris Marshall [mailto:devel.chm...@gmail.com]
>>> Sent: Wednesday, November 19, 2014 4:23 PM
>>> To: LYONS, KENNETH B (KENNETH)
>>> Subject: Re: [Perldl] matching vectors inside a PDL
>>>
>>> <re-cc-ing the perldl list>
>>>
>>> Thanks for the background. If you hit a snag, feel free to
>>> post to the perldl list. We're usually able to help for
>>> specific problems especially if accompanied by code
>>> demonstrating the problem:
>>>
>>> If I have a big byte piddle $a and I multiply it in-place
>>> my PDL session crashes because of a huge intermediate
>>> temp:
>>>
>>> pdl> $a = (10*random(100))->floor->byte;
>>>
>>> pdl> $a->inplace->mult(5,0);
>>> Error message here or crash
>>>
>>> Without a specific example, I would guess that the problem
>>> is the piddle you are multiplying by (or perl scalar) is of
>>> type double which would result in an intermediate temp of
>>> double type which would then collapse down to a byte piddle
>>> again at the end. If both arguments to multiply are of byte
>>> type, you can avoid the big double intermediate temp. E.g.
>>>
>>> pdl> p pdl(5)->type
>>> double
>>> pdl> p byte(5)->type
>>> byte
>>>
>>> Improved type support is planned for the PDL3 work. My
>>> initial ideas for bitfield support can be seen here:
>>>
>>>
>>> http://mailman.jach.hawaii.edu/pipermail/pdl-porters/2013-December/006132.html
>>>
>>> Hope this helps,
>>> Chris
>>>
>>> On Wed, Nov 19, 2014 at 1:42 PM, LYONS, KENNETH B (KENNETH)
>>> <k...@research.att.com> wrote:
>>>> Chris
>>>>
>>>> In answer to your question: my path in was as follows: I wanted to find a
>>>> way to implement an LP on a medium-size problem (<~10K variables), and the
>>>> rest of my code was in perl--so I went looking for an LP implementation in
>>>> perl. I was expecting to find a C-compiled module that would do an LP
>>>> specifically. I found some instances of that sort of thing, but I also
>>>> ran across one using PDL. It didn't do quite what I wanted, but when I
>>>> saw the PDL site, it was obvious this was something I needed to know
>>>> about. I wound up writing my own simplex implementation in PDL to do
>>>> specifically what I needed, and that worked great--and I was pretty blown
>>>> away at the speed. So then I started looking into how I could back up and
>>>> get the datasets I was dealing with implemented as PDLs to start with.
>>>> So I've got a good bit of code now using PDL, not just the little simplex
>>>> program (which was only a few dozen lines--that was pretty easy to
>>>> implement in PDL.)
>>>>
>>>> I continue to have issues with the documentation, though. Just as one
>>>> example from today: the mult function seems to claim that you can get it
>>>> to operate in-place. And for me that was important, because I'm dealing
>>>> with a large dataset (of byte variables). But, not only does "mult" by
>>>> itself cause an error because it isn't exported, but when I try to use it
>>>> as PDL::Ops::Mult(inplace ...), or PDL::Mult(inplace ...), or as
>>>> $piddle->inplace->mult(...), it completely fails to avoid generating a
>>>> large intermediate. That was clobbering my program, repetitively, so I
>>>> finally punted and decided to break that step up into segments with only a
>>>> few hundred thousand elements to multiply in each (using slices), and that
>>>> got me around the problem. But there was nothing in the documentation
>>>> that seemed to suggest that would be necessary. It also seemed, although
>>>> I didn't document this carefully, that changing the default PDL type
>>>> didn't have any impact on the size of that temporary intermediate (I think
>>>> it was using double no matter what I did--whereas using byte would have
>>>> been fine.)
>>>>
>>>> I'd love it, in this context, if there were a PDL type of "bit" by the
>>>> way, since that's actually what this problem is using--it's a 3D binary
>>>> matrix, of ones and zeroes, with up to ~3*10^^9 elements. When the number
>>>> of elements goes above ~200M, when I'm using bytes, I have to do things to
>>>> break it up and process one segment at a time, and it would be nice if
>>>> that weren't necessary--but there is evidently no implementation of a
>>>> "bit" type in PDL.
>>>>
>>>> Ken
>>>>
>>>>
>>
>> -----Original Message-----
>> From: Chris Marshall [mailto:devel.chm...@gmail.com]
>> Sent: Saturday, November 15, 2014 11:42 AM
>> To: Derek Lamb
>> Cc: LYONS, KENNETH B (KENNETH); perldl
>> Subject: Re: [Perldl] matching vectors inside a PDL
>>
>> Hi Ken and welcome to the PDL community!
>>
>>> On Nov 14, 2014, at 1:33 PM, LYONS, KENNETH B (KENNETH)
>>> <k...@research.att.com> wrote:
>>>
>>> Yes, most of this I knew, but thanks. It’s because of that behavior of >
>>> and <, that you mentioned, that I thought that ‘==’ would compare element by
>>> element instead of on the whole vector.
>>>
>>> Have you ever tried, for example, to search the documentation for, say, the
>>> function “list”? it gives you every occurrence of the word “list” in the
>>> documents (which, needless to say, is rather voluminous, and the first few
>>> hundred entries have nothing to do with the function!) there should be some
>>> analog of the “man” command in unix that gives you information about the
>>> *function* without all the other garbage. I think it’s just doing something
>>> akin to a grep thru the documents.
>>
>> In addition to the help/? and apropos/?? in the PDL shells
>> (pdl2 and perldl) there is the command line version pdldoc
>> which can be used starting with 'pdldoc pdldoc' to get the usage.
>>
>> When I do 'pdldoc -a list' I get 45 lines of output all
>> of whose descriptions seem relevant to a general search
>> for something having to do with 'list' including the 'list'
>> command itself. This type of problem is not specific to
>> PDL as searching the docs for any complicated system
>> or program does tend to produce a large number of
>> incomprehensible and not particularly useful results.
>>
>> It is definitely desired to have smarter and more useful
>> documentation searches. The ability to add keywords
>> would be nice and is on the feature request list, I believe.
>>
>> If you are on unix-ish system, you might try something
>> like 'pdldoc -a list | grep --color list' to make the output
>> more visually comprehensible. (We should probably add
>> that to the PDL shell output)
>>
>>> It’s horribly designed in that regard. The software itself is great, and
>>> I’m very happy with the results, but finding the simplest little thing in
>>> the docs can be a total pain!
>>
>> I understand frustration. What would really help PDL
>> development would be to know how you got to using
>> PDL without being introduced to the concepts that would
>> have made the learning curve less steep. (At least you
>> found the mailing list and used it. :-)
>>
>> Happy PDL-ing!
>> Chris
_______________________________________________
Perldl mailing list
Perldl@jach.hawaii.edu
http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
