---------- Forwarded message ---------- From: Chris Marshall <[email protected]> Date: Wed, Nov 19, 2014 at 4:23 PM Subject: Re: [Perldl] matching vectors inside a PDL To: "LYONS, KENNETH B (KENNETH)" <[email protected]>
<re-cc-ing the perldl list> Thanks for the background. If you hit a snag, feel free to post to the perldl list. We're usually able to help for specific problems especially if accompanied by code demonstrating the problem: If I have a big byte piddle $a and I multiply it in-place my PDL session crashes because of a huge intermediate temp: pdl> $a = (10*random(100))->floor->byte; pdl> $a->inplace->mult(5,0); Error message here or crash Without a specific example, I would guess that the problem is the piddle you are multiplying by (or perl scalar) is of type double which would result in an intermediate temp of double type which would then collapse down to a byte piddle again at the end. If both arguments to multiply are of byte type, you can avoid the big double intermediate temp. E.g. pdl> p pdl(5)->type double pdl> p byte(5)->type byte Improved type support is planned for the PDL3 work. My initial ideas for bitfield support can be seen here: http://mailman.jach.hawaii.edu/pipermail/pdl-porters/2013-December/006132.html Hope this helps, Chris On Wed, Nov 19, 2014 at 1:42 PM, LYONS, KENNETH B (KENNETH) <[email protected]> wrote: > Chris > > In answer to your question: my path in was as follows: I wanted to find a > way to implement an LP on a medium-size problem (<~10K variables), and the > rest of my code was in perl--so I went looking for an LP implementation in > perl. I was expecting to find a C-compiled module that would do an LP > specifically. I found some instances of that sort of thing, but I also ran > across one using PDL. It didn't do quite what I wanted, but when I saw the > PDL site, it was obvious this was something I needed to know about. I wound > up writing my own simplex implementation in PDL to do specifically what I > needed, and that worked great--and I was pretty blown away at the speed. So > then I started looking into how I could back up and get the datasets I was > dealing with implemented as PDLs to start with. So I've got a good bit of > code now using PDL, not just the little simplex program (which was only a few > dozen lines--that was pretty easy to implement in PDL.) > > I continue to have issues with the documentation, though. Just as one > example from today: the mult function seems to claim that you can get it to > operate in-place. And for me that was important, because I'm dealing with a > large dataset (of byte variables). But, not only does "mult" by itself cause > an error because it isn't exported, but when I try to use it as > PDL::Ops::Mult(inplace ...), or PDL::Mult(inplace ...), or as > $piddle->inplace->mult(...), it completely fails to avoid generating a large > intermediate. That was clobbering my program, repetitively, so I finally > punted and decided to break that step up into segments with only a few > hundred thousand elements to multiply in each (using slices), and that got me > around the problem. But there was nothing in the documentation that seemed > to suggest that would be necessary. It also seemed, although I didn't > document this carefully, that changing the default PDL type didn't have any > impact on the size of that temporary intermediate (I think it was using > double no matter what I did--whereas using byte would have been fine.) > > I'd love it, in this context, if there were a PDL type of "bit" by the way, > since that's actually what this problem is using--it's a 3D binary matrix, of > ones and zeroes, with up to ~3*10^^9 elements. When the number of elements > goes above ~200M, when I'm using bytes, I have to do things to break it up > and process one segment at a time, and it would be nice if that weren't > necessary--but there is evidently no implementation of a "bit" type in PDL. > > Ken > > > -----Original Message----- > From: Chris Marshall [mailto:[email protected]] > Sent: Saturday, November 15, 2014 11:42 AM > To: Derek Lamb > Cc: LYONS, KENNETH B (KENNETH); perldl > Subject: Re: [Perldl] matching vectors inside a PDL > > Hi Ken and welcome to the PDL community! > >> On Nov 14, 2014, at 1:33 PM, LYONS, KENNETH B (KENNETH) >> <[email protected]> wrote: >> >> Yes, most of this I knew, but thanks. It’s because of that behavior of > >> and <, that you mentioned, that I thought that ‘==’ would compare element by >> element instead of on the whole vector. >> >> Have you ever tried, for example, to search the documentation for, say, the >> function “list”? it gives you every occurrence of the word “list” in the >> documents (which, needless to say, is rather voluminous, and the first few >> hundred entries have nothing to do with the function!) there should be some >> analog of the “man” command in unix that gives you information about the >> *function* without all the other garbage. I think it’s just doing something >> akin to a grep thru the documents. > > In addition to the help/? and apropos/?? in the PDL shells > (pdl2 and perldl) there is the command line version pdldoc > which can be used starting with 'pdldoc pdldoc' to get the usage. > > When I do 'pdldoc -a list' I get 45 lines of output all > of whose descriptions seem relevant to a general search > for something having to do with 'list' including the 'list' > command itself. This type of problem is not specific to > PDL as searching the docs for any complicated system > or program does tend to produce a large number of > incomprehensible and not particularly useful results. > > It is definitely desired to have smarter and more useful > documentation searches. The ability to add keywords > would be nice and is on the feature request list, I believe. > > If you are on unix-ish system, you might try something > like 'pdldoc -a list | grep --color list' to make the output > more visually comprehensible. (We should probably add > that to the PDL shell output) > >> It’s horribly designed in that regard. The software itself is great, and >> I’m very happy with the results, but finding the simplest little thing in >> the docs can be a total pain! > > I understand frustration. What would really help PDL > development would be to know how you got to using > PDL without being introduced to the concepts that would > have made the learning curve less steep. (At least you > found the mailing list and used it. :-) > > Happy PDL-ing! > Chris _______________________________________________ Perldl mailing list [email protected] http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
