Chris, Thank you for your detailed reply.
You mentioned that if I wanted to perfrom many 2D convolution operations in parallel it would suffice to spawn many Perl threads (using Perl's "threads") and to send each one off with a convolution task involving a different image. I believe the code posted in the original email did just this. Do you know why that implementation fails when I use convolveND to perform the convolutions? The same implementation does not fail if you comment out the convolveND part and use conv2d. I'm going to see if I can put my images in a 3D piddle and implement the other solution you suggested, but I would still like to find out why I cannot make convolution using convolveND parallel... Thanks, Perldough 2012/10/16 David Mertens <[email protected]> > Grasswistle - > > You said you needed to process lots of images in "very little time" at the > start of this email thread. Does this mean you were facing a deadline, or > that you need high throughput? You also said that you needed a convolution, > but you didn't say what sort. In particular, all your code works with > images, which are two-dimensional. Are you trying to do 2D convolutions, or > higher dimensional ones? If higher dimensional ones, I suspect that > convolveND allocates some work space, but it is not thread-specific work > space; as the first thread exits, it frees the memory so that the other > threads suffer a segmentation fault when they try to use this global memory > that is no longer allocated. I bet that if we modified the code so that it > created thread-specific work space, it would work, but it's not clear to > me, having glanced through the code, where those modifications need to be > made. > > Performing 2D convolutions on many images is the sort of problem that is > considered "embarrassingly parallel". That is, parallelizing the code is > simply a matter of sending the same job with different data to different > processors. When parallelized, the processing time is cut down by the > number of processors at hand, so instead of taking T time, it takes T/N > time (N being the number of processors). Not all problems are > embarrassingly parallel. In particular, scaling algorithms to very large > sets of data across many processors is typically *not* embarrassingly > parallel and require careful thought to do correctly. > > At any rate, if you are performing 2D convolutions on images that fit well > within memory, it seems to me that you have two options, both of which have > already been discussed in this thread. The first is to spawn many Perl > threads using Perl's "threads" module and have each of them perform 2D > convolutions on your images. In doing this, you would parallelize your code > by hand. Whatever you do, you should not try to share a piddle across > multiple threads. This is not supported with normal PDL, and unfortunately > it does not emit any sensible warnings if you try to do this. (That's > something I've meant to work on since this summer, when I wrote > PDL::Parallel::threads <https://github.com/run4flat/PDL-Parallel-threads>.) > The other solution is to load all the images into a single 3D piddle (or > batches at a time, if there is not enough memory to load them all at once), > and calling conv2d as Chris describes to get the the pthread > parallelization automatically. > > So, everything I've said in the above paragraphs applies to > two-dimensional convolutions. If you actually need higher-dimensional > convolutions, I'll have to reconsider. > > David > > -- > "Debugging is twice as hard as writing the code in the first place. > Therefore, if you write the code as cleverly as possible, you are, > by definition, not smart enough to debug it." -- Brian Kernighan >
_______________________________________________ Perldl mailing list [email protected] http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
