It sounds like M is equal to the number of processors (pipelines) and M >> N. Is that correct?
-berk On Fri, Dec 11, 2009 at 10:40 AM, Biddiscombe, John A. <[email protected]> wrote: > Berk > > The data will be UnstructuredGrid for now. Multiblock, but actually, I don't > really care what each block is, only that I accept one block on each of N > processes, split it into more pieces, and the next filter accepts one (or > more if the numbers don't match up nicely) blocks and process them. The > redistribution shouldn't care what data types, only how many blocks in and > out. > > Looking at RedistributePolyData makes me realize my initial idea is no good. > In my mind I had a pipeline where multiblock datasets are passed down the > pipeline and simply the number of pieces is manipulated to achieve what I > wanted - but I see now that if I have M pieces downstream mapped upstream to > N pieces, what will happen is the readers will be effectively duplicated and > M/N readers will read the same pieces. I don't want this to happen as IO will > be a big problem if readers read the same blocks M/N times. > I was hoping there was a way of simply instructing the pipeline to manage the > pieces, but I see now that this won't work, as there needs to be a specific > Send from each N to their M/N receivers (because the data is physically in > another process, so the pipeline can't see it). This is very annoying as > there must be a class which already does this (block redistribution, rather > than polygon level redistribution), and I would like it to be more 'pipeline > integrated' so that the user doesn't have to explicitly send each time an > algorithm needs it. > > I'll go through RedistributePolyData in depth and see what I can pull out of > it - please feel free to steer me towards another possibility :) > > JB > > >> -----Original Message----- >> From: Berk Geveci [mailto:[email protected]] >> Sent: 11 December 2009 16:09 >> To: Biddiscombe, John A. >> Cc: [email protected] >> Subject: Re: [Paraview] Parallel Data Redistribution >> >> What is the data type? vtkRedistributePolyData and its subclasses do >> this for polydata. It can do load balancing (where you can specify a >> weight for each processor) as well. >> >> -berk >> >> On Fri, Dec 11, 2009 at 9:59 AM, Biddiscombe, John A. <[email protected]> >> wrote: >> > I have a filter pipeline which reads N blocks from disk, this works fine >> on N processors. >> > >> > I now wish to subdivide those N blocks (using a custom filter) to produce >> new data which will consist of M blocks - where M >> N. >> > >> > I wish to run the algorithm on M processors and have the piece information >> transformed between the two filters (reader -> splitter), so that blocks are >> distributed correctly. The reader will Read N blocks (leaving M-N processes >> unoccupied), but the filter which splits them up needs to output a different >> number of pieces and have the full M processes receiving data. >> > >> > I have a reasonably good idea of how to implement this, but I'm wondering >> if any filters already do something similar. I will of course take apart the >> D3 filter for ideas, but I don't need to do a parallel spatial decomposition >> since my blocks are already discrete - I just want to redistribute the >> blocks around and more importantly change the numbers of them between >> filters. >> > >> > If anyone can suggest examples which do this already, please do >> > >> > Thanks >> > >> > JB >> > >> > -- >> > John Biddiscombe, email:biddisco @ cscs.ch >> > http://www.cscs.ch/ >> > CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07 >> > Via Cantonale, 6928 Manno, Switzerland | Fax: +41 (91) 610.82.82 >> > >> > >> > _______________________________________________ >> > Powered by www.kitware.com >> > >> > Visit other Kitware open-source projects at >> http://www.kitware.com/opensource/opensource.html >> > >> > Please keep messages on-topic and check the ParaView Wiki at: >> http://paraview.org/Wiki/ParaView >> > >> > Follow this link to subscribe/unsubscribe: >> > http://www.paraview.org/mailman/listinfo/paraview >> > > _______________________________________________ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView Follow this link to subscribe/unsubscribe: http://www.paraview.org/mailman/listinfo/paraview
