In the past we had the pipeline execution stop if update number of pieces was 0. We can easily add that back in. It would be a one line change in vtkStreamingDemandDrivePipeline::NeedToExecuteData().
Alternatively, the reader could check for UPDATE_NUMBER_OF_PIECES and do nothing it is is 0. I think that's better that requested piece being -1 which is ambiguous. -berk On Tue, Dec 15, 2009 at 2:41 PM, Moreland, Kenneth <[email protected]> wrote: > I think the part that John wants that is missing is the ability for a filter > to say I want no data (or simply, I don’t care). The idea being that during > the RequestUpdateExtent phase you could send a flag to tell the pipeline to > not propagate the request upstream. Instead, simply call RequestData and > propagate back down. However, like Berk said, this is a bad idea because > any upstream objects that do parallel communication will lock up. I would > hope that your reader itself does parallel communication. You’re going to > get contention on your disks otherwise as different processes request the > same pieces. > > I suppose the filter could also simply request piece –1 and the reader could > take that as a flag to read nothing. Assuming neither the executive or > another filter tries to do anything with the piece number you should be OK, > but I don’t know if that will just work. > > -Ken > > > On 12/15/09 9:21 AM, "Berk Geveci" <[email protected]> wrote: > > Hmmm. I don't exactly see the difference between the two (I mean > streaming and distributing). In your case, I would think that a filter > can arbitrarily change the pipeline request to either request nothing > or ask the same piece on two processes. It would like this: > > reader0 - request piece 0 of 1 - filter - request piece 0 of 1 - > update suppressor > reader1 - request piece 0 of 1 - filter - request piece 1 of 1 - > update suppressor > > This should work fine as long as the reader does not have another > consumer (for example a representation if the visibility is on) that > is asking for a different piece. The following should also work > > reader0 - request piece 0 of 1 - filter - request piece 0 of 1 - > update suppressor > reader1 - request piece 0 of 0 - filter - request piece 1 of 1 - > update suppressor > > Of course, if you have any filters that do parallel communication in > between, you are SOL. > > Does that make sense? > > Now if you have structured data and the frickin' extent translator > gets in between, things are not as straightforward - which why I hate > the extent translator. If I had some time/funding, I would write a new > executive that does not use structured extents. > > -berk > > On Tue, Dec 15, 2009 at 11:07 AM, Biddiscombe, John A. <[email protected]> > wrote: >> Berk, >> >> We had a discussion back in 2008, which resides here >> http://www.cmake.org/pipermail/paraview/2008-May/008170.html >> >> Continuing from this, my question of the other day, touches on the same >> problem. >> >> I'd like to manipulate the piece number read by each reader. As mentioned >> before, UPDATE_PIECE is not passed into RequestInformation at first (since >> nobody knows how many pieces there are yet!), so I can't (directly) generate >> information in the reader which is 'piece dependent'. And I can't be sure >> that someone doing streaming won't interfere with piece numbers when using >> the code differently. >> >> For the particle tracer (for example), I'd like to tell the upstream >> pipeline to read no pieces when certain processes are empty of particles >> (currently they update and generate{=read} data when they don't need to). I >> may be able to suppress the forward upstream somehow, but I don't know of an >> easy way for the algorithm to say "Stop" to the executive to prevent it >> updating if the timestep changes, but the algorithm has determined that no >> processing is required (ForwardUpstream of Requests continues unabated). I'd >> like to set the UPdatePiece to -1 to tell the executive to stop operating. >> >> Also : for dynamic load balancing, I'd like to instruct several reader to >> read the same piece - since the algorithm controls (for example) the >> particles the algorithm can internally communicate information about what to >> do amongst its processes, but it can't talk upstream to the readers and >> fudge them. >> >> I am wondering if there is any way of supporting this kind of thing using >> the current information keys and my instinct says no. It seems like the >> update pice and numpieces were really intended for streaming and we need two >> kinds of 'pieces', one for streaming, another for splitting in _parallel_ >> because they aren't quite the same. (Please note that I haven't actually >> tried changing piece requests in the algorithms yet, so I'm only guessing >> that it won't work properly) >> >> <cough> >> UPDATE_STREAM_PIECE >> UPDATE_PARALLEL_PIECE >> <\cough> >> >> Comments? >> >> JB >> >> >>> >>> I would have the reader (most parallel readers do this) generate empty >>> data on all processes of id >= N. Then your filter can redistribute >>> from those N processes to all M processes. I am pretty sure >>> RedistributePolyData can do this for polydata as long as you set the >>> weight to 1 on all processes. Ditto for D3. >>> >>> -berk >>> >>> On Fri, Dec 11, 2009 at 4:13 PM, Biddiscombe, John A. <[email protected]> >>> wrote: >>> > Berk >>> > >>> >> It sounds like M is equal to the number of processors (pipelines) and >>> >> M >> N. Is that correct? >>> > >>> > Yes, That's the idea. N blocks, broken (in place) into M new blocks, >>> > then >>> fanned out to the M processes downstream where they can be processed >>> separately . If it were on a single node, then each block could be a >>> separate 'connection' to a downstream filter, but distributed, an >>> explicit >>> send is needed. >>> > >>> > JB >>> > >>> >> >>> >> -berk >>> >> >>> >> On Fri, Dec 11, 2009 at 10:40 AM, Biddiscombe, John A. >>> >> <[email protected]> >>> >> wrote: >>> >> > Berk >>> >> > >>> >> > The data will be UnstructuredGrid for now. Multiblock, but actually, >>> >> > I >>> >> don't really care what each block is, only that I accept one block on >>> each >>> >> of N processes, split it into more pieces, and the next filter accepts >>> one >>> >> (or more if the numbers don't match up nicely) blocks and process >>> >> them. >>> The >>> >> redistribution shouldn't care what data types, only how many blocks in >>> and >>> >> out. >>> >> > >>> >> > Looking at RedistributePolyData makes me realize my initial idea is >>> >> > no >>> >> good. In my mind I had a pipeline where multiblock datasets are passed >>> down >>> >> the pipeline and simply the number of pieces is manipulated to achieve >>> what >>> >> I wanted - but I see now that if I have M pieces downstream mapped >>> upstream >>> >> to N pieces, what will happen is the readers will be effectively >>> duplicated >>> >> and M/N readers will read the same pieces. I don't want this to happen >>> >> as >>> IO >>> >> will be a big problem if readers read the same blocks M/N times. >>> >> > I was hoping there was a way of simply instructing the pipeline to >>> manage >>> >> the pieces, but I see now that this won't work, as there needs to be a >>> >> specific Send from each N to their M/N receivers (because the data is >>> >> physically in another process, so the pipeline can't see it). This is >>> very >>> >> annoying as there must be a class which already does this (block >>> >> redistribution, rather than polygon level redistribution), and I would >>> like >>> >> it to be more 'pipeline integrated' so that the user doesn't have to >>> >> explicitly send each time an algorithm needs it. >>> >> > >>> >> > I'll go through RedistributePolyData in depth and see what I can >>> >> > pull >>> out >>> >> of it - please feel free to steer me towards another possibility :) >>> >> > >>> >> > JB >>> >> > >>> >> > >>> >> >> -----Original Message----- >>> >> >> From: Berk Geveci [mailto:[email protected]] >>> >> >> Sent: 11 December 2009 16:09 >>> >> >> To: Biddiscombe, John A. >>> >> >> Cc: [email protected] >>> >> >> Subject: Re: [Paraview] Parallel Data Redistribution >>> >> >> >>> >> >> What is the data type? vtkRedistributePolyData and its subclasses >>> >> >> do >>> >> >> this for polydata. It can do load balancing (where you can specify >>> >> >> a >>> >> >> weight for each processor) as well. >>> >> >> >>> >> >> -berk >>> >> >> >>> >> >> On Fri, Dec 11, 2009 at 9:59 AM, Biddiscombe, John A. >>> <[email protected]> >>> >> >> wrote: >>> >> >> > I have a filter pipeline which reads N blocks from disk, this >>> >> >> > works >>> >> fine >>> >> >> on N processors. >>> >> >> > >>> >> >> > I now wish to subdivide those N blocks (using a custom filter) to >>> >> produce >>> >> >> new data which will consist of M blocks - where M >> N. >>> >> >> > >>> >> >> > I wish to run the algorithm on M processors and have the piece >>> >> information >>> >> >> transformed between the two filters (reader -> splitter), so that >>> blocks >>> >> are >>> >> >> distributed correctly. The reader will Read N blocks (leaving M-N >>> >> processes >>> >> >> unoccupied), but the filter which splits them up needs to output a >>> >> different >>> >> >> number of pieces and have the full M processes receiving data. >>> >> >> > >>> >> >> > I have a reasonably good idea of how to implement this, but I'm >>> >> wondering >>> >> >> if any filters already do something similar. I will of course take >>> apart >>> >> the >>> >> >> D3 filter for ideas, but I don't need to do a parallel spatial >>> >> decomposition >>> >> >> since my blocks are already discrete - I just want to redistribute >>> >> >> the >>> >> >> blocks around and more importantly change the numbers of them >>> >> >> between >>> >> >> filters. >>> >> >> > >>> >> >> > If anyone can suggest examples which do this already, please do >>> >> >> > >>> >> >> > Thanks >>> >> >> > >>> >> >> > JB >>> >> >> > >>> >> >> > -- >>> >> >> > John Biddiscombe, email:biddisco @ >>> cscs.ch >>> >> >> > http://www.cscs.ch/ >>> >> >> > CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) >>> 610.82.07 >>> >> >> > Via Cantonale, 6928 Manno, Switzerland | Fax: +41 (91) >>> 610.82.82 >>> >> >> > >>> >> >> > >>> >> >> > _______________________________________________ >>> >> >> > Powered by www.kitware.com >>> >> >> > >>> >> >> > Visit other Kitware open-source projects at >>> >> >> http://www.kitware.com/opensource/opensource.html >>> >> >> > >>> >> >> > Please keep messages on-topic and check the ParaView Wiki at: >>> >> >> http://paraview.org/Wiki/ParaView >>> >> >> > >>> >> >> > Follow this link to subscribe/unsubscribe: >>> >> >> > http://www.paraview.org/mailman/listinfo/paraview >>> >> >> > >>> >> > >>> > >> > _______________________________________________ > Powered by www.kitware.com > > Visit other Kitware open-source projects at > http://www.kitware.com/opensource/opensource.html > > Please keep messages on-topic and check the ParaView Wiki at: > http://paraview.org/Wiki/ParaView > > Follow this link to subscribe/unsubscribe: > http://www.paraview.org/mailman/listinfo/paraview > > > > > **** Kenneth Moreland > *** Sandia National Laboratories > *********** > *** *** *** email: [email protected] > ** *** ** phone: (505) 844-8919 > *** web: http://www.cs.unm.edu/~kmorel > > _______________________________________________ Powered by www.kitware.com Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html Please keep messages on-topic and check the ParaView Wiki at: http://paraview.org/Wiki/ParaView Follow this link to subscribe/unsubscribe: http://www.paraview.org/mailman/listinfo/paraview
