Yes that will work too if they can be parallel partitioned. Thks Amol
E:a...@datatorrent.com | M: 510-449-2606 | Twitter: @*amolhkekre* www.datatorrent.com On Tue, Apr 25, 2017 at 8:54 PM, Bhupesh Chawda <bhup...@datatorrent.com> wrote: > If the operators are not very heavy, perhaps thread local could work.. > > ~ Bhupesh > > > _______________________________________________________ > > Bhupesh Chawda > > E: bhup...@datatorrent.com | Twitter: @bhupeshsc > > www.datatorrent.com | apex.apache.org > > > > On Sat, Apr 22, 2017 at 11:04 PM, Amol Kekre <a...@datatorrent.com> wrote: > >> >> This could be solved by throttling ramp up. We have cases in production >> that do this. If throttling at the input operator is there, then it is a >> simple matter of communicating it via software. >> >> Lets assume the input operator(s) can be throttled from 0% to 100%. They >> all they need to know is the throttle number. Since this comes from >> downstream, you can figure out ways of communicating back. Some ideas (just >> brainstorming) >> 1. Look at commit window to know all operators are past setup (? Can an >> operator know about commit window?) >> 2. Communicate via other mechanism -> Write to a file (setup done, ramp >> up now) >> 3. Use statListner >> 4. Vlad' solution >> etc. >> >> I think #3 may work well. Additionally it can be tuned to also help >> resolve in run-time "how much can the input ingestion operator at". Via #3 >> this can be dynamic, and additionally the dag remains the same, it just >> fine-tunes live. #4 can also do this by passing back meta data on the >> feedback port. >> >> Thks >> Amol >> >> >> >> E:a...@datatorrent.com | M: 510-449-2606 <(510)%20449-2606> | Twitter: @ >> *amolhkekre* >> >> www.datatorrent.com >> >> >> On Fri, Apr 21, 2017 at 7:15 PM, Vlad Rozov <v.ro...@datatorrent.com> >> wrote: >> >>> One possible way is to introduce a delay operator. I am not 100% that it >>> will work, but it is worth trying. Introduce a dummy output port on the >>> operator that takes too long to setup. Connect it to an input port of a >>> delay operator. Connect delay operator output port to a dummy input port of >>> the upstream operator. In the upstream operator do not emit on the actual >>> output port till a tuple is received on the dummy input port. In the >>> downstream operator emit a tuple to the dummy output port in the first >>> beginWindow(). >>> >>> Thank you, >>> >>> Vlad >>> >>> On 4/21/17 12:36, Ganelin, Ilya wrote: >>> >>> Yes, Amol – basically wanted to know how to sensibly handle the case >>> where there’s a big back-buffer to work through. >>> >>> >>> >>> - Ilya Ganelin >>> >>> [image: id:image001.png@01D1F7A4.F3D42980] >>> >>> >>> >>> *From: *Amol Kekre <a...@datatorrent.com> <a...@datatorrent.com> >>> *Reply-To: *"users@apex.apache.org" <users@apex.apache.org> >>> <users@apex.apache.org> <users@apex.apache.org> >>> *Date: *Thursday, April 20, 2017 at 4:27 PM >>> *To: *"users@apex.apache.org" <users@apex.apache.org> >>> <users@apex.apache.org> <users@apex.apache.org> >>> *Subject: *Re: Wait until setup completes prior to dag execution >>> >>> >>> >>> >>> >>> Ilya, >>> >>> Additionally that particular operator will not process data till setup >>> is done. As Pramod pointed out, there is no reason to wait. At worse the >>> buffer-servers will fill up. Are you worried about the operator code need >>> to ramp up in processing data and not get a deluge of it? >>> >>> >>> >>> Thks >>> >>> Amol >>> >>> >>> >>> >>> >>> >>> E:a...@datatorrent.com | M: 510-449-2606 <(510)%20449-2606> | Twitter: @ >>> *amolhkekre* >>> >>> www.datatorrent.com >>> >>> >>> >>> On Thu, Apr 20, 2017 at 4:19 PM, Pramod Immaneni <pra...@datatorrent.com> >>> wrote: >>> >>> Hi, >>> >>> >>> >>> I am assuming it is an operator that is not the input operator that has >>> the lengthy setup function? Any specific reason you don't want to let data >>> in because things don't get committed (like kafka offsets) till all >>> operators process the data. If you are worried about input operators are >>> going to run-away there is back pressure which will put breaks on when the >>> buffers are full. >>> >>> >>> >>> Thanks >>> >>> >>> >>> >>> >>> On Thu, Apr 20, 2017 at 3:36 PM, Ganelin, Ilya < >>> ilya.gane...@capitalone.com> wrote: >>> >>> Is there any way to wait for setup to complete prior to allowing data to >>> flow through the DAG? I have an operator with a lengthy setup function so >>> I’m just wondering what the best way to handle this is. My best solution at >>> the moment is to toggle start with an emission rate of zero and toggle it >>> on once the DAG is launched. >>> >>> >>> >>> - Ilya Ganelin >>> >>> >>> ------------------------------ >>> >>> The information contained in this e-mail is confidential and/or >>> proprietary to Capital One and/or its affiliates and may only be used >>> solely in performance of work or services for Capital One. The information >>> transmitted herewith is intended only for use by the individual or entity >>> to which it is addressed. If the reader of this message is not the intended >>> recipient, you are hereby notified that any review, retransmission, >>> dissemination, distribution, copying or other use of, or taking of any >>> action in reliance upon this information is strictly prohibited. If you >>> have received this communication in error, please contact the sender and >>> delete the material from your computer. >>> >>> >>> >>> >>> >>> ------------------------------ >>> >>> The information contained in this e-mail is confidential and/or >>> proprietary to Capital One and/or its affiliates and may only be used >>> solely in performance of work or services for Capital One. The information >>> transmitted herewith is intended only for use by the individual or entity >>> to which it is addressed. If the reader of this message is not the intended >>> recipient, you are hereby notified that any review, retransmission, >>> dissemination, distribution, copying or other use of, or taking of any >>> action in reliance upon this information is strictly prohibited. If you >>> have received this communication in error, please contact the sender and >>> delete the material from your computer. >>> >>> >>> >> >
