Yes, Amol – basically wanted to know how to sensibly handle the case where there’s a big back-buffer to work through.
- Ilya Ganelin [id:image001.png@01D1F7A4.F3D42980] From: Amol Kekre <a...@datatorrent.com> Reply-To: "users@apex.apache.org" <users@apex.apache.org> Date: Thursday, April 20, 2017 at 4:27 PM To: "users@apex.apache.org" <users@apex.apache.org> Subject: Re: Wait until setup completes prior to dag execution Ilya, Additionally that particular operator will not process data till setup is done. As Pramod pointed out, there is no reason to wait. At worse the buffer-servers will fill up. Are you worried about the operator code need to ramp up in processing data and not get a deluge of it? Thks Amol E:a...@datatorrent.com<mailto:e%3aa...@datatorrent.com> | M: 510-449-2606 | Twitter: @amolhkekre www.datatorrent.com<http://www.datatorrent.com> [https://lh4.googleusercontent.com/NK58t6MfZJkiU9pOu4-gdCwysbYuEM4Aj1lfdg76_SfZZPEsV7kFgaITKVXUFm71JH9b5uI7NjBKItlLgbvADGbJ3dhQpbEehVGLf_tahv7gMR8Z8BooXNiYwjAJvrzptvcxSIwg] On Thu, Apr 20, 2017 at 4:19 PM, Pramod Immaneni <pra...@datatorrent.com<mailto:pra...@datatorrent.com>> wrote: Hi, I am assuming it is an operator that is not the input operator that has the lengthy setup function? Any specific reason you don't want to let data in because things don't get committed (like kafka offsets) till all operators process the data. If you are worried about input operators are going to run-away there is back pressure which will put breaks on when the buffers are full. Thanks On Thu, Apr 20, 2017 at 3:36 PM, Ganelin, Ilya <ilya.gane...@capitalone.com<mailto:ilya.gane...@capitalone.com>> wrote: Is there any way to wait for setup to complete prior to allowing data to flow through the DAG? I have an operator with a lengthy setup function so I’m just wondering what the best way to handle this is. My best solution at the moment is to toggle start with an emission rate of zero and toggle it on once the DAG is launched. - Ilya Ganelin [cid:image002.png@01D2BA9B.E6EA7CF0] ________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer. ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
