Hi Dave, You should be able to do this by sending a Notify with Signal Counter Delta = 0 immediately after the Wait to close the gate and then another Notify with Signal Counter Delta = 1 after you have finished processing to reopen the gate. You will need to create a separate Notify with Signal Counter Delta = 1 that you will need to run once manually to start the process, otherwise everything will get stuck at the Wait processor as the counter will start at 0 (closed) when it needs to be 1 (open).
Thanks, Evan On Tue, May 28, 2019 at 8:44 PM David Gallagher < [email protected]> wrote: > Thanks, Shawn – I had seen your post to the mailing list and that’s where > I got the gist. It’s really close to what I need to do, but it seems to be > designed for a scenario where the data is arriving on a given interval. In > my case, I could have multiple messages arriving at the Wait processor > simultaneously. In my testing, if 3 messages are queued, all 3 will be > passed through the gate immediately once the gate is open. What I would > really want is for each message to close the gate, and for the gate to stay > closed until that message has been processed. I’m racking my brain trying > to think of a way to do that. I’ve also tried to implement the 0/1 gate > that’s mentioned in the documentation, but I’m not sure how it works. If I > set the target count to 0, all wait files pass through immediately. I tried > setting the ‘Releasable FlowFile Count’ to 0 as well, but that didn’t work > either. > > > > Thanks, > > > > Dave > > > > *From:* Shawn Weeks <[email protected]> > *Sent:* Saturday, May 18, 2019 8:46 AM > *To:* [email protected] > *Subject:* Re: Open / Close Gate examples? > > > > See this example as I had a lot of questions about wait and notify earlier > and this helped a lot. > > > > https://gist.github.com/ijokarumawak/9e1a4855934f2bb9661f88ca625bd244 > <https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Fijokarumawak%2F9e1a4855934f2bb9661f88ca625bd244&data=02%7C01%7Cdgallagher%40cleverdevices.com%7Cc2bb5f1ef6b442815ce908d6db8ec4ab%7Cc24ec497266d4560bcda1d93096d9d18%7C0%7C0%7C636937803587114486&sdata=5EhJf0%2FvM%2F4VPKe0z916zcHyVx5UpgzCttu2yUZbCbk%3D&reserved=0> > > > > Thanks > > Shawn > > Sent from my iPhone > > > On May 17, 2019, at 1:59 PM, David Gallagher <[email protected]> > wrote: > > Hi Joe – thanks for answering. I can actually live with the message ack at > the moment. But, I can’t have two messages at once because the underlying > process will deadlock if two flows try to use it at the same time. This is > a problem because my nifi node might be brought up after having been shut > off for a few days, at which point it will read multiple messages off the > queue and attempt to process them. The problem I’m having is getting flow > files to wait for each other. If I use a Notify processor to set a release > signal as indicated in the linked gist, it does release the Wait as > designed. But, the next flowfile then enters the queue and it will get > released when the Notify is triggered again by the GenerateFlowFile. What I > really want to have happen is for each flowfile to block the next flowfile > from entering until it has exited the process. I’m not really sure how to > set that up successfully… > > > > Thanks, > > > > Dave > > > > *From:* Joe Witt <[email protected]> > *Sent:* Friday, May 17, 2019 12:33 PM > *To:* [email protected] > *Subject:* Re: Open / Close Gate examples? > > > > Dave > > > > Using Wait/Notify would ensure you only have one message in flight at a > time (or it can/should). But the message will be ack'd before processed. > > > > For Kafka and some of these message queue mechanisms if we want to offer a > 'do not ack until the whole flow is done' behavior we should update the > processors to allow for that and actually this is a great use case for the > nifi-fn/stateless work that SamH in the community has been talking about. > > > > Thanks > > > > On Fri, May 17, 2019 at 12:30 PM David Gallagher < > [email protected]> wrote: > > Hi – I have a requirement to read one message off RabbitMQ and fully > process that message before I take another message off the queue. I found > some great guidance to setting up a ‘traffic light’ pattern here ( > https://gist.github.com/ijokarumawak/9e1a4855934f2bb9661f88ca625bd244 > <https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgist.github.com%2Fijokarumawak%2F9e1a4855934f2bb9661f88ca625bd244&data=02%7C01%7Cdgallagher%40cleverdevices.com%7Cc2bb5f1ef6b442815ce908d6db8ec4ab%7Cc24ec497266d4560bcda1d93096d9d18%7C0%7C0%7C636937803587114486&sdata=5EhJf0%2FvM%2F4VPKe0z916zcHyVx5UpgzCttu2yUZbCbk%3D&reserved=0>), > but it won’t work for my process because the Notify will get triggered by > the GenerateFlowFile before the long-running process triggers, which will > put two or more messages into the process at once. The documentation for > Notify makes reference to using a 0 value in Signal Counter Delta to > simulate ‘open-close-gate’ flow control, which sounds like what I’m looking > for. Can someone point me in the right direction? > > > > Thanks, > > > > Dave > > > > > >
