I have a single spout instance so that's not a problem. I planned to do this control from another spout listening on a "control" channel, so to speak. If I block on nextTuple I will not get acks from the previous messages to that spout.
Now that you mention it, if I don't have the two spouts in the same worker it won't work. Using an external store like zk sounds like a workable solution. Thanks! Regards, Jg On Aug 14, 2015 3:14 PM, "Kishore Senji" <[email protected]> wrote: > That only works if you want to control a bolt. You cannot control a Spout > from another spout, that spout being part of the topology. What you can do > is that, you have to embed the Kafka consumer to look up the messages from > the control topic and block nextTuple. You also need to store offsets and > make sure it works for every instance of your main spout. This is the > reason I think Zk solution is better. > On Fri, Aug 14, 2015 at 12:05 PM Javier Gonzalez <[email protected]> > wrote: > >> I was thinking of using another spout as "control channel", and from that >> spout manipulate the original spout to cause the nextTuple method to not >> call the next message from the incoming queue (but not block so that acks >> can be processed). A second signal on that spout can manipulate it back to >> BAU. >> >> Thanks, >> Javier >> On Aug 14, 2015 2:59 PM, "Kishore Senji" <[email protected]> wrote: >> >>> This is exactly what is done on deactivate of topology. Storm has rest >>> api to deactivate and you can use that to integrate with your external >>> system. Otherwise you can extend your spout and use Zk to signal it. On >>> receiving signal it can block and unblock appropriately the nextTuple() call >>> On Fri, Aug 14, 2015 at 5:34 AM Javier Gonzalez <[email protected]> >>> wrote: >>> >>>> I'm trying to ensure everything is processed for coordination with an >>>> external system. Therefore, on a given signal, I have to stop the spout, >>>> wait for the acks of all pending/inflight messages, then resume the spout >>>> intake on a second signal. >>>> On Aug 13, 2015 10:01 PM, "Kishore Senji" <[email protected]> wrote: >>>> >>>>> The deactivate() on the spout will be called when the topology is >>>>> deactivated. Below is the output of storm deactivate help. This will >>>>> deactivate the whole topology and this can be from the Storm UI as well. >>>>> Does you topology have more than one Spout and are you looking to stop >>>>> only >>>>> a subset of those Spouts? >>>>> >>>>> $ storm help deactivate >>>>> Syntax: [storm deactivate topology-name] >>>>> >>>>> Deactivates the specified topology's spouts. >>>>> >>>>> >>>>> >>>>> On Thu, Aug 13, 2015 at 2:12 PM Javier Gonzalez <[email protected]> >>>>> wrote: >>>>> >>>>>> On a more broader term, can you share the strategies you've used to >>>>>> pause (not emit anything else into the topology and not read anything >>>>>> else >>>>>> from the data source) a topology's spouts? >>>>>> >>>>>> Thanks, >>>>>> Javier >>>>>> On Aug 13, 2015 2:53 PM, "Javier Gonzalez" <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I have a use case where I would need to stop a spout from emitting >>>>>>> for a period of time. I'm looking at the activate /deactivate methods, >>>>>>> but >>>>>>> there's not much information apart from the API and the java base >>>>>>> classes >>>>>>> have empty implementations. Can anybody shed any insight on how those >>>>>>> work? >>>>>>> >>>>>>> Thanks, >>>>>>> Javier >>>>>>> (storm 0.9.4 btw) >>>>>>> >>>>>>
