That only works if you want to control a bolt. You cannot control a Spout from another spout, that spout being part of the topology. What you can do is that, you have to embed the Kafka consumer to look up the messages from the control topic and block nextTuple. You also need to store offsets and make sure it works for every instance of your main spout. This is the reason I think Zk solution is better. On Fri, Aug 14, 2015 at 12:05 PM Javier Gonzalez <[email protected]> wrote:
> I was thinking of using another spout as "control channel", and from that > spout manipulate the original spout to cause the nextTuple method to not > call the next message from the incoming queue (but not block so that acks > can be processed). A second signal on that spout can manipulate it back to > BAU. > > Thanks, > Javier > On Aug 14, 2015 2:59 PM, "Kishore Senji" <[email protected]> wrote: > >> This is exactly what is done on deactivate of topology. Storm has rest >> api to deactivate and you can use that to integrate with your external >> system. Otherwise you can extend your spout and use Zk to signal it. On >> receiving signal it can block and unblock appropriately the nextTuple() call >> On Fri, Aug 14, 2015 at 5:34 AM Javier Gonzalez <[email protected]> >> wrote: >> >>> I'm trying to ensure everything is processed for coordination with an >>> external system. Therefore, on a given signal, I have to stop the spout, >>> wait for the acks of all pending/inflight messages, then resume the spout >>> intake on a second signal. >>> On Aug 13, 2015 10:01 PM, "Kishore Senji" <[email protected]> wrote: >>> >>>> The deactivate() on the spout will be called when the topology is >>>> deactivated. Below is the output of storm deactivate help. This will >>>> deactivate the whole topology and this can be from the Storm UI as well. >>>> Does you topology have more than one Spout and are you looking to stop only >>>> a subset of those Spouts? >>>> >>>> $ storm help deactivate >>>> Syntax: [storm deactivate topology-name] >>>> >>>> Deactivates the specified topology's spouts. >>>> >>>> >>>> >>>> On Thu, Aug 13, 2015 at 2:12 PM Javier Gonzalez <[email protected]> >>>> wrote: >>>> >>>>> On a more broader term, can you share the strategies you've used to >>>>> pause (not emit anything else into the topology and not read anything else >>>>> from the data source) a topology's spouts? >>>>> >>>>> Thanks, >>>>> Javier >>>>> On Aug 13, 2015 2:53 PM, "Javier Gonzalez" <[email protected]> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I have a use case where I would need to stop a spout from emitting >>>>>> for a period of time. I'm looking at the activate /deactivate methods, >>>>>> but >>>>>> there's not much information apart from the API and the java base classes >>>>>> have empty implementations. Can anybody shed any insight on how those >>>>>> work? >>>>>> >>>>>> Thanks, >>>>>> Javier >>>>>> (storm 0.9.4 btw) >>>>>> >>>>>
