Thank you for your replies.

Cool. We've switch to non-blocking. 
How does one avoid nextTuple() being called too fast (and potentially use up 
too much cpu)? 

We have seen sleeps (in test spouts) and one colleague suggested a wait 
strategy. Which should we do, if any?


From: user@storm.apache.org 
Subject: Re: blocking ISpout::nextTuple()

        

Avoid blocking inside nextTuple(). Use non blocking call (or blocking call with 
timeout) to check external source for new data.  Since the spout (doing I/O) 
are already running asynchronously  from the bolts (doing processing) … I 
personally prefer to keep the spout single threaded If the source provides 
either a non-blocking or blocking-with-wait method of reading. 
  
Blocking too long in the nextTuple() can lead to the spout taking longer to 
process acks (if acking is enabled) and generate updated  metrics. 
-roshan 
  
  

From:  "Ramin Farajollah (BLOOMBERG/ 731 LEX)" <rfarajol...@bloomberg.net>
Reply-To: "user@storm.apache.org" <user@storm.apache.org>, Ramin Farajollah 
<rfarajol...@bloomberg.net>
Date: Tuesday, July 18, 2017 at 9:10 AM
To: "user@storm.apache.org" <user@storm.apache.org>
Subject: Re: blocking ISpout::nextTuple() 

  

Agree on populating the queue on another thread.   

When you say "try to pop the next message", do you mean that it should block 
indefinitely?  

  

My colleagues suggest that it should not. Instead a wait strategy to be used in 
the yaml: 

https://storm.apache.org/releases/1.0.3/javadocs/org/apache/storm/spout/ISpoutWaitStrategy.html
 

  

  

From: user@storm.apache.org  

Subject: Re: blocking ISpout::nextTuple() 


I believe the docs around spouts say nextTuple() should never block.  In the 
spout implementations I've done, I typically spin off a separate work thread in 
the spout's  open() method and push tuples into a shared concurrent queue.  
nextTuple() simply tries to pop the next message off of that shared queue. 

  

On Tue, Jul 18, 2017 at 8:32 AM, Ramin Farajollah (BLOOMBERG/ 731 LEX) 
<rfarajol...@bloomberg.net> wrote: 

Hi,  

  

should a spout block on its source to produce a tuple? 

  

For example, should it read its source from a blocking queue before emit? 

In this case, the spout will indefinitely block before it can emit a tuple. 

  

I read about how ack and fail are processed on the same thread.  

  

1) Will blocking in spout's nextTuple result in blocking the downstream bolts? 

2) How about when a messageId is not specified? 

3) How about when a tuple traverses multiple JVM instances on the same or cross 
machines? 

  

Kind regards 

  

  


<< �gA mind is like a parachute. It doesn't work if it is not open�h Frank Zap 
>>
 
 
 
  


<< �gA mind is like a parachute. It doesn't work if it is not open�h Frank Zap 
>>  


<< �gA mind is like a parachute. It doesn&#39;t work if it is not open�h Frank 
Zap >>

Reply via email to