Thank you for your replies.
Cool. We've switch to non-blocking.
How does one avoid nextTuple() being called too fast (and potentially use up
too much cpu)?
We have seen sleeps (in test spouts) and one colleague suggested a wait
strategy. Which should we do, if any?
From: user@storm.apache.org
Subject: Re: blocking ISpout::nextTuple()
Avoid blocking inside nextTuple(). Use non blocking call (or blocking call with
timeout) to check external source for new data. Since the spout (doing I/O)
are already running asynchronously from the bolts (doing processing) … I
personally prefer to keep the spout single threaded If the source provides
either a non-blocking or blocking-with-wait method of reading.
Blocking too long in the nextTuple() can lead to the spout taking longer to
process acks (if acking is enabled) and generate updated metrics.
-roshan
From: "Ramin Farajollah (BLOOMBERG/ 731 LEX)" <rfarajol...@bloomberg.net>
Reply-To: "user@storm.apache.org" <user@storm.apache.org>, Ramin Farajollah
<rfarajol...@bloomberg.net>
Date: Tuesday, July 18, 2017 at 9:10 AM
To: "user@storm.apache.org" <user@storm.apache.org>
Subject: Re: blocking ISpout::nextTuple()
Agree on populating the queue on another thread.
When you say "try to pop the next message", do you mean that it should block
indefinitely?
My colleagues suggest that it should not. Instead a wait strategy to be used in
the yaml:
https://storm.apache.org/releases/1.0.3/javadocs/org/apache/storm/spout/ISpoutWaitStrategy.html
From: user@storm.apache.org
Subject: Re: blocking ISpout::nextTuple()
I believe the docs around spouts say nextTuple() should never block. In the
spout implementations I've done, I typically spin off a separate work thread in
the spout's open() method and push tuples into a shared concurrent queue.
nextTuple() simply tries to pop the next message off of that shared queue.
On Tue, Jul 18, 2017 at 8:32 AM, Ramin Farajollah (BLOOMBERG/ 731 LEX)
<rfarajol...@bloomberg.net> wrote:
Hi,
should a spout block on its source to produce a tuple?
For example, should it read its source from a blocking queue before emit?
In this case, the spout will indefinitely block before it can emit a tuple.
I read about how ack and fail are processed on the same thread.
1) Will blocking in spout's nextTuple result in blocking the downstream bolts?
2) How about when a messageId is not specified?
3) How about when a tuple traverses multiple JVM instances on the same or cross
machines?
Kind regards
<< �gA mind is like a parachute. It doesn't work if it is not open�h Frank Zap
>>
<< �gA mind is like a parachute. It doesn't work if it is not open�h Frank Zap
>>
<< �gA mind is like a parachute. It doesn't work if it is not open�h Frank
Zap >>
