Hello all,
I have some unclear Storm behavior and need your advice.
I have topology like this:
,-> calculate --.
jms -> spout -> map --> calculate - - > reduce -> jms
| `-> calculate --'
| |
|
local local
fields
or or
grouping
shuffle shuffle
I have parallelism hint set up > 1 for bolts/spouts.
Depends on type of tuple, emitted by spout, I have 'map' bolt executed
different amount of time. 99% of tuples are executed < 200 ms (light),
but 1% could take 40 seconds or more (heavy).
Working on performance optimization, I monitored amount of tuples
emitted by spout and amount 'map' bolts executed in parallel and found
some strange pattern.
In case of 'heavy' request only one 'map bolt' could be executed, but
other tuples are queued in topology waiting while the 'heavy' one will
finish. Take a look on the attached picture.
Queued tuples are light tuples. I have made a thread dump at the moment
of such 'idle' and found that only one 'map bolt' thread is RUNNING,
while all other are in TIMED_WAITING, means Storm didn't deliver tuples
to free bolts.
Even worse, I have made the same test monitoring multiple workers, and
found completely the same picture. One worker has one thread executing
'heavy' request, all other workers are doing nothing even if I have
tuples to be processed.
I'm in doubt, why does it happen and need your help.
Best regards,
Aleksey.
Please visit our website at
http://financialservicesinc.ubs.com/wealth/E-maildisclaimer.html
for important disclosures and information about our e-mail
policies. For your protection, please do not transmit orders
or instructions by e-mail or include account numbers, Social
Security numbers, credit card numbers, passwords, or other
personal information.
