Hi all,
I was reading the attached email thread that was discussed here before and I
have few questions about message timeouts and tick tuples.
I am doing some testing using 0.9.4 version and I have my
topology.message.timeout.secs = 60
I have a bolt that is written using multilang protocol in PHP and is doing the
following:
1. Log a message to indicate process started
2. Sleep for 150 seconds
3. Log a message to indicated process ended
I would have expected the message to expire after 60 seconds but I see the
process completing successfully by logging the 3rd step as well.
My questions -
1. What exactly does the message timeout do? Because it looks like it is
doing nothing in terms of timing out the message. Also, I have
"topology.enable.message.timeouts" set to true
2. Do I need to call something in code to explicitly invoke the a
function that would eventually do the check to expire the message?
3. What does the tick tuple do? Initially I had not set
"topology.tick.tuple.freq.secs" at all. So from reading the attached email
conversation I think it was set to message timeout seconds by default. Later I
set it to 2 secs and my worker won't start and keeps failing. What is an
expected value for this and when should we be configuring this?
P.S: I read documentations but they don't seem to give me a detailed
understanding
Thanks,
Ganesh
--- Begin Message ---
Thanks for all the help explaining how message time out works.
From: 임정택 [mailto:[email protected]]
Sent: Monday, May 25, 2015 10:53 PM
To: [email protected]
Subject: Re: RotatingMap and Spout
Yes, Acker just evaluates that tuple is completed or failed when other bolts
notifies ack / fail about tuple and notify to Spout. Spout handles all remain
works.
In order to clean invalid tuples (timed-out), Acker uses rotating map to
discard tuples which don't respond (ack / fail) to Acker in time.
2015-05-26 11:36 GMT+09:00 Srividhya Shanmugam
<[email protected]<mailto:[email protected]>>:
Thanks again for the explanation. I spent some more time looking at the source
code for executor and acker. The rotating Map instance in acker just holds the
tuples received by the acker. This instance does not have an ExpiredCallBack
implementation. So the message time out delta is not evaluated when rotate
method is invoked here. All those logic actually happens in the executor’s
mk-thread method for spout.
Let me know if my understanding is different.
From: 임정택 [mailto:[email protected]<mailto:[email protected]>]
Sent: Thursday, May 21, 2015 6:40 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: RotatingMap and Spout
2015년 5월 22일 금요일, Srividhya
Shanmugam<[email protected]<mailto:[email protected]>>님이
작성한 메시지:
Thank you. This was helpful. Just to confirm, the RotatingMap is associated
with the Executor thread of the Spout task. The messages are verified for their
expiration when rotate() is invoked on the RotatingMap.
And this gets invoked when for every tick tuple received by the Spout. Is this
correct?
Yes, right.
I also see the acker holding on to a rotating Map…so every tuple emitted by
the spout will be in the rotating Map of both Executor thread for spout task
and the executor thread of the acker bolt. And in both cases, only for tick
tuples the rotate method is invoked. Why there are rotating map instances in
two different threads?
I don't know about design concept about these.
But logically two things are different executors so it can be run from
different workers, and though it runs with same worker, synchronization for the
map could introduce worse performance.
Thanks again,
Srividhya
Thanks!
Jungtaek Lim (HeartSaVioR)
From: 임정택 [mailto:[email protected]]
Sent: Thursday, May 21, 2015 12:36 AM
To: [email protected]<mailto:[email protected]>
Subject: Re: RotatingMap and Spout
Hi.
AFAIK, Spout sets up timer to emit tick tuples each
TOPOLOGY-TICK-TUPLE-FREQ-SECS.
(Please refer setup-ticks!)
Spout changes TOPOLOGY-TICK-TUPLE-FREQ-SECS to
TOPOLOGY-MESSAGEE-TIMEOUT-SECS for the first time, so timer is scheduled to
each TOPOLOGY-MESSAGE-TIMEOUT-SECS.
(Please refer add-acker!)
When Spout receives tick tuple, it calls pending.rotate(). Its size is 2, so
actual tuple expiration time is between TOPOLOGY-MESSAGE-TIMEOUT-SECS and (2 *
TOPOLOGY-MESSAGE-TIMEOUT-SECS).
(Please refer mk-threads :spout)
Hope this helps.
Thanks!
Jungtaek Lim (HeartSaVioR)
2015-05-21 12:40 GMT+09:00 Srividhya Shanmugam
<[email protected]<mailto:[email protected]>>:
Hi,
I have been reading through code and online to find how and where exactly a
tuple gets timed out and therefore the spout’s fail method is invoked. As I
look through the executor.clj class, a RotatingMap is always associated with a
Spout. This is where the emitted tuples pending ack are stored.
Every time the RotatingMap’s rotate() method is called, it invokes the
passed in ExpiryCallback implementation to notify the expired tuple.
What I am trying to understand is who calls the rotate() method?
I also see a put method in the RotatingMap – I am guessing this is invoked
when the Spout emits a new tuple.
Is my understanding correct?
Thank you very much,
Srividhya
This email and any files transmitted with it are confidential, proprietary
and intended solely for the individual or entity to whom they are addressed. If
you have received this email in error please delete it immediately.
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
This email and any files transmitted with it are confidential, proprietary
and intended solely for the individual or entity to whom they are addressed. If
you have received this email in error please delete it immediately.
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
This email and any files transmitted with it are confidential, proprietary
and intended solely for the individual or entity to whom they are addressed. If
you have received this email in error please delete it immediately.
--
Name : 임 정택
Blog : http://www.heartsavior.net / http://dev.heartsavior.net
Twitter : http://twitter.com/heartsavior
LinkedIn : http://www.linkedin.com/in/heartsavior
This email and any files transmitted with it are confidential, proprietary
and intended solely for the individual or entity to whom they are addressed. If
you have received this email in error please delete it immediately.
--- End Message ---
