Thanks for all the help explaining how message time out works. From: 임정택 [mailto:[email protected]] Sent: Monday, May 25, 2015 10:53 PM To: [email protected] Subject: Re: RotatingMap and Spout
Yes, Acker just evaluates that tuple is completed or failed when other bolts notifies ack / fail about tuple and notify to Spout. Spout handles all remain works. In order to clean invalid tuples (timed-out), Acker uses rotating map to discard tuples which don't respond (ack / fail) to Acker in time. 2015-05-26 11:36 GMT+09:00 Srividhya Shanmugam <[email protected]<mailto:[email protected]>>: Thanks again for the explanation. I spent some more time looking at the source code for executor and acker. The rotating Map instance in acker just holds the tuples received by the acker. This instance does not have an ExpiredCallBack implementation. So the message time out delta is not evaluated when rotate method is invoked here. All those logic actually happens in the executor’s mk-thread method for spout. Let me know if my understanding is different. From: 임정택 [mailto:[email protected]<mailto:[email protected]>] Sent: Thursday, May 21, 2015 6:40 PM To: [email protected]<mailto:[email protected]> Subject: Re: RotatingMap and Spout 2015년 5월 22일 금요일, Srividhya Shanmugam<[email protected]<mailto:[email protected]>>님이 작성한 메시지: Thank you. This was helpful. Just to confirm, the RotatingMap is associated with the Executor thread of the Spout task. The messages are verified for their expiration when rotate() is invoked on the RotatingMap. And this gets invoked when for every tick tuple received by the Spout. Is this correct? Yes, right. I also see the acker holding on to a rotating Map…so every tuple emitted by the spout will be in the rotating Map of both Executor thread for spout task and the executor thread of the acker bolt. And in both cases, only for tick tuples the rotate method is invoked. Why there are rotating map instances in two different threads? I don't know about design concept about these. But logically two things are different executors so it can be run from different workers, and though it runs with same worker, synchronization for the map could introduce worse performance. Thanks again, Srividhya Thanks! Jungtaek Lim (HeartSaVioR) From: 임정택 [mailto:[email protected]] Sent: Thursday, May 21, 2015 12:36 AM To: [email protected]<mailto:[email protected]> Subject: Re: RotatingMap and Spout Hi. AFAIK, Spout sets up timer to emit tick tuples each TOPOLOGY-TICK-TUPLE-FREQ-SECS. (Please refer setup-ticks!) Spout changes TOPOLOGY-TICK-TUPLE-FREQ-SECS to TOPOLOGY-MESSAGEE-TIMEOUT-SECS for the first time, so timer is scheduled to each TOPOLOGY-MESSAGE-TIMEOUT-SECS. (Please refer add-acker!) When Spout receives tick tuple, it calls pending.rotate(). Its size is 2, so actual tuple expiration time is between TOPOLOGY-MESSAGE-TIMEOUT-SECS and (2 * TOPOLOGY-MESSAGE-TIMEOUT-SECS). (Please refer mk-threads :spout) Hope this helps. Thanks! Jungtaek Lim (HeartSaVioR) 2015-05-21 12:40 GMT+09:00 Srividhya Shanmugam <[email protected]<mailto:[email protected]>>: Hi, I have been reading through code and online to find how and where exactly a tuple gets timed out and therefore the spout’s fail method is invoked. As I look through the executor.clj class, a RotatingMap is always associated with a Spout. This is where the emitted tuples pending ack are stored. Every time the RotatingMap’s rotate() method is called, it invokes the passed in ExpiryCallback implementation to notify the expired tuple. What I am trying to understand is who calls the rotate() method? I also see a put method in the RotatingMap – I am guessing this is invoked when the Spout emits a new tuple. Is my understanding correct? Thank you very much, Srividhya This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately. -- Name : 임 정택 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately. -- Name : 임 정택 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately. -- Name : 임 정택 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior This email and any files transmitted with it are confidential, proprietary and intended solely for the individual or entity to whom they are addressed. If you have received this email in error please delete it immediately.
