Hi Jungtaek, Sorry I didn’t notice this earlier, as I was the person who filed STORM–513 in the first place.
Having just implemented the new heartbeat protocol in Python (for streamparse) and Perl (for IO::Storm), I’m not crazy about needing to add another heartbeat approach to multiple libraries so soon. I also am against needing to deal with multithreading in Python (where there will be GIL issues) just to accommodate a change to the heartbeat protocol. It seems to me that the workaround you proposed in STORM–742 (where any command the ShellBolt receives counts as a heartbeat) should be sufficient. Thanks, Dan On June 23, 2015 at 6:04:11 PM, 임정택 ([email protected]) wrote: Hi! Since it's about multilang feature and you can use your own implementation of multilang (and I believe multilang library developers are subscribing user group), I wanna get opinion about changing multilang heartbeat mechanism. At Storm 0.9.3, Storm introduces multilang heartbeat feature. http://storm.apache.org/documentation/Multilang-protocol.html If you use Storm 0.9.3 and higher, and didn't know about the change, you may skip this mail. Since it contains some design constraint, I'm trying my best to add workarounds, but it cannot cover whole situation (STORM-738). That's why I want to change mechanism to get rid of design constraint. AS-IS (STORM-513) - When subprocess receives heartbeat tuple, subprocess sends sync to parent. - ShellSpout / ShellBolt updates last heartbeat timestamp when it receives sync. -- added workaround : ShellSpout / ShellBolt updates timestamp when it receives any kind of message. (It doesn't applied to ShellBolt yet, but it's ready for review. STORM-742) - ShellSpout / ShellBolt checks last heartbeat timestamp periodically, and if timestamp is not updated well, it suicides itself. TO-BE (STORM-871) - Subprocess has to update pid file's modified time periodically. -- In default implementation, it updates pid file every 1 sec. -- It should be handled concurrently with executing pending tuples. -- Some languages couldn't implement this clearly, but I don't have an idea what languages could be. - ShellSpout / ShellBolt checks last heartbeat timestamp by reading pid file's modified time periodically, and if timestamp is not updated well, it suicides itself. - Heartbeat tuple is removed. Please let me know your opinion, especially when you're developing multilang libraries. Thanks, Jungtaek Lim (HeartSaVioR)
