Thanks Dan for giving opinion. :) To tell the truth, when I was implementing STORM-513, Sean talks me privately about why design constraint is necessary. It was valid opinion actually.
I was thinking multilang feature should consider whole languages. It blocks introducing whole kinds of approaches, and introduces design constraint finally. After introducing this constraint, Dashengju noticed me that design constraint can't cover some kind of situation which STORM-742 still can't cover it. I agree and change my mind that it's time for multilang feature to drop supporting some kind of languages which doesn't meet future requirements. I know default implementation of Python and Ruby have GIL issue, but AFAIK context switch interval is not too long so it doesn't block heartbeat timer to act on time. (Please let me know when you met GIL issue which blocks one thread to wait over seconds.) I don't expect subprocess to change modified time per exactly 1 sec, and ShellSpout and ShellBolt will adjust it, too. It is replacement of current heartbeat mechanism, so when we introduce new heartbeat, old thing should be removed. It could introduce backward compatibility issue (especially changing protocol) so we should consider what version we can adopt this. Thanks for reading long mail. Thanks, Jungtaek Lim (HeartSaVioR) 2015년 7월 10일 금요일, Dan Blanchard<[email protected]>님이 작성한 메시지: > Hi Jungtaek, > > Sorry I didn’t notice this earlier, as I was the person who filed > STORM–513 <https://issues.apache.org/jira/browse/STORM-513> in the first > place. > > Having just implemented the new heartbeat protocol in Python (for > streamparse <https://github.com/Parsely/streamparse/pull/87>) and Perl (for > IO::Storm > <https://github.com/dan-blanchard/io-storm/commit/d1bac6bcac9fa2f8c6eee5ce3eae7f98eb45930e>), > I’m not crazy about needing to add another heartbeat approach to multiple > libraries so soon. > > I also am against needing to deal with multithreading in Python (where > there will be GIL issues) just to accommodate a change to the heartbeat > protocol. It seems to me that the workaround you proposed in STORM–742 > <https://issues.apache.org/jira/browse/STORM-742> (where any command the > ShellBolt receives counts as a heartbeat) should be sufficient. > > Thanks, > Dan > > > > > On June 23, 2015 at 6:04:11 PM, 임정택 ([email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>) wrote: > > Hi! > > Since it's about multilang feature and you can use your own implementation > of multilang (and I believe multilang library developers are subscribing > user group), I wanna get opinion about changing multilang heartbeat > mechanism. > > At Storm 0.9.3, Storm introduces multilang heartbeat feature. > http://storm.apache.org/documentation/Multilang-protocol.html > If you use Storm 0.9.3 and higher, and didn't know about the change, you > may skip this mail. > > Since it contains some design constraint, I'm trying my best to add > workarounds, but it cannot cover whole situation (STORM-738 > <https://issues.apache.org/jira/browse/STORM-738>). That's why I want to > change mechanism to get rid of design constraint. > > AS-IS (STORM-513 <https://issues.apache.org/jira/browse/STORM-513>) > > - When subprocess receives heartbeat tuple, subprocess sends sync to > parent. > - ShellSpout / ShellBolt updates last heartbeat timestamp when it receives > sync. > -- added workaround : ShellSpout / ShellBolt updates timestamp when it > receives any kind of message. (It doesn't applied to ShellBolt yet, but > it's ready for review. STORM-742 > <https://issues.apache.org/jira/browse/STORM-742>) > - ShellSpout / ShellBolt checks last heartbeat timestamp periodically, and > if timestamp is not updated well, it suicides itself. > > TO-BE (STORM-871 <https://issues.apache.org/jira/browse/STORM-871>) > > - Subprocess has to update pid file's modified time periodically. > -- In default implementation, it updates pid file every 1 sec. > -- It should be handled concurrently with executing pending tuples. > -- Some languages couldn't implement this clearly, but I don't have an > idea what languages could be. > - ShellSpout / ShellBolt checks last heartbeat timestamp by reading pid > file's modified time periodically, and if timestamp is not updated well, it > suicides itself. > - Heartbeat tuple is removed. > > Please let me know your opinion, especially when you're developing > multilang libraries. > > Thanks, > Jungtaek Lim (HeartSaVioR) > > -- Name : 임 정택 Blog : http://www.heartsavior.net / http://dev.heartsavior.net Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior
