We don't have much if any docs on the internals of storm. The hearbeats are written into zookekeeper from here. https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/daemon/supervisor.clj#L503-L521 Before they are written they are translated into a thrift object here (I included the code that changes it back too)
https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/converter.clj#L7-29 The thrift object is defined here https://github.com/apache/storm/blob/master/storm-core/src/storm.thrift#L248-257 The process is done in reverse for reads and is read by nimbus here https://github.com/apache/storm/blob/master/storm-core/src/clj/backtype/storm/daemon/nimbus.clj#L286-297 It is not the simplest thing in the world but it does work. Please be careful when trying to change the thrift code, we want to maintain wire compatibility so any changes need to be optional changes either with defaults, or that the code can properly ignore if it is null. - Bobby On Tuesday, May 19, 2015 7:30 AM, 陳怡仁 <[email protected]> wrote: First, thanks your tips :) I am newbie for storm and now study how supervisor detect or maintain worker nodes resource situation and how supervisor trigger its heart beats to nimbus. Are there any available document on this issue or which part of storm source code i can trace to know the mechanism? thanks. -YR 2015-05-19 1:43 GMT+08:00 Bobby Evans <[email protected]>: > We have been trying to add in resource aware scheduling to storm, and it > is still a ways out before we release it as open source, but for us we are > including the resources each supervisor has as part of the supervisor > heartbeat. > - Bobby > > > > On Friday, May 15, 2015 4:23 AM, 陳怡仁 <[email protected]> wrote: > > > Hi guys, > > I try to build a custom scheduler on storm 0.9.4 that need dynamically > detect specific hardware like Nvidia GPGPU on each Worker Node, but current > storm can not do that. > so i add my detect mechanism code in storm but always get the device > information where storm Nimbus run. > Anyone could please give me some tips that where storm Supervisor > maintain its information and how it communicate with Nimbus? > > Thanks. > -YR > > > >
