Hi Lukas, Somewhat... The idea is a monitoring 'service'. Hosted. Where its hosted is a separate conversation.
I agree with most everything you stated. Having a local agent running is the way to go. I'm avoiding remote polling and little to no advantage in distributing as a plugin. The one advantage is that it is very clean and simple to install, but such a tight dependency has its downsides, as you mentioned. I don't agree on not using java. Folks using elasticsearch already have a JVM installed obviously, so its an easy jar drop and run. I'm not sure why you would consider it a heavy process. Have you been looking at my old jboss code? ;-) rrd is lighter. No objection there. So is Perl. At first the data from elasticsearch is meant to be condensed and sent across the wire... Small payloads. At some point, I may move to storing the full blobs, but that can task a system and certainly add to hardware costs for retention. Across x,000 clusters, storing monitoring metrics form histograms and analysis can get heavy. On Jan 21, 2014 11:24 PM, "Lukáš Vlček" <[email protected]> wrote: > Roy, > > if you are in fact talking about implementation of monitoring system for > ES then the following are my 2 cents: > > - regarding how to poll data from ES nodes, I would not implement it as a > "plugin" or anything similar that is dependent on ES platform itself. It > should be an external process and possibly very light process (Java process > is not light). The impact on ES should be as minimal as possible and it > should not rely on plugin system of ES either. I would consider looking at > something like RRD tools (and extensions on top of it). Really lightweight > process that sits on each node, kicks once a while, grabs all metrics via > REST API and stores it into local storage. It should then try to send these > data into central store but it may not be always available thus storing it > locally is important in case the connection to central store is restored at > later time. The other argument why not to use plugin system of ES is that > you should be able to upgrade monitoring system independently on ES cluster > - you can not do this with ES plugins. > > - should it poll the same stats from every single node? I think it should! > If ES cluster gets into trouble is can be because not every part of the > cluster sees "the same picture". IMO the only way how to learn about this > is collecting individual "pictures from every node". It is also the most > simple way of doing it (and you should definitely aim for simplicity). > Solving duplicities on the central storage side later should be possible > but you might not even need to do it if the size of collected data is not a > problem for you. The part where you need to be very careful when polling > the same stats from each node is making sure this is not putting high load > on the ES cluster. If it is possible to poll from the local node only (like > using _local in URL) then opt for it. Though I am no sure if this option is > still available post 1.0.0.RC1 release. > > Regards, > Lukas > > > On Tue, Jan 21, 2014 at 10:09 PM, [email protected] < > [email protected]> wrote: > >> You must deploy plugins to all nodes, so you will still have the plugin >> available when nodes come and go. You could add an action so that you can >> turn on/off a plugin explicitly by remote command. Or you can define a >> condition so a plugin can decide when to run. For example if you want a >> singleton, you can check if you run on the master node, and execute only >> then. >> >> Jörg >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFQUHak0-PQ%3DLW3C0_0GQy8EwJThOKC__RhGgzGJk4MvA%40mail.gmail.com >> . >> >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/UWiNFuu6bw4/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAO9cvUbcy%2B-F1EdrOAJE%2Bh8Ud8JcbnpWpAf78ijZ4yO0jn6RYA%40mail.gmail.com > . > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANTOpgtZuTk%2BQkNii7SoZWtf%3DkQ%3DYJ5%2BxJP1i3kAa1Yw4M0k0Q%40mail.gmail.com. For more options, visit https://groups.google.com/groups/opt_out.
