Re: [Ganglia-developers] gmetad and rrdtool scalability

2009-12-25 Thread Carlo Marcelo Arenas Belon
On Thu, Dec 24, 2009 at 12:10:51PM +, Daniel Pocock wrote: Vladimir Vuksan wrote: The issue is value of this data. If these were financial transactions than no loss would be acceptable however these are not. They are performance, trending data which get averaged down as time goes by

Re: [Ganglia-developers] gmetad and rrdtool scalability

2009-12-24 Thread Daniel Pocock
Vladimir Vuksan wrote: On Mon, 21 Dec 2009, Spike Spiegel wrote: a. Get all the rrds (rsync) from gmetad2 before you restart gmetad1 which unless you have small amount or data or fast network between the two nodes won't complete before the next write is initiated, meaning they won't be

Re: [Ganglia-developers] gmetad and rrdtool scalability

2009-12-21 Thread Spike Spiegel
On Sun, Dec 20, 2009 at 7:35 PM, Vladimir Vuksan vl...@vuksan.com wrote: If you lose a day or two or even a week of trending data that is not gonna be disaster as long as that data is present somewhere else. sure, but where? how would the ganglia frontend tell? Thus I proposed a simple

Re: [Ganglia-developers] gmetad and rrdtool scalability

2009-12-21 Thread Vladimir Vuksan
On Mon, 21 Dec 2009, Spike Spiegel wrote: a. Get all the rrds (rsync) from gmetad2 before you restart gmetad1 which unless you have small amount or data or fast network between the two nodes won't complete before the next write is initiated, meaning they won't be identical. Granted they

Re: [Ganglia-developers] gmetad and rrdtool scalability

2009-12-20 Thread Spike Spiegel
On Mon, Dec 14, 2009 at 2:00 AM, Vladimir Vuksan vli...@veus.hr wrote: I think you guys are complicating much :-). Can't you simply have multiple gmetads in different sites poll a single gmond. That way if one gmetad fails data is still available and updated on the other gmetads. That is what

Re: [Ganglia-developers] gmetad and rrdtool scalability

2009-12-20 Thread Spike Spiegel
On Mon, Dec 14, 2009 at 10:28 AM, Carlo Marcelo Arenas Belon care...@sajinet.com.pe wrote: a) you are only concerned with redundancy and not looking for scalability - when I say scalability, I refer to the idea of maybe 3 or more gmetads running in parallel collecting data from huge numbers of

Re: [Ganglia-developers] gmetad and rrdtool scalability

2009-12-20 Thread Jesse Becker
On Sun, Dec 20, 2009 at 10:49, Spike Spiegel fsm...@gmail.com wrote: On Mon, Dec 14, 2009 at 2:00 AM, Vladimir Vuksan vli...@veus.hr wrote: I think you guys are complicating much :-). Can't you simply have multiple gmetads in different sites poll a single gmond. That way if one gmetad fails

Re: [Ganglia-developers] gmetad and rrdtool scalability

2009-12-20 Thread Jesse Becker
On Sun, Dec 20, 2009 at 11:02, Spike Spiegel fsm...@gmail.com wrote: On Mon, Dec 14, 2009 at 10:28 AM, Carlo Marcelo Arenas Belon care...@sajinet.com.pe wrote: a) you are only concerned with redundancy and not looking for scalability - when I say scalability, I refer to the idea of maybe 3 or

Re: [Ganglia-developers] gmetad and rrdtool scalability

2009-12-20 Thread Carlo Marcelo Arenas Belon
On Sun, Dec 20, 2009 at 04:02:36PM +, Spike Spiegel wrote: On Mon, Dec 14, 2009 at 10:28 AM, Carlo Marcelo Arenas Belon care...@sajinet.com.pe wrote: b) you can afford to have duplicate storage - if your storage requirements are huge (retaining a lot of historic data or lot's of data

Re: [Ganglia-developers] gmetad and rrdtool scalability

2009-12-14 Thread Daniel Pocock
Vladimir Vuksan wrote: I think you guys are complicating much :-). Can't you simply have multiple gmetads in different sites poll a single gmond. That way if one gmetad fails data is still available and updated on the other gmetads. That is what we used to do. That is a good solution under

Re: [Ganglia-developers] gmetad and rrdtool scalability

2009-12-14 Thread Carlo Marcelo Arenas Belon
On Mon, Dec 14, 2009 at 09:26:01AM +, Daniel Pocock wrote: Vladimir Vuksan wrote: I think you guys are complicating much :-). Can't you simply have multiple gmetads in different sites poll a single gmond. That way if one gmetad fails data is still available and updated on the other

Re: [Ganglia-developers] gmetad and rrdtool scalability

2009-12-13 Thread Spike Spiegel
On Fri, Dec 11, 2009 at 1:34 PM, Daniel Pocock dan...@pocock.com.au wrote: Thanks for sharing this - could you comment on the total number of RRDs per gmetad, and do you use rrdcached? the largest colo has 140175 rrds and we use the tmpfs + cron hack, no rrdcached. I was thinking about

Re: [Ganglia-developers] gmetad and rrdtool scalability

2009-12-13 Thread Vladimir Vuksan
I think you guys are complicating much :-). Can't you simply have multiple gmetads in different sites poll a single gmond. That way if one gmetad fails data is still available and updated on the other gmetads. That is what we used to do. Vladimir On Sun, 13 Dec 2009, Spike Spiegel wrote:

Re: [Ganglia-developers] gmetad and rrdtool scalability

2009-12-11 Thread Daniel Pocock
Spike Spiegel wrote: On Wed, Nov 25, 2009 at 4:20 PM, Daniel Pocock dan...@pocock.com.au wrote: One problem I've been wondering about recently is the scalability of gmetad/rrdtool. [cut] In a particularly large organisation, moving around the RRD files as clusters grow could

Re: [Ganglia-developers] gmetad and rrdtool scalability

2009-12-05 Thread Spike Spiegel
On Wed, Nov 25, 2009 at 4:20 PM, Daniel Pocock dan...@pocock.com.au wrote: One problem I've been wondering about recently is the scalability of gmetad/rrdtool. [cut] In a particularly large organisation, moving around the RRD files as clusters grow could become quite a chore.  Is anyone

[Ganglia-developers] gmetad and rrdtool scalability

2009-11-25 Thread Daniel Pocock
One problem I've been wondering about recently is the scalability of gmetad/rrdtool. Various events could impact the gmetad server load: - adding more clusters - in this case, the admin can easily decide to add the new cluster on a new gmetad server if the existing server is overloaded -