Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-04-02 Thread Adrian Sevcenco
On 03/27/2014 10:07 PM, Daniel Pocock wrote:
 
 
 I made up a rough diagram about how Ganglia 4.x could look:
 
 https://raw.githubusercontent.com/ganglia/monitor-core/master/doc/planning/ganglia-4.x.png
 
 The biggest change is the introduction of MongoDB
 
 Instead of having the gmetad serve up an XML every time somebody asks to
 see the web page, the gmetad will just store current values into MongoDB.
Well, i will post again my ideas/problems with the hope that somebody
more proficient in coding and with a lot more time have the same needs
as i do.

So (the big ones):
1. one of my biggest problems with gmond is that the cluster tagging is
done at the level of xml and is not a per-host tag. this will allow to
aggregate info from different nodes and grouping per cluster tag be done
at storage level (gmetad)

2. for the storage back-end i would recommend postgres : we (our
experiment) use a tool that write all the info in postgres (many
hundredth thousands parameters/s) and the admins and developers assured
me that postgres was the best choice.

Some minor things:
1. It would be useful to have in mod_cpu all the info (or at least the
4-5 lines) of /proc/cpuinfo namely : processor, vendor_id, cpu family,
model, model name (most important), stepping

2. instead of :
Operating SystemLinux
Operating System Release3.13.0-1.el6.elrepo.x86_64

(info obtained from uname i imagine) it should print info obtained from
lsb_release and uname :
uname -o
lsb_release -irc
uname -r
uname -m

3. if its possible (at least with the title of experiment) to add
host_uuid variable to host variables. I understood from Daniel that
something like this is already in a branch

Thank you!
Adrian




smime.p7s
Description: S/MIME Cryptographic Signature
--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-28 Thread Maxime Brugidou
I don't understand why all this is necessary.

I strongly disagree with the horizontal scalability of mongoDB ( i run a
very large mongodb cluster in production in addition to other databases)
and would rather suggest a pluggable backend with a simpler default (like
text files or maybe postgresql or mysql).

Same thing for rabbitmq (although this is easier to maintain in
production). The reason why I liked ganglia in the first place is because
of its simplicity and ease of installation and maintenance.

We also host a very large graphite installation which is a nightmare and we
use it for various use cases. I feel like ganglia is getting closer to
graphite.

I am a heavy ganglia user, having thousands of bare metal servers running
gmond. My main use case is to have a host metrics with nice predefined
graphs without much work to set up. It's a great idea to upgrade the web
front end because the php code is rather hard to follow whenever you need
to patch it, however I think that the web front end should keep things
simple.

Whenever we need to do complex stuff we use other metrics systems like
graphite (and we can send host data from gmond to graphite already).
On Mar 27, 2014 9:09 PM, Daniel Pocock dan...@pocock.com.au wrote:



 I made up a rough diagram about how Ganglia 4.x could look:


 https://raw.githubusercontent.com/ganglia/monitor-core/master/doc/planning/ganglia-4.x.png

 The biggest change is the introduction of MongoDB

 Instead of having the gmetad serve up an XML every time somebody asks to
 see the web page, the gmetad will just store current values into MongoDB.

 This means that web frameworks (like PHP) can query the data from
 MongoDB, which is much more horizontally scalable and more suited to
 serving this data.  For large sites where many users access the web
 reports, this will be very useful.

 MongoDB is also a backend for rsyslog daemon now and could potentially
 be a Nagios backend, so it would be a great way to unify monitoring data.

 The introduction of RabbitMQ is an optional dependency.  It would allow
 users to send commands from the web interface.

 One of the motivations for this work is the Google Summer of Code
 projects.  Each student can potentially work on a different part of this
 puzzle and at the end of the year we could launch it as Ganglia 4.0 if
 people like it.

 Regards,

 Daniel



 --
 ___
 Ganglia-general mailing list
 Ganglia-general@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/ganglia-general

--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-28 Thread Alexander Karner
Hi!

I think we should continue to put an emphasis on portability:
Ganglia is not only used in Linux environments but also on AIX, HP-UX, 
Solaris etc.
This includes both, gmond and gmetad (+webserver).

Personally I'd suggest to check, which tools are available in those 
platforms. For the DB this could be for example postgresql.

How would the current architectural overview match a Grid of Grids 
environment? Would this allow us to search for a system on the central 
gmtead WebUI and then jump to the specific grid server?
And: Would this also match actual requirements about partition mobility? 

Mit freundlichen Grüßen / Kind regards

Alexander Karner





From:   Daniel Pocock dan...@pocock.com.au
To: ganglia-general@lists.sourceforge.net, 
Date:   27.03.2014 21:08
Subject:[Ganglia-general] Ganglia 4.x architecture planning





I made up a rough diagram about how Ganglia 4.x could look:

https://raw.githubusercontent.com/ganglia/monitor-core/master/doc/planning/ganglia-4.x.png


The biggest change is the introduction of MongoDB

Instead of having the gmetad serve up an XML every time somebody asks to
see the web page, the gmetad will just store current values into MongoDB.

This means that web frameworks (like PHP) can query the data from
MongoDB, which is much more horizontally scalable and more suited to
serving this data.  For large sites where many users access the web
reports, this will be very useful.

MongoDB is also a backend for rsyslog daemon now and could potentially
be a Nagios backend, so it would be a great way to unify monitoring data.

The introduction of RabbitMQ is an optional dependency.  It would allow
users to send commands from the web interface.

One of the motivations for this work is the Google Summer of Code
projects.  Each student can potentially work on a different part of this
puzzle and at the end of the year we could launch it as Ganglia 4.0 if
people like it.

Regards,

Daniel


--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general




smime.p7s
Description: S/MIME Cryptographic Signature
--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-28 Thread Daniel Pocock
On 28/03/14 09:07, Alexander Karner wrote:
 Hi!

 I think we should continue to put an emphasis on portability:
 Ganglia is not only used in Linux environments but also on AIX, HP-UX,
 Solaris etc.
 This includes both, gmond and gmetad (+webserver).


In the earlier reply from Adam, the idea of a plugin / module API was
raised.  So we could have a plugin that emulates the status quo and we
could also have a plugin that does PostgreSQL

Does the gmond module API work effectively on AIX and the other
platforms you mention?  Does the current build system correctly generate
DSOs on some or all of those platforms or are there any platforms where
people are still relying on static linking?

 Personally I'd suggest to check, which tools are available in those
 platforms. For the DB this could be for example postgresql.

 How would the current architectural overview match a Grid of Grids
 environment? Would this allow us to search for a system on the central
 gmtead WebUI and then jump to the specific grid server?
 And: Would this also match actual requirements about partition mobility?


Can you please clarify this question, when you say current, do you mean
the existing gmetad 3.6 or the gmetad 4.x that is proposed in the diagram?

For my 4.x proposal, I believe that we could still provide a lightweight
daemon that reads from MongoDB (or whatever) and serves up an XML just
like the current gmetad port 8651/8652 output.  People who want a
hierarchy of gmetad servers could then configure the next gmetad up the
hierarchy to poll that XML service daemon.


--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-28 Thread Daniel Pocock
On 27/03/14 21:43, Alex Dean wrote:
 On Mar 27, 2014, at 3:07 PM, Daniel Pocock dan...@pocock.com.au wrote:

 The introduction of RabbitMQ is an optional dependency.  It would allow
 users to send commands from the web interface.
 1. Why add the extra dependency on rabbitmq? As long as you're adding a 
 persistent data store (mongo), couldn't it also serve as a message queue?

RabbitMQ is an optional component.  Currently there is no way to send
commands to gmetad from the web UI.  So people who choose not to compile
with the RabbitMQ dependency would not lose any feature they already have.

I agree there are other ways to do this - but some users (e.g. those
with security concerns) may want to heavily restrict access to MongoDB
but provide more widespread access to RabbitMQ.  Eventually, even the
gmond collectors could also listen on a RabbitMQ topic and people could
send them a command e.g. to flush old hosts when DMAX=0


 2. Why will the web UI start using Python flask?

It is just an option as well.  Many sites already customize the web UI,
so we could provide some trivial sample for serving CSV data using Flask
and for those who like that, they can build around it and those who like
PHP can use that.  PHP and the current UI is certainly not being deprecated.



--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-28 Thread Michael Perzl
Hi Daniel,

with the introduction of MongoDB you would exclude all big-endian 
architectures immediately as MongoDB is little-endian only.
Although there has been big-endian support requested for MongoDB and 
some attempts have been made in this direction but at the moment MongoDB 
is still little-endian only and thus you would for instance 
automatically exclude architectures like ppc64, sparc(64) and others.

Regards,
Michael

On 03/27/2014 09:07 PM, Daniel Pocock wrote:

 I made up a rough diagram about how Ganglia 4.x could look:

 https://raw.githubusercontent.com/ganglia/monitor-core/master/doc/planning/ganglia-4.x.png

 The biggest change is the introduction of MongoDB

 Instead of having the gmetad serve up an XML every time somebody asks to
 see the web page, the gmetad will just store current values into MongoDB.

 This means that web frameworks (like PHP) can query the data from
 MongoDB, which is much more horizontally scalable and more suited to
 serving this data.  For large sites where many users access the web
 reports, this will be very useful.

 MongoDB is also a backend for rsyslog daemon now and could potentially
 be a Nagios backend, so it would be a great way to unify monitoring data.

 The introduction of RabbitMQ is an optional dependency.  It would allow
 users to send commands from the web interface.

 One of the motivations for this work is the Google Summer of Code
 projects.  Each student can potentially work on a different part of this
 puzzle and at the end of the year we could launch it as Ganglia 4.0 if
 people like it.

 Regards,

 Daniel


 --
 ___
 Ganglia-general mailing list
 Ganglia-general@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/ganglia-general



--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-28 Thread Daniel Pocock
On 28/03/14 11:41, Michael Perzl wrote:
 Hi Daniel,

 with the introduction of MongoDB you would exclude all big-endian
 architectures immediately as MongoDB is little-endian only.
 Although there has been big-endian support requested for MongoDB and
 some attempts have been made in this direction but at the moment
 MongoDB is still little-endian only and thus you would for instance
 automatically exclude architectures like ppc64, sparc(64) and others.


This is quite a reasonable objection and it is the very reason I put
this up for discussion rather than just diving into it

Did you see the later comments about a plugin approach, do you feel that
satisfies your concerns?



--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-28 Thread Daniel Pocock
On 28/03/14 14:39, Aaron Nichols wrote:
 On Fri, Mar 28, 2014 at 1:45 AM, Maxime Brugidou
 maxime.brugi...@gmail.com mailto:maxime.brugi...@gmail.com wrote:

 I don't understand why all this is necessary.

 I strongly disagree with the horizontal scalability of mongoDB (
 i run a very large mongodb cluster in production in addition to
 other databases) and would rather suggest a pluggable backend with
 a simpler default (like text files or maybe postgresql or mysql).


 I'll just 2nd this opinion. My immediate reaction to the addition of
 mongo is rejection - my experience with it at scale has been awful 
 on a small scale it always feels like overkill. Further, since the
 data being served seems like a relatively small dataset it's unclear
 to me why there cannot be more options. For our use case (which is


That has already been discussed in the thread - are you OK with the
plugin idea?


 ~180k metrics) an in-memory datastore for metadata should be more than
 sufficient assuming it can be re-hydrated from some persistent source
 which could simply be waiting for a polling interval of gmetad. I have
 no need for the addition of nagios / rsyslog events. 



Knowing you have 180k metrics is only one factor

How many users (both humans and processes) want to view your data in
real-time?  Is it just used by sysadmins, for example, or does everybody
in every development or support team access Ganglia?


 I've never looked at Ganglia as a horizontally scalable system and
 accepted that because it made things simple. If I need to scale it out
 I need to partition my grids / clusters. 


Agreed - that has always worked and it would continue to work that way
too.  This is also good for people who have multiple small sites, etc.

--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-28 Thread Khrist Hansen
Happy AIX Ganglia user here thanks to all of Dr. Perzl’s generous efforts!

 

:)

 

 

From: Alexander Karner [mailto:a...@de.ibm.com] 
Sent: Friday, March 28, 2014 3:07 AM
To: Daniel Pocock
Cc: ganglia-general@lists.sourceforge.net
Subject: Re: [Ganglia-general] Ganglia 4.x architecture planning

 

Hi! 

I think we should continue to put an emphasis on portability: 
Ganglia is not only used in Linux environments but also on AIX, HP-UX,
Solaris etc. 
This includes both, gmond and gmetad (+webserver). 

Personally I'd suggest to check, which tools are available in those
platforms. For the DB this could be for example postgresql. 

How would the current architectural overview match a Grid of Grids
environment? Would this allow us to search for a system on the central
gmtead WebUI and then jump to the specific grid server? 
And: Would this also match actual requirements about partition mobility? 

Mit freundlichen Grüßen / Kind regards 

Alexander Karner 





From:Daniel Pocock dan...@pocock.com.au 
To:ganglia-general@lists.sourceforge.net, 
Date:27.03.2014 21:08 
Subject:[Ganglia-general] Ganglia 4.x architecture planning 

  _  






I made up a rough diagram about how Ganglia 4.x could look:

 
https://raw.githubusercontent.com/ganglia/monitor-core/master/doc/planning/
ganglia-4.x.png
https://raw.githubusercontent.com/ganglia/monitor-core/master/doc/planning/g
anglia-4.x.png

The biggest change is the introduction of MongoDB

Instead of having the gmetad serve up an XML every time somebody asks to
see the web page, the gmetad will just store current values into MongoDB.

This means that web frameworks (like PHP) can query the data from
MongoDB, which is much more horizontally scalable and more suited to
serving this data.  For large sites where many users access the web
reports, this will be very useful.

MongoDB is also a backend for rsyslog daemon now and could potentially
be a Nagios backend, so it would be a great way to unify monitoring data.

The introduction of RabbitMQ is an optional dependency.  It would allow
users to send commands from the web interface.

One of the motivations for this work is the Google Summer of Code
projects.  Each student can potentially work on a different part of this
puzzle and at the end of the year we could launch it as Ganglia 4.0 if
people like it.

Regards,

Daniel



--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/ganglia-general
https://lists.sourceforge.net/lists/listinfo/ganglia-general



--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-28 Thread Vladimir Vuksan

  
  
We may want to clarify what services
  from Mongo are we gonna use. Is the intention to use it as a
  key/value store or use it for things like capped collections
(http://docs.mongodb.org/manual/tutorial/use-capped-collections-for-fast-writes-and-reads/).
  
  Currently we do have support for memcached. I actually use
  memcached for most of our alerting at this point instead of
  querying gmetad.
  
  Vladimir
  
  On 03/28/2014 09:51 AM, Daniel Pocock wrote:


  
  On 28/03/14 14:39, Aaron Nichols
wrote:
  
  


  
On Fri, Mar 28, 2014 at 1:45 AM,
  Maxime Brugidou maxime.brugi...@gmail.com
  wrote:
  
I don't understand why all this is
  necessary.
I strongly disagree with the "horizontal
  scalability" of mongoDB ( i run a very large mongodb
  cluster in production in addition to other databases)
  and would rather suggest a pluggable backend with a
  simpler default (like text files or maybe postgresql
  or mysql).
  
  
  
  I'll just 2nd this opinion. My immediate reaction to
the addition of mongo is rejection - my experience with
it at scale has been awful  on a small scale it
always feels like overkill. Further, since the data
being served seems like a relatively small dataset it's
unclear to me why there cannot be more options. For our
use case (which is 

  

  
  
  
  That has already been discussed in the thread - are you OK with
  the plugin idea?
  
  
  

  

  ~180k metrics) an in-memory datastore for metadata
should be more than sufficient assuming it can be
re-hydrated from some persistent source which could
simply be waiting for a polling interval of gmetad. I
have no need for the addition of nagios / rsyslog
events.
  
  

  

  
  
  
  Knowing you have 180k metrics is only one factor
  
  How many users (both humans and processes) want to view your data
  in real-time? Is it just used by sysadmins, for example, or does
  everybody in every development or support team access Ganglia?
  
  
  

  

  I've never looked at Ganglia as a horizontally
scalable system and accepted that because it made things
simple. If I need to scale it out I need to partition my
grids / clusters.

 

  

  
  
  Agreed - that has always worked and it would continue to work that
  way too. This is also good for people who have multiple small
  sites, etc.


  


--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-28 Thread Aaron Nichols
On Fri, Mar 28, 2014 at 1:45 AM, Maxime Brugidou
maxime.brugi...@gmail.comwrote:

 I don't understand why all this is necessary.

 I strongly disagree with the horizontal scalability of mongoDB ( i run a
 very large mongodb cluster in production in addition to other databases)
 and would rather suggest a pluggable backend with a simpler default (like
 text files or maybe postgresql or mysql).


I'll just 2nd this opinion. My immediate reaction to the addition of mongo
is rejection - my experience with it at scale has been awful  on a small
scale it always feels like overkill. Further, since the data being served
seems like a relatively small dataset it's unclear to me why there cannot
be more options. For our use case (which is ~180k metrics) an in-memory
datastore for metadata should be more than sufficient assuming it can be
re-hydrated from some persistent source which could simply be waiting for a
polling interval of gmetad. I have no need for the addition of nagios /
rsyslog events.

I've never looked at Ganglia as a horizontally scalable system and accepted
that because it made things simple. If I need to scale it out I need to
partition my grids / clusters.

Not understanding what cool new capabilities will result from having a
database like Mongo backing the metadata my guess is that I'd stick w/ 3.x
for as long as I could and then find another solution. If the addition of
this complexity added considerable capabilities that were *actually*
realized then perhaps I'd use it, but I can't imagine what that would be.

Thanks,
Aaron

-- 
Twitter: @anichols
Blog: http://www.opsbs.com
Pro: http://www.linkedin.com/in/anichols
--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-28 Thread Aaron Nichols
On Fri, Mar 28, 2014 at 7:51 AM, Daniel Pocock dan...@pocock.com.au wrote:


 That has already been discussed in the thread - are you OK with the plugin
 idea?


I think so - having an easy built-in default would be ideal.


 Knowing you have 180k metrics is only one factor

 How many users (both humans and processes) want to view your data in
 real-time?  Is it just used by sysadmins, for example, or does everybody in
 every development or support team access Ganglia?


We've actually fronted ganglia-web w/ another app for organizing the graphs
so our demands are likely different than others who use the web UI
directly, I simply provided the metric count as a reference for metadata
size. In our case many different development teams access ganglia via this
web frontend and I understand that this increases demand on gmetad, however
I've seen a variety of improvements that can be applied to the existing
system to get us beyond the performance requirements we have today.

I'm not rejecting the use case some may have for a different solution, I'm
just pointing out it doesn't have immediate value to me.

-- 
Twitter: @anichols
Blog: http://www.opsbs.com
Pro: http://www.linkedin.com/in/anichols
--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


[Ganglia-general] Ganglia 4.x architecture planning

2014-03-27 Thread Daniel Pocock


I made up a rough diagram about how Ganglia 4.x could look:

https://raw.githubusercontent.com/ganglia/monitor-core/master/doc/planning/ganglia-4.x.png

The biggest change is the introduction of MongoDB

Instead of having the gmetad serve up an XML every time somebody asks to
see the web page, the gmetad will just store current values into MongoDB.

This means that web frameworks (like PHP) can query the data from
MongoDB, which is much more horizontally scalable and more suited to
serving this data.  For large sites where many users access the web
reports, this will be very useful.

MongoDB is also a backend for rsyslog daemon now and could potentially
be a Nagios backend, so it would be a great way to unify monitoring data.

The introduction of RabbitMQ is an optional dependency.  It would allow
users to send commands from the web interface.

One of the motivations for this work is the Google Summer of Code
projects.  Each student can potentially work on a different part of this
puzzle and at the end of the year we could launch it as Ganglia 4.0 if
people like it.

Regards,

Daniel


--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-27 Thread Daniel Pocock


On 27/03/14 21:16, Adam Compton wrote:
 I'm in favor of teaching gmetad how to send the metrics it collects to a
 wider variety of things, particularly if there's a plugin interface for
 writing them.
 

That is how rsyslog does it actually - MongoDB is just one of their
output modules, called ommongo

It should be relatively easy to use the module loading code from gmond
as a template for a similar thing in gmetad


--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-27 Thread Adam Compton
On 3/27/14 1:20 PM, Daniel Pocock wrote:

 On 27/03/14 21:16, Adam Compton wrote:
 I'm in favor of teaching gmetad how to send the metrics it collects to a
 wider variety of things, particularly if there's a plugin interface for
 writing them.

 That is how rsyslog does it actually - MongoDB is just one of their
 output modules, called ommongo

 It should be relatively easy to use the module loading code from gmond
 as a template for a similar thing in gmetad

That's exciting to hear. I've been working on getting gmetad to emit 
metrics to OpenTSDB, and there's a lot of code duplication between that, 
and the carbon emitter, and the riemann emitter, etc.

- Adam


--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-27 Thread Adam Compton
I'm in favor of teaching gmetad how to send the metrics it collects to a 
wider variety of things, particularly if there's a plugin interface for 
writing them.

- Adam


On 3/27/14 1:07 PM, Daniel Pocock wrote:

 I made up a rough diagram about how Ganglia 4.x could look:

 https://raw.githubusercontent.com/ganglia/monitor-core/master/doc/planning/ganglia-4.x.png

 The biggest change is the introduction of MongoDB

 Instead of having the gmetad serve up an XML every time somebody asks to
 see the web page, the gmetad will just store current values into MongoDB.

 This means that web frameworks (like PHP) can query the data from
 MongoDB, which is much more horizontally scalable and more suited to
 serving this data.  For large sites where many users access the web
 reports, this will be very useful.

 MongoDB is also a backend for rsyslog daemon now and could potentially
 be a Nagios backend, so it would be a great way to unify monitoring data.

 The introduction of RabbitMQ is an optional dependency.  It would allow
 users to send commands from the web interface.

 One of the motivations for this work is the Google Summer of Code
 projects.  Each student can potentially work on a different part of this
 puzzle and at the end of the year we could launch it as Ganglia 4.0 if
 people like it.

 Regards,

 Daniel


 --
 ___
 Ganglia-general mailing list
 Ganglia-general@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/ganglia-general


--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-27 Thread Alex Dean

On Mar 27, 2014, at 3:07 PM, Daniel Pocock dan...@pocock.com.au wrote:

 The introduction of RabbitMQ is an optional dependency.  It would allow
 users to send commands from the web interface.

1. Why add the extra dependency on rabbitmq? As long as you're adding a 
persistent data store (mongo), couldn't it also serve as a message queue?

2. Why will the web UI start using Python flask?


alex
--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general


Re: [Ganglia-general] Ganglia 4.x architecture planning

2014-03-27 Thread Bernard Li
Hi Adam:

This already exists as gmetad-python -- I have been told that it's not
as scalable as gmetad-C but I would really like someone to pick it up
and run with it:

https://github.com/ganglia/monitor-core/tree/master/gmetad-python

Cheers,

Bernard

On Thu, Mar 27, 2014 at 1:16 PM, Adam Compton acomp...@quantcast.com wrote:
 I'm in favor of teaching gmetad how to send the metrics it collects to a
 wider variety of things, particularly if there's a plugin interface for
 writing them.

 - Adam


 On 3/27/14 1:07 PM, Daniel Pocock wrote:

 I made up a rough diagram about how Ganglia 4.x could look:

 https://raw.githubusercontent.com/ganglia/monitor-core/master/doc/planning/ganglia-4.x.png

 The biggest change is the introduction of MongoDB

 Instead of having the gmetad serve up an XML every time somebody asks to
 see the web page, the gmetad will just store current values into MongoDB.

 This means that web frameworks (like PHP) can query the data from
 MongoDB, which is much more horizontally scalable and more suited to
 serving this data.  For large sites where many users access the web
 reports, this will be very useful.

 MongoDB is also a backend for rsyslog daemon now and could potentially
 be a Nagios backend, so it would be a great way to unify monitoring data.

 The introduction of RabbitMQ is an optional dependency.  It would allow
 users to send commands from the web interface.

 One of the motivations for this work is the Google Summer of Code
 projects.  Each student can potentially work on a different part of this
 puzzle and at the end of the year we could launch it as Ganglia 4.0 if
 people like it.

 Regards,

 Daniel


 --
 ___
 Ganglia-general mailing list
 Ganglia-general@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/ganglia-general


 --
 ___
 Ganglia-general mailing list
 Ganglia-general@lists.sourceforge.net
 https://lists.sourceforge.net/lists/listinfo/ganglia-general

--
___
Ganglia-general mailing list
Ganglia-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ganglia-general