Re: [Puppet Users] Historical reporting, BI from puppetdb?
I mentioned Pentaho (albeit spelled incorrectly) as an example, as I saw it mentioned a number of times online due to its open source community variant and full feature set. Thanks for the links, it appears that slurping data into Pentaho via puppetdb's REST API or otherwise is quite possible if enough time is put into it. I'm still wondering if anyone is doing the type of BI I'm talking about today against puppetdb? If so, with what tools? I'm willing to try out a couple solutions, but if there are others doing this successfully, I'd rather not reinvent the wheel. Thanks, RCA On Wednesday, June 18, 2014 12:10:48 AM UTC-5, Ken Barber wrote: And others: http://wiki.pentaho.com/display/EAI/JSON+Input http://wiki.pentaho.com/display/EAI/HTTP+Client Perhaps, you could combine the JSON input perhaps on the contents of a 'puppetdb export' tarball if you wanted to analyze the data from a previous backup :-). ken. On Tue, Jun 17, 2014 at 10:01 PM, Ken Barber k...@puppetlabs.com javascript: wrote: Ryan, What about something like this? http://wiki.pentaho.com/display/EAI/Rest+Client This page seems to mix in general actions with integration steps, but there are more integration types available here: http://wiki.pentaho.com/display/EAI/Pentaho+Data+Integration+Steps ken. On Tue, Jun 17, 2014 at 9:53 PM, Ken Barber k...@puppetlabs.com javascript: wrote: Sorry, do you mean Pentaho? On Tue, Jun 17, 2014 at 9:53 PM, Ken Barber k...@puppetlabs.com javascript: wrote: Thanks, good to know. While the REST API would be the method to get at the data, my issue is that I'm not capable of writing a web app + data repository that can generate web-based reports, etc. I've actually gotten into the habit of running one-off queries using the API with curl to get YAML-formatted reports, which in part got me wanting more. Question, this BI tool Pentah, what formats/apis is it able to ingest beyond SQL? Are there any ETL capabilities built into this tool? Would something like CSV work? ken. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f7735e83-a9f3-444b-bac0-f3db5a8a147f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[Puppet Users] Historical reporting, BI from puppetdb?
I use puppetdb + puppetboard, which are very useful to see the current state of my environment. Puppetboard also provides a very nice representation of each agent's most recent reports. However, I want to take it to the next level and create custom historical reports for business intelligence (eg How many changes per day over the last month? How many RHEL5 virtual systems as of $DATE?). I've looked online and not found many solutions on this. Might Foreman provide this? Even if it doesI'm not sure I'm ready for that commitment. Since puppetdb uses PostgreSQL, I'd imagine most any BI tool that supports it (eg Pentah) could be configured to get what I want. However, is anyone successfully doing this? -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f434802e-822c-4a69-88a6-c4651e3337f8%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Historical reporting, BI from puppetdb?
I use puppetdb + puppetboard, which are very useful to see the current state of my environment. Puppetboard also provides a very nice representation of each agent's most recent reports. However, I want to take it to the next level and create custom historical reports for business intelligence (eg How many changes per day over the last month? How many RHEL5 virtual systems as of $DATE?). I've looked online and not found many solutions on this. Might Foreman provide this? Even if it doesI'm not sure I'm ready for that commitment. Since puppetdb uses PostgreSQL, I'd imagine most any BI tool that supports it (eg Pentah) could be configured to get what I want. However, is anyone successfully doing this? So something to note, while this is an interesting idea, you have to be careful. We make zero commitments about the stability of the database schema today ... so expect an upgrade to break things, especially if we have a migration. In short, accessing the data directly is not a supported stable API. Not saying it won't work, but just letting you know the risks involved here. Not sure how useful it is, we do have a lot of complex query capability just built into PDB via the REST interface which might be able to provide what you need: http://docs.puppetlabs.com/puppetdb/2.0/api/query/v4/query.html ken. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAE4bNTmU6MhRDXN2mQ0_4T%3DOTpz-ZboWfUeMZ7%3Dzd-V8nW%3DGvg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Historical reporting, BI from puppetdb?
You absolutely want to build that against the puppetdb rest interface and not by hitting postgres directly. I'd be interested in getting at this data as well, if you find a way or build a neat little application to do it. On Tue, Jun 17, 2014 at 3:35 PM, Ken Barber k...@puppetlabs.com wrote: I use puppetdb + puppetboard, which are very useful to see the current state of my environment. Puppetboard also provides a very nice representation of each agent's most recent reports. However, I want to take it to the next level and create custom historical reports for business intelligence (eg How many changes per day over the last month? How many RHEL5 virtual systems as of $DATE?). I've looked online and not found many solutions on this. Might Foreman provide this? Even if it doesI'm not sure I'm ready for that commitment. Since puppetdb uses PostgreSQL, I'd imagine most any BI tool that supports it (eg Pentah) could be configured to get what I want. However, is anyone successfully doing this? So something to note, while this is an interesting idea, you have to be careful. We make zero commitments about the stability of the database schema today ... so expect an upgrade to break things, especially if we have a migration. In short, accessing the data directly is not a supported stable API. Not saying it won't work, but just letting you know the risks involved here. Not sure how useful it is, we do have a lot of complex query capability just built into PDB via the REST interface which might be able to provide what you need: http://docs.puppetlabs.com/puppetdb/2.0/api/query/v4/query.html ken. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAE4bNTmU6MhRDXN2mQ0_4T%3DOTpz-ZboWfUeMZ7%3Dzd-V8nW%3DGvg%40mail.gmail.com . For more options, visit https://groups.google.com/d/optout. -- Spencer Krum (619)-980-7820 -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CADt6FWOZV6vF28M1CwWL6sjb_JJJ0x4WXEzojbySnEBMDX%2BeEQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Historical reporting, BI from puppetdb?
Thanks, good to know. While the REST API would be the method to get at the data, my issue is that I'm not capable of writing a web app + data repository that can generate web-based reports, etc. I've actually gotten into the habit of running one-off queries using the API with curl to get YAML-formatted reports, which in part got me wanting more. On Tuesday, June 17, 2014 5:36:07 PM UTC-5, Ken Barber wrote: I use puppetdb + puppetboard, which are very useful to see the current state of my environment. Puppetboard also provides a very nice representation of each agent's most recent reports. However, I want to take it to the next level and create custom historical reports for business intelligence (eg How many changes per day over the last month? How many RHEL5 virtual systems as of $DATE?). I've looked online and not found many solutions on this. Might Foreman provide this? Even if it doesI'm not sure I'm ready for that commitment. Since puppetdb uses PostgreSQL, I'd imagine most any BI tool that supports it (eg Pentah) could be configured to get what I want. However, is anyone successfully doing this? So something to note, while this is an interesting idea, you have to be careful. We make zero commitments about the stability of the database schema today ... so expect an upgrade to break things, especially if we have a migration. In short, accessing the data directly is not a supported stable API. Not saying it won't work, but just letting you know the risks involved here. Not sure how useful it is, we do have a lot of complex query capability just built into PDB via the REST interface which might be able to provide what you need: http://docs.puppetlabs.com/puppetdb/2.0/api/query/v4/query.html ken. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/e56002b6-946f-4a30-957c-a45df13a751e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Historical reporting, BI from puppetdb?
Thanks, good to know. While the REST API would be the method to get at the data, my issue is that I'm not capable of writing a web app + data repository that can generate web-based reports, etc. I've actually gotten into the habit of running one-off queries using the API with curl to get YAML-formatted reports, which in part got me wanting more. Question, this BI tool Pentah, what formats/apis is it able to ingest beyond SQL? Are there any ETL capabilities built into this tool? Would something like CSV work? ken. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAE4bNTkHGudYw9BEUUK2PhemOdB-znzwvaHJ08SU0avBt0yztQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Historical reporting, BI from puppetdb?
Sorry, do you mean Pentaho? On Tue, Jun 17, 2014 at 9:53 PM, Ken Barber k...@puppetlabs.com wrote: Thanks, good to know. While the REST API would be the method to get at the data, my issue is that I'm not capable of writing a web app + data repository that can generate web-based reports, etc. I've actually gotten into the habit of running one-off queries using the API with curl to get YAML-formatted reports, which in part got me wanting more. Question, this BI tool Pentah, what formats/apis is it able to ingest beyond SQL? Are there any ETL capabilities built into this tool? Would something like CSV work? ken. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAE4bNTktZTMegALk8Az6Q6PpoxgSzLUHBmJVL%2BYwBX%3DpF0bVtQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Historical reporting, BI from puppetdb?
Ryan, What about something like this? http://wiki.pentaho.com/display/EAI/Rest+Client This page seems to mix in general actions with integration steps, but there are more integration types available here: http://wiki.pentaho.com/display/EAI/Pentaho+Data+Integration+Steps ken. On Tue, Jun 17, 2014 at 9:53 PM, Ken Barber k...@puppetlabs.com wrote: Sorry, do you mean Pentaho? On Tue, Jun 17, 2014 at 9:53 PM, Ken Barber k...@puppetlabs.com wrote: Thanks, good to know. While the REST API would be the method to get at the data, my issue is that I'm not capable of writing a web app + data repository that can generate web-based reports, etc. I've actually gotten into the habit of running one-off queries using the API with curl to get YAML-formatted reports, which in part got me wanting more. Question, this BI tool Pentah, what formats/apis is it able to ingest beyond SQL? Are there any ETL capabilities built into this tool? Would something like CSV work? ken. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAE4bNTmkViViz%3D466%3D_rrkiCGF7bRY7GmmcAnq1DYSyX3Ag1Ow%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet Users] Historical reporting, BI from puppetdb?
And others: http://wiki.pentaho.com/display/EAI/JSON+Input http://wiki.pentaho.com/display/EAI/HTTP+Client Perhaps, you could combine the JSON input perhaps on the contents of a 'puppetdb export' tarball if you wanted to analyze the data from a previous backup :-). ken. On Tue, Jun 17, 2014 at 10:01 PM, Ken Barber k...@puppetlabs.com wrote: Ryan, What about something like this? http://wiki.pentaho.com/display/EAI/Rest+Client This page seems to mix in general actions with integration steps, but there are more integration types available here: http://wiki.pentaho.com/display/EAI/Pentaho+Data+Integration+Steps ken. On Tue, Jun 17, 2014 at 9:53 PM, Ken Barber k...@puppetlabs.com wrote: Sorry, do you mean Pentaho? On Tue, Jun 17, 2014 at 9:53 PM, Ken Barber k...@puppetlabs.com wrote: Thanks, good to know. While the REST API would be the method to get at the data, my issue is that I'm not capable of writing a web app + data repository that can generate web-based reports, etc. I've actually gotten into the habit of running one-off queries using the API with curl to get YAML-formatted reports, which in part got me wanting more. Question, this BI tool Pentah, what formats/apis is it able to ingest beyond SQL? Are there any ETL capabilities built into this tool? Would something like CSV work? ken. -- You received this message because you are subscribed to the Google Groups Puppet Users group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAE4bNTmG2ugH4KOp-TsjVC67wUKqJghRDrfHQrpVtcvKV0_yCA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
