Re: Interesting stuff; Solr as a syslog store.
Thanks Antonio for sharing this. I believe this could be one of the interesting case studies for Solr In Action, if you are interested in sharing a bit more - I am sure the authors would be more interested for upcoming revisions. -- K K. On 02/12/2010 06:02 PM, Antonio Lobato wrote: Hey everyone, I don't actually have a question, but I just thought I'd share something really cool that I did with Solr for our company. We run a good amount of servers, well into the several hundreds, and naturally we need a way to centralize all of the system logs. For a while we used a commercial solution to centralize and search our logs, but they wanted to charge us tens of thousands of dollars for just one gigabyte/day more of indexed data. So I said forget it, I'll write my own solution! We already use Solr for some of our other backend searching systems, so I came up with an idea to index all of our logs to Solr. I wrote a daemon in perl that listens on the syslog port, and pointed every single system's syslog to forward to this single server. From there, this daemon will write to a Solr indexing server after parsing them into fields, such as date/time, host, program, pid, text, etc. I then wrote a cool javascript/ajax web front end for Solr searching, and bam. Real time searching of all of our syslogs from a web interface, for no cost! Just thought this would be a neat story to share with you all. I've really grown to love Solr, it's something else! Thanks, -Antonio
Interesting stuff; Solr as a syslog store.
Hey everyone, I don't actually have a question, but I just thought I'd share something really cool that I did with Solr for our company. We run a good amount of servers, well into the several hundreds, and naturally we need a way to centralize all of the system logs. For a while we used a commercial solution to centralize and search our logs, but they wanted to charge us tens of thousands of dollars for just one gigabyte/day more of indexed data. So I said forget it, I'll write my own solution! We already use Solr for some of our other backend searching systems, so I came up with an idea to index all of our logs to Solr. I wrote a daemon in perl that listens on the syslog port, and pointed every single system's syslog to forward to this single server. From there, this daemon will write to a Solr indexing server after parsing them into fields, such as date/time, host, program, pid, text, etc. I then wrote a cool javascript/ajax web front end for Solr searching, and bam. Real time searching of all of our syslogs from a web interface, for no cost! Just thought this would be a neat story to share with you all. I've really grown to love Solr, it's something else! Thanks, -Antonio
Re: Interesting stuff; Solr as a syslog store.
Am 13.02.2010 um 03:02 schrieb Antonio Lobato: Just thought this would be a neat story to share with you all. I've really grown to love Solr, it's something else! Hi Antonio, Great. Would you also share the source code somewhere! May the Source be with you. Thanks. Olivier
