Re: Use Solr / Lucene to search in a Logfile
Hello Robert, At my company, we are working on a generic log collector that uses Solr to provide search capabilities. What the collector does basically is this (this is greatly dumbed down !) : * collect a log line (read it from a file, receive it from the network, ... ) * parse it through a set of regular expressions, searching for known log formats ( apache CLF, ... ) * if there is a match, store the results as a set of keys/values ( url : http://www.apache.org , source : XXX , raw_log : xx , ... ) * insert the set as a document in the Solr backend, using the REST interface. Therefore I would advise you to adapt this workflow to suit your own needs : have a script looking for new lines in your log file, parse them in order to extract the relevant information you need, store the results as keys/values sets, then insert them into Solr via a http call. My company's product is probably overkill for what you need to do, and we'd probably need to develop a specific log parser for your log format, but if you are willing to give it a try feel free to contact me ! Greetings, Matthieu HUIN On 06/05/2011 21:40, Robert Naczinski wrote: Hi, thanks for the reply. I did not know that. Is there still a way to use Solr or Lucene? Or Apache Nutch would be not be bad. Could I maybe write a customized DIH? Greetings, Robert 2011/5/6 Otis Gospodneticotis_gospodne...@yahoo.com: Loggly.com
Use Solr / Lucene to search in a Logfile
Hello, we want to search large log4j logfiles withSolr / Lucene and find any lines with a special argument. ( for examle: with a defined userid ) It should be a solr indput document for each row in the file. The log file is growing continuously and the search index must be refreshed. Has already anybidy implemented something like this and can give me a tip ? Greetings, Robert
Re: Use Solr / Lucene to search in a Logfile
Hi Robert, Have you considered just using Loggly.com ? Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Lucene ecosystem search :: http://search-lucene.com/ - Original Message From: Robert Naczinski robert.naczin...@googlemail.com To: solr-user@lucene.apache.org Sent: Fri, May 6, 2011 3:01:45 AM Subject: Use Solr / Lucene to search in a Logfile Hello, we want to search large log4j logfiles withSolr / Lucene and find any lines with a special argument. ( for examle: with a defined userid ) It should be a solr indput document for each row in the file. The log file is growing continuously and the search index must be refreshed. Has already anybidy implemented something like this and can give me a tip ? Greetings, Robert
Re: Use Solr / Lucene to search in a Logfile
Hi, thanks for the reply. I did not know that. Is there still a way to use Solr or Lucene? Or Apache Nutch would be not be bad. Could I maybe write a customized DIH? Greetings, Robert 2011/5/6 Otis Gospodnetic otis_gospodne...@yahoo.com: Loggly.com
