Hi Mihai, This sounds like something you could do in a pre-processing pipeline before indexing with Elasticsearch. Have you heard of Logstash? It is designed to slurp up logs, filter them (including detecting and removing duplicates) then insert them in Elasticsearch (or elsewhere). It can handle live streaming of logs or can be run on existing log files. Definitely check it out, but I'd imagine if not Logstash, some other kind of pre-processing is going to be your best bet.
Regards, Joshua On Tuesday, 3 March 2015 01:45:18 UTC+11, Mihai Lucaciu wrote: > > Hi, > > I am new to Elasticsearch which I understand can do much more than this... > but could it be used just for that ? > > I am storing 100GB of log files daily. The data scientists require this > log data to not contain duplicate log lines. Duplicates may come within the > same log file, with two sequential log files - it's better to expect any > possible scenario. > > What I would like to achieve is to use Elasticsearch to detect & remove > the duplicate log lines from all logs in an HDFS directory. Can this be > done ? > > Thank you, > Mihai > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b47ca8d1-6ed2-4165-a021-820b26aa44de%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
