Hi Eugene, once you've identified the duplicate documents in Elasticsearch, you can remove these using the Elasticsearch Delete API (https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-delete.html) or the Delete By Query plugin (https://www.elastic.co/guide/en/elasticsearch/plugins/2.4/delete-by-query-usage.html).
Cheers, Jochen On Thursday, 3 November 2016 11:54:49 UTC+1, Eugene Gwon wrote: > > Hello, Jochen. > > I've checked that link but unfortunately, I'm new to ElasticSearch. and It > seems like that link only shows 'how to check duplicate data'. > does it mean that I insert data first and check duplicate, and than delete > duplicated data? > > I'm confusing :( > I'm really sorry but would you please explain more detail? > > Thanks for your help, Jochen. > > On Thursday, November 3, 2016 at 7:08:29 PM UTC+9, Jochen Schalanda wrote: >> >> Hi Eugene, >> >> Graylog itself doesn't filter out "duplicate" data. >> >> You could check your Elasticsearch indices using the approach laid out in >> https://qbox.io/blog/minimizing-document-duplication-in-elasticsearch >> after ingesting the missing data. >> >> Cheers, >> Jochen >> >> On Thursday, 3 November 2016 09:48:42 UTC+1, Eugene Gwon wrote: >>> >>> Hello all, >>> I have a problem. please help :( >>> I'm running 3 node of Elasticsearch. 1 master + data, and 2 data node. >>> log shipper is fluentd. >>> Unfortunately, today, I accidentally restarted graylog server. >>> this caused lost of some data, which sent from another servers. >>> >>> so I must re-insert data, but WITHOUT duplication. >>> Is it possible? >>> >>> or, can I just filtering duplicated data when search? >>> >>> Thanks for your help. >>> >>> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/577d2204-b1da-4616-8741-3d047d99863e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
