Sorry, Michael. I am having some issues to share any code right now. It seems we need to go through an internal verification of anything we want to share. BTW, the curator script I mentioned is very simple and nothing special. It doesn't worth to waste any time on waiting for it.
Cheers, Ali On Tue, Nov 28, 2017 at 9:56 AM, Michael Miklavcic < michael.miklav...@gmail.com> wrote: > It's a worthy mention. Our existing pruner wouldn't be able to handle Solr > without modification, so we'd either need something native to Solr or > something custom. > > Mike > > On Mon, Nov 27, 2017 at 3:46 PM, James Sirota <jsir...@apache.org> wrote: > > > One thing to keep in mind, as we will be introducing Solr shortly, is to > > find if something similar to curator exists for Solr. But we'll cross > that > > bridge when we get there > > > > 22.11.2017, 22:58, "Ali Nazemian" <alinazem...@gmail.com>: > > > Sure. I will have a chat internally and come back to you shortly. It > was > > a > > > quick and dirty work actually just to fix this temporarily. However, it > > > might be a good starting point. > > > > > > On Thu, Nov 23, 2017 at 3:31 PM, Michael Miklavcic < > > > michael.miklav...@gmail.com> wrote: > > > > > >> Thanks Ali, that's good feedback. Would you be willing to share any > of > > your > > >> Curator calls/config and use cases with the community? I'd love to > add > > it > > >> to a document around ES pruning in the short term, and maybe we could > > look > > >> at how to build this into indexing at some point. > > >> > > >> Cheers, > > >> Mike > > >> > > >> On Nov 22, 2017 8:53 PM, "Ali Nazemian" <alinazem...@gmail.com> > wrote: > > >> > > >> > We tried to use it, but we had the same issue. It was not > > documented. We > > >> > tried to use it, and we had some issues. It also was not exactly > > what we > > >> > wanted, so we decided to create something from scratch by using > > >> > Elasticsearch Curator. We wanted to have an ability to manage > > different > > >> > prune mechanism for different feeds. Having a hard threshold to > > remove > > >> > index and Soft threshold to close that index. Maybe it can be a > > feature > > >> to > > >> > add to the indexing JSON config file per feed. > > >> > > > >> > Cheers, > > >> > Ali > > >> > > > >> > On Thu, Nov 23, 2017 at 12:20 PM, Michael Miklavcic < > > >> > michael.miklav...@gmail.com> wrote: > > >> > > > >> > > From what I can tell, the data pruner isn't documented anywhere, > > so I'm > > >> > > curious if anybody is using this, and if so, how are you using > it? > > >> > > > > >> > > - > > >> > > https://github.com/apache/metron/blob/master/metron- > > >> > > platform/metron-data-management/README.md > > >> > > - > > >> > > https://github.com/apache/metron/blob/master/metron- > > >> > > platform/metron-data-management/src/main/java/org/ > > >> > > apache/metron/dataloads/bulk/ElasticsearchDataPrunerRunner.java > > >> > > - > > >> > > https://github.com/apache/metron/blob/master/metron- > > >> > > platform/metron-data-management/src/main/java/org/ > > >> > > apache/metron/dataloads/bulk/DataPruner.java > > >> > > > > >> > > It looks to me that it allows you to specify the start date and a > > >> number > > >> > of > > >> > > days for lookback from the start date to purge along with a regex > > >> pattern > > >> > > to match the index name. It also does not look like it has any > > built-in > > >> > > scheduling semantics, so I assume this was a cron job. I think > that > > >> about > > >> > > covers it. Anything I've missed? > > >> > > > > >> > > I'm adding a quick doc write-up to METRON-939 ( > > >> > > https://github.com/apache/metron/pull/840) for using Curator to > > prune > > >> > > indices from Elasticsearch. It is desirable to make sure I've > > covered > > >> > > existing use cases. > > >> > > > > >> > > Best, > > >> > > Mike > > >> > > > > >> > > > >> > > > >> > > > >> > -- > > >> > A.Nazemian > > >> > > > > > > > -- > > > A.Nazemian > > > > ------------------- > > Thank you, > > > > James Sirota > > PMC- Apache Metron > > jsirota AT apache DOT org > > > -- A.Nazemian
