Thanks Ali, that's good feedback. Would you be willing to share any of your Curator calls/config and use cases with the community? I'd love to add it to a document around ES pruning in the short term, and maybe we could look at how to build this into indexing at some point.
Cheers, Mike On Nov 22, 2017 8:53 PM, "Ali Nazemian" <[email protected]> wrote: > We tried to use it, but we had the same issue. It was not documented. We > tried to use it, and we had some issues. It also was not exactly what we > wanted, so we decided to create something from scratch by using > Elasticsearch Curator. We wanted to have an ability to manage different > prune mechanism for different feeds. Having a hard threshold to remove > index and Soft threshold to close that index. Maybe it can be a feature to > add to the indexing JSON config file per feed. > > Cheers, > Ali > > On Thu, Nov 23, 2017 at 12:20 PM, Michael Miklavcic < > [email protected]> wrote: > > > From what I can tell, the data pruner isn't documented anywhere, so I'm > > curious if anybody is using this, and if so, how are you using it? > > > > - > > https://github.com/apache/metron/blob/master/metron- > > platform/metron-data-management/README.md > > - > > https://github.com/apache/metron/blob/master/metron- > > platform/metron-data-management/src/main/java/org/ > > apache/metron/dataloads/bulk/ElasticsearchDataPrunerRunner.java > > - > > https://github.com/apache/metron/blob/master/metron- > > platform/metron-data-management/src/main/java/org/ > > apache/metron/dataloads/bulk/DataPruner.java > > > > It looks to me that it allows you to specify the start date and a number > of > > days for lookback from the start date to purge along with a regex pattern > > to match the index name. It also does not look like it has any built-in > > scheduling semantics, so I assume this was a cron job. I think that about > > covers it. Anything I've missed? > > > > I'm adding a quick doc write-up to METRON-939 ( > > https://github.com/apache/metron/pull/840) for using Curator to prune > > indices from Elasticsearch. It is desirable to make sure I've covered > > existing use cases. > > > > Best, > > Mike > > > > > > -- > A.Nazemian >
