It's a worthy mention. Our existing pruner wouldn't be able to handle Solr without modification, so we'd either need something native to Solr or something custom.
Mike On Mon, Nov 27, 2017 at 3:46 PM, James Sirota <[email protected]> wrote: > One thing to keep in mind, as we will be introducing Solr shortly, is to > find if something similar to curator exists for Solr. But we'll cross that > bridge when we get there > > 22.11.2017, 22:58, "Ali Nazemian" <[email protected]>: > > Sure. I will have a chat internally and come back to you shortly. It was > a > > quick and dirty work actually just to fix this temporarily. However, it > > might be a good starting point. > > > > On Thu, Nov 23, 2017 at 3:31 PM, Michael Miklavcic < > > [email protected]> wrote: > > > >> Thanks Ali, that's good feedback. Would you be willing to share any of > your > >> Curator calls/config and use cases with the community? I'd love to add > it > >> to a document around ES pruning in the short term, and maybe we could > look > >> at how to build this into indexing at some point. > >> > >> Cheers, > >> Mike > >> > >> On Nov 22, 2017 8:53 PM, "Ali Nazemian" <[email protected]> wrote: > >> > >> > We tried to use it, but we had the same issue. It was not > documented. We > >> > tried to use it, and we had some issues. It also was not exactly > what we > >> > wanted, so we decided to create something from scratch by using > >> > Elasticsearch Curator. We wanted to have an ability to manage > different > >> > prune mechanism for different feeds. Having a hard threshold to > remove > >> > index and Soft threshold to close that index. Maybe it can be a > feature > >> to > >> > add to the indexing JSON config file per feed. > >> > > >> > Cheers, > >> > Ali > >> > > >> > On Thu, Nov 23, 2017 at 12:20 PM, Michael Miklavcic < > >> > [email protected]> wrote: > >> > > >> > > From what I can tell, the data pruner isn't documented anywhere, > so I'm > >> > > curious if anybody is using this, and if so, how are you using it? > >> > > > >> > > - > >> > > https://github.com/apache/metron/blob/master/metron- > >> > > platform/metron-data-management/README.md > >> > > - > >> > > https://github.com/apache/metron/blob/master/metron- > >> > > platform/metron-data-management/src/main/java/org/ > >> > > apache/metron/dataloads/bulk/ElasticsearchDataPrunerRunner.java > >> > > - > >> > > https://github.com/apache/metron/blob/master/metron- > >> > > platform/metron-data-management/src/main/java/org/ > >> > > apache/metron/dataloads/bulk/DataPruner.java > >> > > > >> > > It looks to me that it allows you to specify the start date and a > >> number > >> > of > >> > > days for lookback from the start date to purge along with a regex > >> pattern > >> > > to match the index name. It also does not look like it has any > built-in > >> > > scheduling semantics, so I assume this was a cron job. I think that > >> about > >> > > covers it. Anything I've missed? > >> > > > >> > > I'm adding a quick doc write-up to METRON-939 ( > >> > > https://github.com/apache/metron/pull/840) for using Curator to > prune > >> > > indices from Elasticsearch. It is desirable to make sure I've > covered > >> > > existing use cases. > >> > > > >> > > Best, > >> > > Mike > >> > > > >> > > >> > > >> > > >> > -- > >> > A.Nazemian > >> > > > > > -- > > A.Nazemian > > ------------------- > Thank you, > > James Sirota > PMC- Apache Metron > jsirota AT apache DOT org >
