Hi Simon, That's true for very large clusters. However, always having lots of shards and indices are not optimum for a small cluster. First, I think having hourly index cannot be one recipe for every cluster without considering the ingestion rate, cluster size and the type of query. Second, query result would be blocked by the slowest shard/index. For example, if I want to have a near-realtime query result for the last 30 days, It doesn't matter how fast Elasticsearch can respond to 719 of the indices. The query result is blocked by the result of the last index. In addition to that, most of the frequent queries target the most recent index, so index-query congestion is not avoidable in this way. Generally, finding the best strategy for indexing depends on the type of use cases and queries and it would be nice to customise it.
Can you please guide me where I can find the corresponding configuration parameter in Metron for this matter? Cheers, Ali On Fri, Jul 14, 2017 at 7:55 PM, Simon Elliston Ball < [email protected]> wrote: > You could change the index data format. One word of caution here though; > the last time I saw this done it caused huge problems with locking on > ingest against people running queries on the current day’s data and tended > to knock recent relevant indexes out of disk cache at the OS level. It > might look like it will help a bit for ingest initially, but with load on > the end user side, it’s probably going to kill your disks at any reasonable > scale. > > Simon > > > On 14 Jul 2017, at 10:31, Ali Nazemian <[email protected]> wrote: > > > > Hi, > > > > I am investigating different tuning aspects, and I was wondering how I > can change the policy of Elasticsearch indexing. Currently, as a default > behaviour, events are stored in separate indices hourly. How can I change > this behaviour? Is this a hard-coded design or I can change it through > configurations? > > > > Cheers, > > Ali > > -- A.Nazemian
