A few examples of index recalc performance differences: In 1.1.6 ... 2015-09-04_03:56:38.19431 INFO [RebuildIndexRangesJob] Calculated range of [graylog_2075] in [9ms]. 2015-09-04_03:56:38.20487 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1800] in [10ms]. 2015-09-04_03:56:38.21176 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1802] in [6ms]. 2015-09-04_03:56:38.22028 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1801] in [8ms]. 2015-09-04_03:56:38.23003 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1808] in [9ms]. 2015-09-04_03:56:38.23855 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1807] in [8ms]. 2015-09-04_03:56:38.24877 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1809] in [10ms]. 2015-09-04_03:56:38.26011 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1804] in [11ms]. 2015-09-04_03:56:38.26902 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1803] in [8ms]. 2015-09-04_03:56:38.28049 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1806] in [11ms]. 2015-09-04_03:56:38.29289 INFO [RebuildIndexRangesJob] Calculated range of [graylog_1805] in [12ms]. 2015-09-04_03:56:38.41557 INFO [RebuildIndexRangesJob] Done calculating index ranges for 800 indices. Took 10608ms.
In 1.2.1: ... 2015-10-03_17:00:59.70210 INFO [EsIndexRangeService] Calculated range of [graylog_3358] in [1609ms]. 2015-10-03_17:01:05.63202 INFO [EsIndexRangeService] Calculated range of [graylog_3357] in [50ms]. 2015-10-03_17:01:08.85062 INFO [EsIndexRangeService] Calculated range of [graylog_3359] in [47ms]. 2015-10-03_17:01:12.39922 INFO [EsIndexRangeService] Calculated range of [graylog_3365] in [119ms]. 2015-10-03_17:01:15.80573 INFO [EsIndexRangeService] Calculated range of [graylog_3364] in [40ms]. 2015-10-03_17:01:19.29222 INFO [EsIndexRangeService] Calculated range of [graylog_3367] in [43ms]. 2015-10-03_17:01:22.98674 INFO [EsIndexRangeService] Calculated range of [graylog_3366] in [142ms]. 2015-10-03_17:01:29.74936 INFO [EsIndexRangeService] Calculated range of [graylog_3361] in [120ms]. 2015-10-03_17:01:34.11754 INFO [EsIndexRangeService] Calculated range of [graylog_3360] in [128ms]. 2015-10-03_17:01:37.91544 INFO [EsIndexRangeService] Calculated range of [graylog_3363] in [134ms]. 2015-10-03_17:01:41.41116 INFO [EsIndexRangeService] Calculated range of [graylog_3362] in [237ms]. 2015-10-03_17:01:45.31576 INFO [RebuildIndexRangesJob] Done calculating index ranges for 1855 indices. Took 9551409ms. For good measure, I tried scaling the ES cluster up and down. Performance is exactly the same whether I run on a single large server vs 4 smaller servers. On Friday, October 2, 2015 at 9:29:37 AM UTC-6, Jesse Skrivseth wrote: > > Note that for the 1800 indices we have in this instance, the ~5 second > delay between each index during range calculation adds up to ~150 minutes > of additional delay when calculating ranges. My "> 20 minutes" comment > should be more like "> 2.5 hours", plus actual time spent calculating > ranges. > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/073650ab-3892-4657-a341-a83d8ab78dec%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
