Hi Jesse, did you manually trigger the calculation of all the index ranges?
During normal operations of Graylog 1.2.x, only the index range of the latest index should be calculated and stored (in contrast to Graylog 1.1.x and earlier, which most of the time calculated all index ranges). Cheers, Jochen On Saturday, 3 October 2015 19:11:15 UTC+2, Jesse Skrivseth wrote: > > A few examples of index recalc performance differences: > > In 1.1.6 > ... > 2015-09-04_03:56:38.19431 INFO [RebuildIndexRangesJob] Calculated range > of [graylog_2075] in [9ms]. > 2015-09-04_03:56:38.20487 INFO [RebuildIndexRangesJob] Calculated range > of [graylog_1800] in [10ms]. > 2015-09-04_03:56:38.21176 INFO [RebuildIndexRangesJob] Calculated range > of [graylog_1802] in [6ms]. > 2015-09-04_03:56:38.22028 INFO [RebuildIndexRangesJob] Calculated range > of [graylog_1801] in [8ms]. > 2015-09-04_03:56:38.23003 INFO [RebuildIndexRangesJob] Calculated range > of [graylog_1808] in [9ms]. > 2015-09-04_03:56:38.23855 INFO [RebuildIndexRangesJob] Calculated range > of [graylog_1807] in [8ms]. > 2015-09-04_03:56:38.24877 INFO [RebuildIndexRangesJob] Calculated range > of [graylog_1809] in [10ms]. > 2015-09-04_03:56:38.26011 INFO [RebuildIndexRangesJob] Calculated range > of [graylog_1804] in [11ms]. > 2015-09-04_03:56:38.26902 INFO [RebuildIndexRangesJob] Calculated range > of [graylog_1803] in [8ms]. > 2015-09-04_03:56:38.28049 INFO [RebuildIndexRangesJob] Calculated range > of [graylog_1806] in [11ms]. > 2015-09-04_03:56:38.29289 INFO [RebuildIndexRangesJob] Calculated range > of [graylog_1805] in [12ms]. > 2015-09-04_03:56:38.41557 INFO [RebuildIndexRangesJob] Done calculating > index ranges for 800 indices. Took 10608ms. > > In 1.2.1: > ... > 2015-10-03_17:00:59.70210 INFO [EsIndexRangeService] Calculated range of > [graylog_3358] in [1609ms]. > 2015-10-03_17:01:05.63202 INFO [EsIndexRangeService] Calculated range of > [graylog_3357] in [50ms]. > 2015-10-03_17:01:08.85062 INFO [EsIndexRangeService] Calculated range of > [graylog_3359] in [47ms]. > 2015-10-03_17:01:12.39922 INFO [EsIndexRangeService] Calculated range of > [graylog_3365] in [119ms]. > 2015-10-03_17:01:15.80573 INFO [EsIndexRangeService] Calculated range of > [graylog_3364] in [40ms]. > 2015-10-03_17:01:19.29222 INFO [EsIndexRangeService] Calculated range of > [graylog_3367] in [43ms]. > 2015-10-03_17:01:22.98674 INFO [EsIndexRangeService] Calculated range of > [graylog_3366] in [142ms]. > 2015-10-03_17:01:29.74936 INFO [EsIndexRangeService] Calculated range of > [graylog_3361] in [120ms]. > 2015-10-03_17:01:34.11754 INFO [EsIndexRangeService] Calculated range of > [graylog_3360] in [128ms]. > 2015-10-03_17:01:37.91544 INFO [EsIndexRangeService] Calculated range of > [graylog_3363] in [134ms]. > 2015-10-03_17:01:41.41116 INFO [EsIndexRangeService] Calculated range of > [graylog_3362] in [237ms]. > 2015-10-03_17:01:45.31576 INFO [RebuildIndexRangesJob] Done calculating > index ranges for 1855 indices. Took 9551409ms. > > For good measure, I tried scaling the ES cluster up and down. Performance > is exactly the same whether I run on a single large server vs 4 smaller > servers. > > On Friday, October 2, 2015 at 9:29:37 AM UTC-6, Jesse Skrivseth wrote: >> >> Note that for the 1800 indices we have in this instance, the ~5 second >> delay between each index during range calculation adds up to ~150 minutes >> of additional delay when calculating ranges. My "> 20 minutes" comment >> should be more like "> 2.5 hours", plus actual time spent calculating >> ranges. >> > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/fe56e05b-12da-497c-955b-88aebd4844d4%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
