Sorry, I am not really following you. :) Maybe reading about our existing index model first brings some more light into the topic: http://support.torch.sh/help/kb/general/the-graylog2-index-model-explained
We are not planning to change the index model at all. All we are doing is adding another retention strategy which you can configure to deleted based on message time and not the index count. Graylog2 is taking care of all the rest from there. You can still use the REST APIs if you want to do that manually. Keeping each index as small as possible gives you better responsiveness on shorter search time periods but also brings ElasticSearch overhead. Always depends on how many messages you have in total, what hardware you are running and what your common searches and queries look like. On Tue, Mar 4, 2014 at 12:15 AM, Aristoteles Neto <[email protected]> wrote: > I see. The use case is indeed to be able to retain x number of months worth > of logs, which isn't really dependent upon the index cycling method. > > My assumption was that would've been more effective to accomplish it by > rotating indexes - as it would require less I/O for deletion. The way I > understood your message, I gather than the method you prefer would be > something like: > > index1 - X messages, dated from A-B-C > index2 - X messages, dated from D-E-F > index3 - X messages, dated from G-H-I > ... > > Which means that deleting messages older than E would mean delete index3, and > messages older than E within index2. For a count deletion, you would say keep > 2 indexes (thereby deleting index3). > > On my assumption this would have become: > index1 - X messages, dated from A > index2 - Y messages, dated from B > index3 - Z messages, dated from C > index4 - N messages, dated from D > index5 - M messages, dated from E > index6 - L messages, dated from F > ... > > Which means that deleting messages older than E would mean delete index6 > onwards. For a count deletion, if you wanted to keep C messages, you would > delete any messages after sum(x,y,z...)>C (including whole indexes). > > > They both achieve the same outcome - to be able to retain either based on > date, or based on amount - so I'm happy either way. :-) > > I do wonder whether keeping each index as small as possible make any > difference towards responsiveness and resource usage (upon deletion). > > Aristoteles Neto > [email protected] > > > > On 4/03/2014, at 10:16 am, Lennart Koopmann <[email protected]> wrote: > >> >> Hey Aristoteles, >> >> we are not planning to change the index cycling to something like >> daily or hourly. This is because we want maximum flexibility by >> defining index sizes with message count limits. What we are working on >> is **retention cleaning** based on timeframes like days, weeks, months >> or years. >> >> What is your use case that you need daily or hourly indices for? Happy >> to help! :) >> >> Cheers, >> Lennart >> >> On Mon, Mar 3, 2014 at 2:22 AM, Aristoteles Neto <[email protected]> wrote: >>> Hi Guys, >>> >>> Love the work you guys are doing with Graylog2, and I'd love to replace >>> Logstash with Graylog! >>> >>> Unfortunately theres a few things that are preventing me from doing so. >>> >>> I understand The Daily/Hourly index splitting and retention option was >>> originally planned for v0.20. Now that v0.20.1 is out, was there a more >>> accurate timeframe for having the feature available? Looking through the >>> mailing list, it seems like a frequently touched on subject, so apologies if >>> this has been asked previously. >>> >>> This is one of the things preventing the move from Logstash. >>> >>> Aristoteles Neto >>> [email protected] >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "graylog2" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to [email protected]. >>> For more options, visit https://groups.google.com/groups/opt_out. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "graylog2" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
