That's a massive volume! You should be able to get this to work, though you'll need to plan things and have access to a large number of servers and easy provisioning to let you expand as you grow.
Given the volume you may want to consider hourly indexes. Index size isn't so much a problem as segment size, you should aim to keep each segment between 10-20GB to allow for optimal reallocation and merging. You should also consider a tiered system, have your active (eg last 6-12 hours) indexes on a few high performance machines with SSDs/flash to allow optimal indexing and retrieval, and then move the indexes onto slower SAS/SATA drives after that time. A lot of what you may need depends on our retention period and how you expect to have the data accessed. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [email protected] web: www.campaignmonitor.com On 20 September 2014 15:22, Nelson Jeppesen <[email protected]> wrote: > Can ES scale to 30TB / day, and still be usable? > > This is a typical logstash/elasticsearch/kibana setup. I have a small > environment logging 20GB / day that seems to work fine. At 30TB, very > little will be able to cached into ram, can ES still be usable at that > point? > > Also, what's is the best way to pick the proper index creation rate (per > day, per hour?). Is there a guideline for max. index size? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/865e67f5-bcd4-4247-9f39-813424d6747c%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/865e67f5-bcd4-4247-9f39-813424d6747c%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bM1f-2kXZq2w3jHnTuaAnT7s2Qsm0D5kE6JmsFHGvkmg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
