I'm running elasticsearch much smaller than this, but with a PowerEdge R900 with 2 X7350 CPUs, and 64 GB of RAM (24GB heap for elasticsearch) I'm able to sustain something like 80GB per day (1/16 your volume). Some of the latest Intel CPUs are about 4 times as powerful as the X7350, so extrapolating from my results, with very new hardware you can probably do 1.25TB per day on around 5 nodes with 2 CPUs, 256GB RAM, and 8 disks each. I haven't had an opportunity to test this yet, and even if this is possible, you should probably get have more nodes than this; hardware failure, growth, or a sudden increase in logging volume from a problem can take down a cluster that's running at full capacity all the time.
I'd encourage you to put elasticsearch on some of your systems to generate some benchmarks. I've never tried clustering elasticsearch with more than 5 hosts. At 1300 systems, each would be doing around 15 KB/s, which is essentially trivial. You might try taking splunk off 2 dozen systems or so, and committing them to elasticsearch, then see how well they keep up with the load you're generating. Data from your particular setup will almost always be the best sort to have. -- The information transmitted in this email is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this email in error, please contact the sender and permanently delete the email from any computer. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d465a805-0ada-4398-b4d8-f8ab56e4f34b%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
