Since netflow data is not text, you're using elasticsearch like a distributed "SQL" database. You should turn of analysis for all of the netflow fields within your ES template, and drop any fields you don't REALLY really need.
Also, consider using pmacct (pmacct.net) as a pre-aggregator and "shipper", if you aggregate to something like every 1min you may get the compression you're looking for and still have good data granularity. Also, 1/2TB per month is pretty cheap... disk is cheap man. CPU and mem is expensive, and you're burning a helluva lot of that at 25k netflow packets/sec... On Tuesday, April 22, 2014 4:26:14 AM UTC-7, horst knete wrote: > Hi, >> > > thanks for your quick response. > > @Jörg: The 576 GB i calculated was from the results we got as we tested > the input of the netflow data ( we tested it 15 Minutes and got about 200MB > of Data ). > > Regarding to your answers i will try adjust the mapping as best as > possible (i think disabling _source and _all will do a good job) and see > how it impacts on Kibana. > > I will update you in this thread how much Space were saved with this new > Settings. > > >> >> > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/404b474c-69d0-4dc4-aed9-7574c3082fde%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
