Hi, These are nice and small and require no analysis. Turn of _all, tweak merge rate, use high refresh interval, give ES/Lucene a good buffer, look at xa log flush settings, etc. and you should be able to get to 150K/sec without requiring dozens of servers.
Otis -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Monday, March 17, 2014 11:10:16 PM UTC-4, [email protected] wrote: > > We have a bunch of different things going into elasticsearch, mostly > network-related telemetry. The latest painful one is IPFIX (netflow). The > aggregation we're working on looks like this: > > { > "_index": "ipfix-2014.03.17", > "_type": "logs", > "_id": "oA5uhOc9QpCaH9iGPlY7AQ", > "_score": null, > "_source": { > "peer_ip_src": "207.46.32.122", > "ip_dst": "157.56.106.160", > "as_src": 12670, > "mask_dst": 27, > "as_path": "", > "ip_src": "92.102.0.0", > "bytes": 80, > "port_dst": 3544, > "mask_src": 16, > "stamp_inserted": "2014-03-17 09:18:00", > "stamp_updated": "2014-03-17 10:31:28", > "packets": 1, > "@version": "1", > "@timestamp": "2014-03-17T12:19:27.244Z", > "source": "ipfix" > }, > "sort": [ > 1395058767244 > ] > } > > > There's just a really stupid high volume of really small messages that > need little in the way of analysis (which we need to turn off). > > > > On Monday, March 17, 2014 2:45:04 PM UTC-7, Mo wrote: > >> It would be helpful to see the document size along with the parameters. >> On Mon, Mar 17, 2014 at 2:27 PM, <[email protected]> wrote: >> >>> I got 25K per second using 3 nodes 32G EC2 system. >>> >>> I am also interested in something like 150k/sec of indexing speed, and >>> need help. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/ba9799e0-82e8-4337-bd11-e3fbdd9a5f4f%40googlegroups.com<https://groups.google.com/d/msgid/elasticsearch/ba9799e0-82e8-4337-bd11-e3fbdd9a5f4f%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/aaa05eca-ff6a-4b8b-a8ba-9ded45d0c939%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
