Hi Martin, Why SHARDS=4,
while my guess with two nodes there should be 2 configured if I would follow the directions by graylog? On Thursday, June 12, 2014 11:01:27 AM UTC+2, Martin René Mortensen wrote: > > Hi Asad, > > Im running a graylog2 0.20.2 setup with ~5000 msgs/s and peaks around > 10000 msgs/s. It can be tricky to setup, especially if you also want to be > able to search through it all with decent response times. > > I found that increasing the number of elasticsearch nodes helped immensely > with both indexing and search performance, as if elasticsearch just likes > more nodes. > > This is my setup: > > 2 8vcpu elasticsearch 0.90.10 nodes > 1 5vcpu graylog2-server 0.20.2 node with udp syslog input > 1 1vcpu graylog2-web 0.20.2 node > > I use following tunings in /etc/elasticsearch/elasticsearch.conf: > > index.translog.flush_threshold_ops: 50000 > index.refresh_interval: 15s > > #index.cache.field.type: soft > index.cache.field.max_size: 10000 > threadpool.bulk.queue_size: 500 > > > > I use following settings in /etc/graylog2/server.conf: > > elasticsearch_shards = 4 > elasticsearch_replicas = 0 > > elasticsearch_analyzer = standard > output_batch_size = 60000 > processbuffer_processors = 40 > outputbuffer_processors = 60 > processor_wait_strategy = blocking > ring_size = 8192 > > and for /etc/graylog2/web.conf on web node: > > # Higher time-out to avoid failures > timeout.DEFAULT=60s > > > Im not sure how much it can take, but we have peaks at >10.000 msgs/s. I > also have alot of custom drools rules on my graylog2 instance making field > extractions of all the cisco asa and ace logs into , which uses alot of the > CPU on that node. > > Hope this helps pointing you in the right direction. > > /Martin > > On Wednesday, 11 June 2014 10:44:12 UTC+2, Arie wrote: >> >> Hi Asad, >> >> Searching around I found a very fine article about Graylog2 with >> Elasticsearch, maybe there is some info >> in it to help you out. I am trying to build my own Elasticsearch cluster >> here. >> >> http://edgeofsanity.net/article/2012/12/26/elasticsearch-for-logging.html >> >> >> Arie. >> >> >> On Monday, June 9, 2014 2:37:52 AM UTC+2, Asad Mehmood wrote: >>> >>> Good day! >>> >>> Recently I started implementing log monitoring and analysis system using >>> graylog2, we will have around 12,000 message / second. Though in staging we >>> are not even near that number but the cluster is not stable. >>> >>> Sometimes ES discovery fails because either the PC is in I/O wait or >>> there are too many processes in each core. >>> I tried to tune the settings by one way or another the cluster finds a >>> way to fail, as for my setup there are some limitation for a a while to use >>> high speed I/O so I need to either stick with slow disks or divide the >>> setup in a way that recent logs remain in high speed disks and older are >>> moved to low performance cluster. I was hoping if someone can help me >>> formulate or calculate a formula to decide how many nodes I need for ES >>> cluster, graylog2-server, radio and Kafka. >>> >>> There is another problem with KAFKA input if i shutdown Kafka, zookeeper >>> or radio, the messages stop coming and I need to Terminate Kafka input and >>> Launch a new input. >>> Also the message throughput while using KAFKA and Radio is far less than >>> using direct inputs with graylog2-benchmark tool. >>> >>> Current Setup >>> 2 Nodes for Log Collector and Radio (8 Gb, 2 Core Xeon ) >>> 1. Graylog2-server + graylog2-web (16 Gb, 4 Core Xeon ) >>> 1. Graylog2-server + elasticsearch (16 Gb, 4 Core Xeon ) >>> 3. Elasticsearch + Kafka Node (16 Gb, 4 Core Xeon ) >>> >>> The message throughput in peak hours will be 12000 / second and to >>> implement this system in production, the system needs to withstand stress >>> test of 20.000 message / second. >>> >>> I will appreciate if anyone here can help me with formulating the >>> performance requirements by quantifying them. >>> >>> >>> regards, >>> >>> Asad >>> >>> >>> >>> -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
