Seems like some filter is discarding messages. Do you have drools rules in place or any extractors running?
On Wed, Mar 12, 2014 at 1:36 PM, Dmitri Stoljarov <[email protected]> wrote: > Hi, > > I'm running GL 0.20.1 (web & server) on one dedicated server (16x2.93GHz, > 32GB RAM) and Elasticsearch (v0.90.10) on second dedicated server > (16x2.93GHz, 32GB RAM, SSD disks). ES is not used by any other applications, > except this GL2 server. > Both servers are in the same network. No firewalls, IDS/IPSs or content > filters between them. > > Graylog2-server configuration is default, except following lines: > processbuffer_processors = 10 > outputbuffer_processors = 10 > ring_size = 2048 > > On GL2 server i've setup Raw/Plaintext TCP local input. > For testing purposes i have plain text file with exactly 10 log lines. > > Now i'm pumping logs (10 lines) to graylog: cat test-10.log |nc localhost > 6667 > > 1st run: > ES (head plugin) shows 6 events > GL2: 6 messages > > 2nd run (~20-50 seconds after previous run): > ES: 7 events > GL2: 7 events > > 3rd run (~20-50 seconds after previous run): > ES: 13 events > GL2: 13 events > > 4th run (~20-50 seconds after previous run): > ES: 17 events > GL2: 17 events > > 5th run (~20-50 seconds after previous run): > ES: 28 events > GL2: 28 events > > 6th run (~20-50 seconds after previous run): > ES: 29 events > GL2: 29 events > > 7th run (~2 minutes after previous run): > ES: 33 events > GL2: 33 events > > I have also run tcpdump (tcpdump -i lo -s0 -axX -w /tmp/gl2.pcap port 6667) > - pcap files always contains data of 10 events. > > GL2-server in debug mode shows 10 messages > "org.graylog2.inputs.raw.RawProcessor - Adding received raw message" (with > correct log data at the end of message) > 10 messages "org.graylog2.filters.StreamMatcherFilter - Routed message" > 10 messages "org.graylog2.buffers.processors.ProcessBufferProcessor - > Finished processing message. Writing to output buffer" > And only 5 messages "org.graylog2.buffers.processors.OutputBufferProcessor - > Writing message batch to [ElasticSearch Output]. Size <1>". Exactly same > number of messages were written to ES. > > > So, my question is - what's happening to some logs? Why aren't they stored? > > regards, > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
