Hey Dmitri, thanks for your detailled analysis! The TCP input logging as UDP input was a mistake on our side but just on the logger configuration. (fixed in f6a978eb151c88f6241aabf4c7bd93c76be70bac) Thanks for spotting this!
The extreme amount of logging for users and auth is fine in TRACE mode. The missing messages are still strange. Did you try listening on 127.0.0.1 already? This avoids big parts of the network stack. Thanks, Lennart On Mon, May 5, 2014 at 9:14 PM, Dmitri Stoljarov <[email protected]> wrote: > Hi, > > Just downloaded and installed graylog2 v 0.20.2-rc1. > Configuration of graylog2 is default, except mongoDB and some ES settings > (my configuration file: http://dimka.ee/foo/graylog2.conf.txt) > In Graylog2 i do not have any streams, drools or field extractors. Only > thing is set - is RawTCP Gelf input on port 4452. > > Elasticsearch's version is 0.90.10. Here's my Elasticsearch configuration > file: http://dimka.ee/foo/elasticsearch.yml.txt > > Graylog server was started with "--debug" command line key. After gl2 server > and webserver were up and running i enabled all four dropdowns in logging > page to "TRACE" level. > > Next step was to send some logs (just 5 lines) to graylog2 server with > following command "cat 5.log | nc 172.28.28.27 4452" (here's my 5.log file: > http://dimka.ee/foo/5.log.txt). > > Result was: from 5 messages only 3 messages were in Elasticsearch. Here's > full debug-trace file of graylog server > (http://dimka.ee/foo/graylog2-server.log.3.messages.gz) > > After that, i stopped all graylog2 server and webserver instances, cleared > elasticsearch database and repeated again. > Result was: from 5 messages only 4 messages were in Elasticsearch. Here's > graylog2 server log file from my second try > (http://dimka.ee/foo/graylog2-server.log.4.messages.gz) > > Server, where i run these test is totally idle. All software (ES, mongo, > GL2) is on the same server. But i don't think that it's an issue for just 5 > lines of logs. > > Here's some my observations of gathered logs: > > 1. I created RawTCP input, but logs are showing UDP: > 2014-05-05 19:18:32,219 INFO : org.graylog2.inputs.raw.udp.RawUDPInput - > Started raw TCP input on /172.28.28.27:4452 > > 2. Both times all 5 messages are getting to graylog2 server with unique IDs. > All messages are sent to [StreamMatcher], [Static field appender], > [Blacklister], [Rewriter] processors. OutputBufferProcessor processes also > all messages. But "Message ids in batch of …." happens only 3 or 4 times in > my case, not 5. > > 3. security.realm.MongoDbAuthorizationRealm - almost 700 events per second > is written to log file. Isn't it too much? > > 4. org.graylog2.users.User - also almost 700 events per second fired. > > > Any ideas or additional steps i can do to help solve this issue? > > best regards, > > > > > -- > You received this message because you are subscribed to the Google Groups > "graylog2" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
