On Friday, August 26, 2016 at 10:54:03 AM UTC-7, Dustin Tennill wrote: > > > Now for the issue(s): > 1. We see only incoming log message from a single source when searching > the last five minutes. It is always the same source. This happens even we > KNOW there are other log data from past five minutes. If we change to the > past hour, all logs are there and appear correct. If we search past 15 > minutes, we see all log data. Sometimes we log into the second node and can > only see messages from this single source. > 2. Streams - while the counts are there and appear correct, actually > clicking into a stream and searching doesn't show any messages. Again, if > you search past the 15 minute mark all messages are visible. > > Is this normal? I couldn't find a guide or set of specific instructions > for what to do on the second node. It all seemed obvious, but I am > wondering what we missed. > > Virtually every time I've run into weirdness like this, it has been a time issue. All servers should be running on UTC and sync'ed to a common NTP time server. Java itself should be configured to UTC with -Duser.timezone=UTC , I don't know what OS you're running, on Red Hat derived systems it should be in the GRAYLOG_SERVER_JAVA_OPTS= variable in the file /etc/sysconfig/graylog-server along with your tweaks to -Xms and Xmx . Also, make sure that both nodes are pointing at the exact same Mongo server and exact same set of Elasticsearch nodes and that the Elasticsearch nodes are similarly configured in UTC with its java opts set to UTC. And finally, make sure that your *source* servers are set to UTC and is sync'ed via NTP. If your source servers' time is off, then the time can be off in the syslog messages that Graylog is receiving.
You can also check your Elasticsearch cluster's health to see if it has pending tasks, delayed unassigned shards, etc. that could be holding up processing. E.g. curl -XGET 'http://localhost:9200/_cluster/health?pretty=true' . But I seriously doubt it... In general, if you're following the rules adding a second Graylog instance is pretty easy. The hard part is creating a load balancer to spread the syslog messages across them, which is why I have syslog-ng in my infrastructure. But of course if your time is off, then everything's going to be off. -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/ac9f306e-2ec3-4829-9e52-ef8e37a75585%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
