Eric/Jochen - thanks for the feedback. We have been working on upgrade/moving elasticsearch data to larger nodes, and I feel fairly confident it's healthy.
We will check everything out and get back with findings in the next day or so. Thanks !! On Mon, Aug 29, 2016 at 4:33 PM, Eric Green <[email protected]> wrote: > > > On Friday, August 26, 2016 at 10:54:03 AM UTC-7, Dustin Tennill wrote: >> >> >> Now for the issue(s): >> 1. We see only incoming log message from a single source when searching >> the last five minutes. It is always the same source. This happens even we >> KNOW there are other log data from past five minutes. If we change to the >> past hour, all logs are there and appear correct. If we search past 15 >> minutes, we see all log data. Sometimes we log into the second node and can >> only see messages from this single source. >> 2. Streams - while the counts are there and appear correct, actually >> clicking into a stream and searching doesn't show any messages. Again, if >> you search past the 15 minute mark all messages are visible. >> >> Is this normal? I couldn't find a guide or set of specific instructions >> for what to do on the second node. It all seemed obvious, but I am >> wondering what we missed. >> >> > Virtually every time I've run into weirdness like this, it has been a time > issue. All servers should be running on UTC and sync'ed to a common NTP > time server. Java itself should be configured to UTC with > -Duser.timezone=UTC , I don't know what OS you're running, on Red Hat > derived systems it should be in the GRAYLOG_SERVER_JAVA_OPTS= variable in > the file /etc/sysconfig/graylog-server along with your tweaks to -Xms and > Xmx . Also, make sure that both nodes are pointing at the exact same Mongo > server and exact same set of Elasticsearch nodes and that the Elasticsearch > nodes are similarly configured in UTC with its java opts set to UTC. And > finally, make sure that your *source* servers are set to UTC and is sync'ed > via NTP. If your source servers' time is off, then the time can be off in > the syslog messages that Graylog is receiving. > > You can also check your Elasticsearch cluster's health to see if it has > pending tasks, delayed unassigned shards, etc. that could be holding up > processing. E.g. curl -XGET 'http://localhost:9200/_ > cluster/health?pretty=true' . But I seriously doubt it... > > In general, if you're following the rules adding a second Graylog instance > is pretty easy. The hard part is creating a load balancer to spread the > syslog messages across them, which is why I have syslog-ng in my > infrastructure. But of course if your time is off, then everything's going > to be off. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Graylog Users" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/graylog2/aY-dDoYGOfw/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/graylog2/ac9f306e-2ec3-4829-9e52-ef8e37a75585%40googlegroups.com > <https://groups.google.com/d/msgid/graylog2/ac9f306e-2ec3-4829-9e52-ef8e37a75585%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- ---------------------------------------- Dustin Tennill -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/CABKoSWLR8%2BgrXyfWv6szsOx8iqt6_tXGY0HoXo11FhNgXNY69w%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
