Hi there We operate a 2 node GL/ES setup on GCE and witnessed these days an underlying host crash, for which the VM restarted. As we were just present when this happened, we checked how GL scopes with this situation when the crashed node comes back up in the live environment, joining the cluster.
When the node joined again, we noticed in "System / Nodes" screen that a negative number was indicated as unprocessed message in the journal for this node (other node was shown normally). Eg. it said 'There are *-*xxx unprocessed messages in the journal.' The negative number decreased until 0 up to finally a small positive number after which the message next to the 'Action' button started showing 'Processing [positive] messages per second' again (before was 0 indicated here). Question: is this normal that a negative amount of unprocessed messages can be shown or is this a bug? Further: even though the loadbalancer immediately directed traffic to the other node in the cluster, we noticed that we're missing some messages from the minute of the crash. We currently had not modified the default setting of "message_journal_flush_age = 1m" in the server.conf, for which probably we lost the messages in the moment of the crash before the journal was flushed. Is this the expected behaviour? I so far understood that all messages shall be written directly to the journal and at 'message_journal_flush_age' intervals written to Elasticsearch. How could we lose messages here? In order to verify that messages were lost on that node, can you tell me what's the syntax for searching messages that were received over a certain node? I only found the possibility to search by input - eg. "gl_source_input:<input-here>", however since the input is a global input it will show messages that were received over either node A or B for this input. I would like to query only messages received from this input over node A for example. How to do this? Thanks for your help and keep up the great work, GL is a fantastic product! Cheers Marcel -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
