Hi Marcel, Question: is this normal that a negative amount of unprocessed messages can > be shown or is this a bug? >
That's a (display) bug as there is really no possibility to have a negative number of unprocessed messages in the journal without breaking the laws of space and time. ;-) > Is this the expected behaviour? I so far understood that all messages > shall be written directly to the journal and at 'message_journal_flush_age' > intervals written to Elasticsearch. How could we lose messages here? > The message_journal_flush_age setting specifies how long to wait (at most) until the Kafka log is flushed to disk. The message_journal_flush_interval setting specifies how many messages should be buffered (at most) until the Kafka log is flushed to disk. The equivalent settings in the Kafka documentation (https://kafka.apache.org/08/configuration.html) are log.flush.interval.messages and log.flush.interval.ms. Flushing the Kafka log too often imposes a certain performance impact so it is a tradeoff between performance and how many messages you'll lose in case of an outage of the processing node (not Elasticsearch). The settings for indexing batches of messages into Elasticsearch can be modified in the Graylog configuration with the output_batch_size and output_flush_interval settings ( https://github.com/Graylog2/graylog2-server/blob/1.0.1/misc/graylog2.conf#L193-203 ). > In order to verify that messages were lost on that node, can you tell me > what's the syntax for searching messages that were received over a certain > node? I only found the possibility to search by input - eg. > "gl_source_input:<input-here>", however since the input is a global input > it will show messages that were received over either node A or B for this > input. I would like to query only messages received from this input over > node A for example. How to do this? > You can use the (internal) message attributes gl2_source_node and gl2_source_input for this. An example query might look like "gl2_source_node:3c1749a2-b7ae-4e23-b761-f0b666e50e36 gl2_source_input:52f51313e4b0fa055889cc5a". You can also show messages received via a specific input in the web interface at System -> Inputs and select "Messages from this input" in the "Action" menu next to a specific input. Cheers, Jochen -- You received this message because you are subscribed to the Google Groups "graylog2" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
