Hi We are using the Kafka input to pull logs into graylog, and when running load tests against our system, the graylog journal begins to fill up, since the elastic search indexing can't keep up.
My understanding is that the disk journal is just an internal Kafka topic. Since we are already using Kafka to buffer messages, this seems redundant. (Also, since we are running graylog in docker the journal is transient without configuring appropriate docker volumes). We have plenty of storage in the Kafka cluster, so there should be no problem buffering messages until graylog can catch up. I was hoping that enabling throttling on the Kafka input would do this. I.e. Only read messages in as fast as they can be processed by elastic search. However, that doesn't seem to be the case. Even with throttling enabled, the disk journal still begins to fill up. How should I go about achieving this behavior? Would disabling the disk journal help? Why isn't throttling slowing down the rate at which messages are being read in? -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/b61c1f16-441d-4e89-8568-a5002b884792%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
