[graylog2] Re: Disk Journal / Kafka Input / Throttling
Hi Jochen, I think what Eli is driving at is that throttling starts too late. I think there should be a (configurable) threshold that starts throttling when the internal journal exceeds a certail level. Cheers, Carlos -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/c21073fd-e9be-40a7-8bfb-51bf6de47533%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Disk Journal / Kafka Input / Throttling
Hi Eli, Graylog should already throttle message consumption from an external Kafka broker if processing cannot keep up and the disk journal and the processing buffer are running full. Cheers, Jochen On Wednesday, 20 July 2016 04:10:36 UTC+2, Eli Jordan wrote: > > Thanks for the clarification Jochen. > > Do you know if its possible to throttle the kafka input, so that messages > are buffered in kafka rather than in GrayLogs internal journal? Enabling > throttling on the input didn't seem to slow down the rate at which messages > are consumed. (note: we are running 2.0.3) > > On Tuesday, 19 July 2016 22:04:59 UTC+10, Jochen Schalanda wrote: >> >> Hi Eli, >> >> On Tuesday, 19 July 2016 13:18:49 UTC+2, Eli Jordan wrote: >>> >>> My understanding is that the disk journal is just an internal Kafka >>> topic. Since we are already using Kafka to buffer messages, this seems >>> redundant. (Also, since we are running graylog in docker the journal is >>> transient without configuring appropriate docker volumes). >>> >> >> That's not quite correct. Graylog is using the journal implementation >> from Apache Kafka internally but it's not a full-fledged Kafka broker, e. >> g. the whole Kafka networking stack is missing (it's there for the Kafka >> client in the Kafka inputs, of course). >> >> Cheers, >> Jochen >> > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/be6219d1-27e2-4b10-847f-4b11054babab%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Disk Journal / Kafka Input / Throttling
Thanks for the clarification Jochen. Do you know if its possible to throttle the kafka input, so that messages are buffered in kafka rather than in GrayLogs internal journal? Enabling throttling on the input didn't seem to slow down the rate at which messages are consumed. (note: we are running 2.0.3) On Tuesday, 19 July 2016 22:04:59 UTC+10, Jochen Schalanda wrote: > > Hi Eli, > > On Tuesday, 19 July 2016 13:18:49 UTC+2, Eli Jordan wrote: >> >> My understanding is that the disk journal is just an internal Kafka >> topic. Since we are already using Kafka to buffer messages, this seems >> redundant. (Also, since we are running graylog in docker the journal is >> transient without configuring appropriate docker volumes). >> > > That's not quite correct. Graylog is using the journal implementation from > Apache Kafka internally but it's not a full-fledged Kafka broker, e. g. the > whole Kafka networking stack is missing (it's there for the Kafka client in > the Kafka inputs, of course). > > Cheers, > Jochen > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/15bd2efe-2d3a-48e4-930f-e3055e07a02d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[graylog2] Re: Disk Journal / Kafka Input / Throttling
Hi Eli, On Tuesday, 19 July 2016 13:18:49 UTC+2, Eli Jordan wrote: > > My understanding is that the disk journal is just an internal Kafka topic. > Since we are already using Kafka to buffer messages, this seems redundant. > (Also, since we are running graylog in docker the journal is transient > without configuring appropriate docker volumes). > That's not quite correct. Graylog is using the journal implementation from Apache Kafka internally but it's not a full-fledged Kafka broker, e. g. the whole Kafka networking stack is missing (it's there for the Kafka client in the Kafka inputs, of course). Cheers, Jochen -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to graylog2+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/5561f4cc-2290-484f-b32e-c63a7d572957%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
