Re: Logstash and Filebeat guaranteed delivery

[Gary Gregory]

Of course RE Volkan, my first sentence must have been poorly written. I
meant to say that I know who Volkan is and that he is great, but, that's
not the point. I was trying to say that a peronal GitHub account, in
general, can't be compared to an official Apache account.

Gary

On Fri, Dec 1, 2023, 12:25 PM Matt Sicker  wrote:

> I may be biased, but I wouldn’t consider Volkan a random GitHub user!
> However, that does raise an interesting point: we could link to third party
> plugins and such to help curate it.
>
> > On Dec 1, 2023, at 5:00 AM, Gary Gregory  wrote:
> >
> > Hi all,
> >
> > (Don't take his the wrong way Volkan ;-)
> > Assuming that I don't know who you are, why would I pick a random github
> > user's custom appender instead of an official Log4j appender? If your
> > appender is "battle-tested", why not move it to Log4j (or Redis?)
> >
> > Gary
> >
> >
> > On Fri, Dec 1, 2023, 4:08 AM Volkan Yazıcı  wrote:
> >
> >> I appreciate your thoughts on this subject. We can eventually convert
> this
> >> into a chapter in the Log4j manual. My goal is to be able to make a
> >> statement as follows:
> >>
> >> *When Log4j is configured with Y, Y, Z settings, it can provide
> guaranteed
> >> delivery against certain types of log sinks such as A, B, C.*
> >>
> >> *A – You need to make sure A has ... feature enabled. Further, it has
> ...
> >> caveat.*
> >> *B – You need to make sure B has ... feature enabled and ...*
> >> *C – ...*
> >>
> >>
> >> That is, a cookbook for users with recipes for guaranteed delivery.
> >>
> >> [I respond to your message below inline.]
> >>
> >> On Thu, Nov 30, 2023 at 9:34 PM Ralph Goers  >
> >> wrote:
> >>
> >>> Notice that neither of the links you have provided use the term
> >>> “guaranteed delivery”. That is because that is not really what they are
> >>> providing. In addition, notice that Logstash says "Input plugins that
> do
> >>> not use a request-response protocol cannot be protected from data
> loss”,
> >>
> >>
> >> But see the rest of that statement
> >> <
> >>
> https://www.elastic.co/guide/en/logstash/current/persistent-queues.html#persistent-queues-limitations
> >>>
> >> : *"Plugins such as beats and http, which do have an acknowledgement
> >> capability, are well protected by this [Logstash persistent] queue."*
> >>
> >>
> >>> and "Data may be lost if an abnormal shutdown occurs before the
> >> checkpoint
> >>> file has been committed”.
> >>
> >>
> >> See the following statement further down in that page: *"To avoid losing
> >> data in the persistent queue, you can set `queue.checkpoint.writes: 1`
> to
> >> force a checkpoint after each event is written."*
> >>
> >> These two make me conclude that, if configured correctly (e.g., using
> >> `http` plugin in combination with `queue.checkpoint.writes: 1`),
> Logstash
> >> can deliver guaranteed delivery. Am I mistaken?
> >>
> >>
> >>> As for using Google Cloud that would default the whole point. If your
> >> data
> >>> center has lost contact with the outside world it won’t be able to get
> to
> >>> Google Cloud.
> >>>
> >>
> >> But that cannot be an argument against using Google Cloud as a log sink
> >> with guaranteed delivery. An in-house Flume server can go down too. Let
> me
> >> know if I miss your point here.
> >>
> >>
> >>> While Redis would work it would require a) an application component
> that
> >>> interacts with Redis such as a Redis Appender (which we don’t have) b)
> a
> >>> Redis deployment c) a Logstash (or some other Redis consumer) to
> forward
> >>> the event. It is a lot simpler to configure Flume than to do all of
> that.
> >>>
> >>
> >> For one, there is a battle-tested Log4j Redis Appender
> >> , which enabled us to
> remove
> >> `log4j-redis` in `main`.
> >>
> >> Indeed, Flume can deliver what Redis+Logstash do. Though my point is,
> not
> >> that Redis has a magical feature set, but there *are* several log sink
> >> stacks one can build using modern stock components and provide
> guaranteed
> >> delivery. I would like to document some of those, if not best-practices,
> >> known-to-work solutions. This way we can enable our users to make a
> >> well-informed decision and pick the best approach that fits into their
> >> existing stack.
> >>
> >> On Thu, Nov 30, 2023 at 9:34 PM Ralph Goers  >
> >> wrote:
> >>
> >>> Volkan,
> >>>
> >>> Notice that neither of the links you have provided use the term
> >>> “guaranteed delivery”. That is because that is not really what they are
> >>> providing. In addition, notice that Logstash says "Input plugins that
> do
> >>> not use a request-response protocol cannot be protected from data
> loss”,
> >>> and "Data may be lost if an abnormal shutdown occurs before the
> >> checkpoint
> >>> file has been committed”. Note that Flume’s FileChannel does not face
> the
> >>> second issue while the first would also be a problem if it is using a
> >>> source that doesn’t support acknowledgements.However, Log4j’s
> >> 

Re: [log4j] Upgrade `2.x` compiler baseline to Java 17

[Volkan Yazıcı]

Done. Will port this to `main` next week.

On Thu, Nov 30, 2023 at 5:03 PM Volkan Yazıcı  wrote:

> Heads up! #2021 
> bumps the `2.x` baseline to Java 17. Everything works locally. If CI agrees
> too, I will merge it tomorrow and start porting to `main`.
>

Re: [Flume] Integration with OpenTelemetry

[Matt Sicker]

> On Nov 30, 2023, at 12:13 PM, Christian Grobmeier  
> wrote:
> 
> Please do. I would also like to learn more about this stuff and maybe we can 
> make Chainsaw to receive Flume messages or OTel things too!

I think that would make sense. When I do a deeper review of OpenTelemetry, I’ll 
suggest what components here can integrate with what.

Re: Logstash and Filebeat guaranteed delivery

[Matt Sicker]

I may be biased, but I wouldn’t consider Volkan a random GitHub user! However, 
that does raise an interesting point: we could link to third party plugins and 
such to help curate it.

> On Dec 1, 2023, at 5:00 AM, Gary Gregory  wrote:
> 
> Hi all,
> 
> (Don't take his the wrong way Volkan ;-)
> Assuming that I don't know who you are, why would I pick a random github
> user's custom appender instead of an official Log4j appender? If your
> appender is "battle-tested", why not move it to Log4j (or Redis?)
> 
> Gary
> 
> 
> On Fri, Dec 1, 2023, 4:08 AM Volkan Yazıcı  wrote:
> 
>> I appreciate your thoughts on this subject. We can eventually convert this
>> into a chapter in the Log4j manual. My goal is to be able to make a
>> statement as follows:
>> 
>> *When Log4j is configured with Y, Y, Z settings, it can provide guaranteed
>> delivery against certain types of log sinks such as A, B, C.*
>> 
>> *A – You need to make sure A has ... feature enabled. Further, it has ...
>> caveat.*
>> *B – You need to make sure B has ... feature enabled and ...*
>> *C – ...*
>> 
>> 
>> That is, a cookbook for users with recipes for guaranteed delivery.
>> 
>> [I respond to your message below inline.]
>> 
>> On Thu, Nov 30, 2023 at 9:34 PM Ralph Goers 
>> wrote:
>> 
>>> Notice that neither of the links you have provided use the term
>>> “guaranteed delivery”. That is because that is not really what they are
>>> providing. In addition, notice that Logstash says "Input plugins that do
>>> not use a request-response protocol cannot be protected from data loss”,
>> 
>> 
>> But see the rest of that statement
>> <
>> https://www.elastic.co/guide/en/logstash/current/persistent-queues.html#persistent-queues-limitations
>>> 
>> : *"Plugins such as beats and http, which do have an acknowledgement
>> capability, are well protected by this [Logstash persistent] queue."*
>> 
>> 
>>> and "Data may be lost if an abnormal shutdown occurs before the
>> checkpoint
>>> file has been committed”.
>> 
>> 
>> See the following statement further down in that page: *"To avoid losing
>> data in the persistent queue, you can set `queue.checkpoint.writes: 1` to
>> force a checkpoint after each event is written."*
>> 
>> These two make me conclude that, if configured correctly (e.g., using
>> `http` plugin in combination with `queue.checkpoint.writes: 1`), Logstash
>> can deliver guaranteed delivery. Am I mistaken?
>> 
>> 
>>> As for using Google Cloud that would default the whole point. If your
>> data
>>> center has lost contact with the outside world it won’t be able to get to
>>> Google Cloud.
>>> 
>> 
>> But that cannot be an argument against using Google Cloud as a log sink
>> with guaranteed delivery. An in-house Flume server can go down too. Let me
>> know if I miss your point here.
>> 
>> 
>>> While Redis would work it would require a) an application component that
>>> interacts with Redis such as a Redis Appender (which we don’t have) b) a
>>> Redis deployment c) a Logstash (or some other Redis consumer) to forward
>>> the event. It is a lot simpler to configure Flume than to do all of that.
>>> 
>> 
>> For one, there is a battle-tested Log4j Redis Appender
>> , which enabled us to remove
>> `log4j-redis` in `main`.
>> 
>> Indeed, Flume can deliver what Redis+Logstash do. Though my point is, not
>> that Redis has a magical feature set, but there *are* several log sink
>> stacks one can build using modern stock components and provide guaranteed
>> delivery. I would like to document some of those, if not best-practices,
>> known-to-work solutions. This way we can enable our users to make a
>> well-informed decision and pick the best approach that fits into their
>> existing stack.
>> 
>> On Thu, Nov 30, 2023 at 9:34 PM Ralph Goers 
>> wrote:
>> 
>>> Volkan,
>>> 
>>> Notice that neither of the links you have provided use the term
>>> “guaranteed delivery”. That is because that is not really what they are
>>> providing. In addition, notice that Logstash says "Input plugins that do
>>> not use a request-response protocol cannot be protected from data loss”,
>>> and "Data may be lost if an abnormal shutdown occurs before the
>> checkpoint
>>> file has been committed”. Note that Flume’s FileChannel does not face the
>>> second issue while the first would also be a problem if it is using a
>>> source that doesn’t support acknowledgements.However, Log4j’s
>> FlumeAppender
>>> always gets acks.
>>> 
>>> To make this clearer let me review the architecture for my implementation
>>> again.
>>> 
>>> First the phone system maintains a list of ip addresses that can handle
>>> Radius accounting records. We host 2 Flume servers in the same data
>> center
>>> as the phone system and configure the phone system with their ip
>> addresses.
>>> The Radius records will be sent to those Flume servers which will accept
>>> them with our custom Radius Source. That converts them to JSON and passes
>>> the JSON to the File 

Re: Logstash and Filebeat guaranteed delivery

[Gary Gregory]

Hi all,

(Don't take his the wrong way Volkan ;-)
Assuming that I don't know who you are, why would I pick a random github
user's custom appender instead of an official Log4j appender? If your
appender is "battle-tested", why not move it to Log4j (or Redis?)

Gary


On Fri, Dec 1, 2023, 4:08 AM Volkan Yazıcı  wrote:

> I appreciate your thoughts on this subject. We can eventually convert this
> into a chapter in the Log4j manual. My goal is to be able to make a
> statement as follows:
>
> *When Log4j is configured with Y, Y, Z settings, it can provide guaranteed
> delivery against certain types of log sinks such as A, B, C.*
>
> *A – You need to make sure A has ... feature enabled. Further, it has ...
> caveat.*
> *B – You need to make sure B has ... feature enabled and ...*
> *C – ...*
>
>
> That is, a cookbook for users with recipes for guaranteed delivery.
>
> [I respond to your message below inline.]
>
> On Thu, Nov 30, 2023 at 9:34 PM Ralph Goers 
> wrote:
>
> > Notice that neither of the links you have provided use the term
> > “guaranteed delivery”. That is because that is not really what they are
> > providing. In addition, notice that Logstash says "Input plugins that do
> > not use a request-response protocol cannot be protected from data loss”,
>
>
> But see the rest of that statement
> <
> https://www.elastic.co/guide/en/logstash/current/persistent-queues.html#persistent-queues-limitations
> >
> : *"Plugins such as beats and http, which do have an acknowledgement
> capability, are well protected by this [Logstash persistent] queue."*
>
>
> > and "Data may be lost if an abnormal shutdown occurs before the
> checkpoint
> > file has been committed”.
>
>
> See the following statement further down in that page: *"To avoid losing
> data in the persistent queue, you can set `queue.checkpoint.writes: 1` to
> force a checkpoint after each event is written."*
>
> These two make me conclude that, if configured correctly (e.g., using
> `http` plugin in combination with `queue.checkpoint.writes: 1`), Logstash
> can deliver guaranteed delivery. Am I mistaken?
>
>
> > As for using Google Cloud that would default the whole point. If your
> data
> > center has lost contact with the outside world it won’t be able to get to
> > Google Cloud.
> >
>
> But that cannot be an argument against using Google Cloud as a log sink
> with guaranteed delivery. An in-house Flume server can go down too. Let me
> know if I miss your point here.
>
>
> > While Redis would work it would require a) an application component that
> > interacts with Redis such as a Redis Appender (which we don’t have) b) a
> > Redis deployment c) a Logstash (or some other Redis consumer) to forward
> > the event. It is a lot simpler to configure Flume than to do all of that.
> >
>
> For one, there is a battle-tested Log4j Redis Appender
> , which enabled us to remove
> `log4j-redis` in `main`.
>
> Indeed, Flume can deliver what Redis+Logstash do. Though my point is, not
> that Redis has a magical feature set, but there *are* several log sink
> stacks one can build using modern stock components and provide guaranteed
> delivery. I would like to document some of those, if not best-practices,
> known-to-work solutions. This way we can enable our users to make a
> well-informed decision and pick the best approach that fits into their
> existing stack.
>
> On Thu, Nov 30, 2023 at 9:34 PM Ralph Goers 
> wrote:
>
> > Volkan,
> >
> > Notice that neither of the links you have provided use the term
> > “guaranteed delivery”. That is because that is not really what they are
> > providing. In addition, notice that Logstash says "Input plugins that do
> > not use a request-response protocol cannot be protected from data loss”,
> > and "Data may be lost if an abnormal shutdown occurs before the
> checkpoint
> > file has been committed”. Note that Flume’s FileChannel does not face the
> > second issue while the first would also be a problem if it is using a
> > source that doesn’t support acknowledgements.However, Log4j’s
> FlumeAppender
> > always gets acks.
> >
> > To make this clearer let me review the architecture for my implementation
> > again.
> >
> > First the phone system maintains a list of ip addresses that can handle
> > Radius accounting records. We host 2 Flume servers in the same data
> center
> > as the phone system and configure the phone system with their ip
> addresses.
> > The Radius records will be sent to those Flume servers which will accept
> > them with our custom Radius Source. That converts them to JSON and passes
> > the JSON to the File Channel. Once the File Channel has written them to
> > disk the source responds back to the phone system with an ACK that the
> > record was received. It the record is not processed quickly enough (I
> > believe it is 100ms) then the phone system will try a different ip
> address
> > assuming it couldn’t be delivered by the first server.  Another thread
> > 

Re: Logstash and Filebeat guaranteed delivery

[Volkan Yazıcı]

I appreciate your thoughts on this subject. We can eventually convert this
into a chapter in the Log4j manual. My goal is to be able to make a
statement as follows:

*When Log4j is configured with Y, Y, Z settings, it can provide guaranteed
delivery against certain types of log sinks such as A, B, C.*

*A – You need to make sure A has ... feature enabled. Further, it has ...
caveat.*
*B – You need to make sure B has ... feature enabled and ...*
*C – ...*


That is, a cookbook for users with recipes for guaranteed delivery.

[I respond to your message below inline.]

On Thu, Nov 30, 2023 at 9:34 PM Ralph Goers 
wrote:

> Notice that neither of the links you have provided use the term
> “guaranteed delivery”. That is because that is not really what they are
> providing. In addition, notice that Logstash says "Input plugins that do
> not use a request-response protocol cannot be protected from data loss”,


But see the rest of that statement

: *"Plugins such as beats and http, which do have an acknowledgement
capability, are well protected by this [Logstash persistent] queue."*


> and "Data may be lost if an abnormal shutdown occurs before the checkpoint
> file has been committed”.


See the following statement further down in that page: *"To avoid losing
data in the persistent queue, you can set `queue.checkpoint.writes: 1` to
force a checkpoint after each event is written."*

These two make me conclude that, if configured correctly (e.g., using
`http` plugin in combination with `queue.checkpoint.writes: 1`), Logstash
can deliver guaranteed delivery. Am I mistaken?


> As for using Google Cloud that would default the whole point. If your data
> center has lost contact with the outside world it won’t be able to get to
> Google Cloud.
>

But that cannot be an argument against using Google Cloud as a log sink
with guaranteed delivery. An in-house Flume server can go down too. Let me
know if I miss your point here.


> While Redis would work it would require a) an application component that
> interacts with Redis such as a Redis Appender (which we don’t have) b) a
> Redis deployment c) a Logstash (or some other Redis consumer) to forward
> the event. It is a lot simpler to configure Flume than to do all of that.
>

For one, there is a battle-tested Log4j Redis Appender
, which enabled us to remove
`log4j-redis` in `main`.

Indeed, Flume can deliver what Redis+Logstash do. Though my point is, not
that Redis has a magical feature set, but there *are* several log sink
stacks one can build using modern stock components and provide guaranteed
delivery. I would like to document some of those, if not best-practices,
known-to-work solutions. This way we can enable our users to make a
well-informed decision and pick the best approach that fits into their
existing stack.

On Thu, Nov 30, 2023 at 9:34 PM Ralph Goers 
wrote:

> Volkan,
>
> Notice that neither of the links you have provided use the term
> “guaranteed delivery”. That is because that is not really what they are
> providing. In addition, notice that Logstash says "Input plugins that do
> not use a request-response protocol cannot be protected from data loss”,
> and "Data may be lost if an abnormal shutdown occurs before the checkpoint
> file has been committed”. Note that Flume’s FileChannel does not face the
> second issue while the first would also be a problem if it is using a
> source that doesn’t support acknowledgements.However, Log4j’s FlumeAppender
> always gets acks.
>
> To make this clearer let me review the architecture for my implementation
> again.
>
> First the phone system maintains a list of ip addresses that can handle
> Radius accounting records. We host 2 Flume servers in the same data center
> as the phone system and configure the phone system with their ip addresses.
> The Radius records will be sent to those Flume servers which will accept
> them with our custom Radius Source. That converts them to JSON and passes
> the JSON to the File Channel. Once the File Channel has written them to
> disk the source responds back to the phone system with an ACK that the
> record was received. It the record is not processed quickly enough (I
> believe it is 100ms) then the phone system will try a different ip address
> assuming it couldn’t be delivered by the first server.  Another thread
> reads the records from the File Channel and sends them to a Flume in a
> different data center for processing. This follows the same pattern. The
> Avro Sink serializes the record and sends it to the target Flume. That
> Flume writes the record to a File channel and the Avro Source responds with
> an ACK that the record was received, at which point the originating Flume
> will remove it from the File Channel.
>
> If you will notice, the application itself knows that delivery is
> guaranteed because it gets an ACK to