Volkan, Notice that neither of the links you have provided use the term “guaranteed delivery”. That is because that is not really what they are providing. In addition, notice that Logstash says "Input plugins that do not use a request-response protocol cannot be protected from data loss”, and "Data may be lost if an abnormal shutdown occurs before the checkpoint file has been committed”. Note that Flume’s FileChannel does not face the second issue while the first would also be a problem if it is using a source that doesn’t support acknowledgements.However, Log4j’s FlumeAppender always gets acks.
To make this clearer let me review the architecture for my implementation again. First the phone system maintains a list of ip addresses that can handle Radius accounting records. We host 2 Flume servers in the same data center as the phone system and configure the phone system with their ip addresses. The Radius records will be sent to those Flume servers which will accept them with our custom Radius Source. That converts them to JSON and passes the JSON to the File Channel. Once the File Channel has written them to disk the source responds back to the phone system with an ACK that the record was received. It the record is not processed quickly enough (I believe it is 100ms) then the phone system will try a different ip address assuming it couldn’t be delivered by the first server. Another thread reads the records from the File Channel and sends them to a Flume in a different data center for processing. This follows the same pattern. The Avro Sink serializes the record and sends it to the target Flume. That Flume writes the record to a File channel and the Avro Source responds with an ACK that the record was received, at which point the originating Flume will remove it from the File Channel. If you will notice, the application itself knows that delivery is guaranteed because it gets an ACK to say so. Due to this, Filbeat cannot possibly implement guaranteed delivery. The application will expect that once it writes to a file or to System.out delivery is guaranteed, which really cannot be true. As for using Google Cloud that would default the whole point. If your data center has lost contact with the outside world it won’t be able to get to Google Cloud. While Redis would work it would require a) an application component that interacts with Redis such as a Redis Appender (which we don’t have) b) a Redis deployment c) a Logstash (or some other Redis consumer) to forward the event. It is a lot simpler to configure Flume than to do all of that. Ralph > On Nov 30, 2023, at 4:32 AM, Volkan Yazıcı <vol...@yazi.ci> wrote: > > Ralph, could you elaborate on your response, please? AFAIK, Logstash and > Filebeat provide guaranteed delivery, if configured correctly. As a matter of > fact they have docs (here and here) explaining how to do it – actually, there > are several ways on how to do it. What makes you think they don't provide > guaranteed delivery? > > I have implemented two different types of logging pipelines with guaranteed > delivery: > • > Using a Google Cloud BigQuery appender > • Using a Redis appender (Redis queue is ingested to Elasticsearch > through Logstash) > I want to learn where I can potentially violate the delivery guarantee. > > On Thu, Nov 30, 2023 at 5:54 AM Ralph Goers <ralph.go...@dslextreme.com> > wrote: > Fluentbit, Fluentd, Logstash, and Filebeat are the main tools used for log > forwarding. While they all have some amount of plugability none of the are as > flexible as Flume. In addition, as I have mentioned before, none of them > provide guaranteed delivery so I would never recommend them for forwarding > audit logs.
