Hello all. I have an interesting case where we lose data in the event of a flume crash, which is easily reproducible when we kill -9 the flume agent.
I believe that this may be because the Flume Sink is issuing a commit before it actually completes the fs sync. If this is the case then the last few commits just before the crash would have removed events from the queue even though those events will needed to perform a recovery. My question is, are those events still possibly in the WAL? If so, is it possible so somehow roll back the queue to a point in time before the commits were processed, and restart from that state? How would I accomplish this? Nothing in this message is intended to constitute an electronic signature unless a specific statement to the contrary is included in this message. Confidentiality Note: This message is intended only for the person or entity to which it is addressed. It may contain confidential and/or privileged material. Any review, transmission, dissemination or other use, or taking of any action in reliance upon this message by persons or entities other than the intended recipient is prohibited and may be unlawful. If you received this message in error, please contact the sender and delete it from your computer.
