Sounds interesting. Actually a source like this diminishes the need for disk based buffering channels. And sounds like it does handle log file rotation. Which is awesome.
A few questions: - How does it know when to stop tailing the current file and switch to or start tailing another file - When there is a backlog of many files being built up... how does it order the files for consumption - Sounds like there is some C/C++ native code + JNI to work with inodes ? what api are you using. - does it auto delete the consumed files ? FYI: - For Windows Support: http://www.codeproject.com/Articles/4692/How-to-get-a-notification-if-change-occurs-in-a-sp -roshan On Thu, Oct 9, 2014 at 6:58 PM, Otis Gospodnetic <[email protected] > wrote: > Same here! > The only-on-UNIX part is not great, and I'm wondering if the code could be > structured in a way that allow one (or Flume devs) to add something that > also runs under WInblows, even if it cannot be as clever as the UNIX side. > But even without this, I think what you describe is better than other > tailing implementations because it handles rotations and reads from the > last known checkpoint. > > Btw., because the position in the file is checkpointed periodically, does > that mean that it is possible that, after a restart, some number of lines > that have already been tailed, will be read again? > > Could you also compare your implementation with these two: > 1) https://issues.apache.org/jira/browse/FLUME-2344 (and the comment from > May 2 2014, which points to 2) below) > 2) > > http://search-hadoop.com/m/DEeB4urEUJ&subj=Re+Flume+Jambalaya+A+Flume+Plugin+with+Multiple+Components > > Otis > -- > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > Solr & Elasticsearch Support * http://sematext.com/ > > > On Thu, Oct 9, 2014 at 6:24 AM, Santiago Mola <[email protected]> wrote: > > > 2014-10-09 8:22 GMT+02:00 飯島賢志 <[email protected]>: > > > > > Can we contribute it to the Apache Flume community? > > > If acceptable I will attach the patch of it in JIRA. > > > > > > > That would be definitely useful for us. > > > > Best, > > -- > > > > Santiago M. Mola > > > > > > <http://www.stratio.com/> > > Avenida de Europa, 26. Ática 5. 3ª Planta > > 28224 Pozuelo de Alarcón, Madrid > > Tel: +34 91 352 59 42 // *@stratiobd <https://twitter.com/StratioBD>* > > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
