If the checkpointing interval is 30 seconds (by default), and dualCheckpoints are enabled (in case, the agent was interrupted while writing a checkpoint), then replay should happen only from the last 30 secs (worst case 60 secs). Not sure if this is happening in your case, or a Full replay is happening.
Thanks, Rufus On Wed, Jun 24, 2015 at 10:40 PM, Shady Xu <[email protected]> wrote: > I have tried 1.6, replaying log has been faster, but not enough. We have G > bytes of logs, replaying these logs still takes us hours even days. This is > frustrating, and has been the biggest concern for us to use it in a larger > scale. > > 2015-06-01 15:32 GMT+08:00 Hari Shreedharan <[email protected]>: > >> 1.6 has been released. We were waiting for maven central to sync up. Now >> that it is on central, I will post the update on the site tomorrow. >> >> >> On Sunday, May 31, 2015, Shady Xu <[email protected]> wrote: >> >>> I noticed that Flume 1.6 has been released on Github but not the >>> official website. I have compiled some of the modules from source myself >>> (for other reasons), but I'm not sure compiling the whole project is a >>> good idea. >>> >>> We have tons of data, every time we change the configurations, replaying >>> log takes us way too many hours... >>> >>> 2015-04-17 12:38 GMT+08:00 Hari Shreedharan <[email protected]>: >>> >>>> Changes that went into Flume 1.6 should improve replay time. Flume 1.6 >>>> will be out in a few days. >>>> >>>> >>>> Thanks, >>>> Hari >>>> >>>> On Thu, Apr 16, 2015 at 7:55 PM, Shady Xu <[email protected]> wrote: >>>> >>>>> Every time I restart Flume NG, it will try to replay the log and the >>>>> process usually takes hours. During this time, Flume does not take any >>>>> data >>>>> from the source. >>>>> >>>>> So how can I make the replay faster? >>>>> >>>> >>>> >>> >> >> -- >> >> Thanks, >> Hari >> >> >
