> On June 21, 2012, 3:14 a.m., Will McQueen wrote:
> > +1 lgtm!
> >
> > I tested with this config:
> >
> > {code}
> > agent.channels = c1
> > agent.sources = r1
> > agent.sinks = k1
> > #
> > agent.channels.c1.type = FILE
> > agent.channels.c1.checkpointDir = /tmp/flume-check
> > agent.channels.c1.dataDirs = /tmp/flume-data
> > agent.channels.c1.checkpointInterval=30000
> > #
> > agent.sources.r1.channels = c1
> > agent.sources.r1.type = SEQ
> > agent.sources.r1.interceptors = i1
> > agent.sources.r1.interceptors.i1.type =
> > org.apache.flume.interceptor.TimestampInterceptor$Builder
> > #
> > agent.sinks.k1.channel = c1
> > agent.sinks.k1.type = LOGGER
> > {code}
> >
> > The patch looks solid according to my tests. I tried starting and
> > restarting the agent multiple times. The replay happens very quickly, and
> > I'm able to stop the client by doing ctrl-C on the agent's terminal window.
> >
> > There's an issue not directly related to filechannel that I observed when
> > starting the agent, then stopping it within about 3 secs, and then starting
> > it back up. This is a separate issue being addressed in FLUME-1296.
> >
> > Cheers,
> > Will
For you entertainment: https://reviews.apache.org/r/5467/ ;)
- Hari
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/5436/#review8442
-----------------------------------------------------------
On June 21, 2012, 1:29 a.m., Arvind Prabhakar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/5436/
> -----------------------------------------------------------
>
> (Updated June 21, 2012, 1:29 a.m.)
>
>
> Review request for Flume and Brock Noland.
>
>
> Description
> -------
>
> The file channel uses a disk-serialized in-memory checkpointing mechanism.
> When the channel is full and the capacity is large, these checkpoints take a
> long time to serialize and deserialize. For example, a channel with 1M
> entries could take many minutes to boot up. Similarly, a boot up of a largely
> full channel would require the replay of all log events to reconstruct the
> correct state. Due to this latency issues and the failure interaction of the
> channel with the LifeCycleSupervisor, the system could get into an unusable
> state easily as evident from the FLUME-1232 issue.
>
> This patch modifies the checkpointing mechanism as follows:
> * The FlumeEventQueue itself represents a checkpoint that is maintained as a
> memory mapped file.
> * During checkpointing, a marker is introduced in active logs which is used
> to skip records during display.
>
> In order to ensure correctness, a reader/writer lock is used where the reader
> lock is used by consumers operating against the channel while the writer lock
> is used to facilitate checkpointing. Some limitations of this approach are:
>
> * The total number of active log files is now limited to a maximum of 1024.
> * Dynamic resizing of the channel capacity is no longer allowed unless the
> checkpoint is rebuild from scratch which can cause significant delay in
> startup.
>
>
> This addresses bug FLUME-1232.
> https://issues.apache.org/jira/browse/FLUME-1232
>
>
> Diffs
> -----
>
>
> /trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Checkpoint.java
> 1351988
>
> /trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java
> 1351988
>
> /trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannelConfiguration.java
> 1351988
>
> /trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FlumeEventQueue.java
> 1351988
>
> /trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
> 1351988
>
> /trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
> 1351988
>
> /trunk/flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java
> 1351988
>
> /trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestCheckpoint.java
> 1351988
>
> /trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java
> 1351988
>
> /trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFlumeEventQueue.java
> 1351988
>
> /trunk/flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLog.java
> 1351988
> /trunk/flume-ng-core/src/main/java/org/apache/flume/sink/LoggerSink.java
> 1351988
>
> Diff: https://reviews.apache.org/r/5436/diff/
>
>
> Testing
> -------
>
> Ran all tests. Did some manual testing. Will be doing more manual testing and
> cleanup as necessary while the review is underway.
>
>
> Thanks,
>
> Arvind Prabhakar
>
>
