----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7177/#review11797 -----------------------------------------------------------
Thinking about this more, I think we should put some kind of limit on how fast we roll due to an exception. If the file system was full, it looks we'd end up creating a lot of log-* files which could cause us to run out of file ids (2^31) much faster than expected. For example if we threw an exception right after creating the file (due to writing to it), we could see a scenario where: a) write fails b) we roll c) roll creates an empty log-* file but is unable to write to it, exception thrown d) repeat all of this happening quite quickly since none of it would actually hit disk. Another option is to only increment the file id if we are able to successfully write out the metadata file and sync it to disk. - Brock Noland On Sept. 20, 2012, 4:17 p.m., Brock Noland wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7177/ > ----------------------------------------------------------- > > (Updated Sept. 20, 2012, 4:17 p.m.) > > > Review request for Flume. > > > Description > ------- > > This resolves the issue by introducing a special exception > LogFileClosedException which the caller catches when the file has closed and > retries. Additionally LogFileExceededMaxSizeException is introduced. Instead > of making the asynchronous check, an exception is thrown when the max size is > exceeded. The caller than rolls the log to a new writer. > > Additionally, the error handling logic has been rolled into one method. > > > This addresses bug FLUME-1593. > https://issues.apache.org/jira/browse/FLUME-1593 > > > Diffs > ----- > > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java > e36eafb > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java > 8071140 > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileRetryableIOException.java > PRE-CREATION > > flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java > 8baf8fe > > flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java > 87d9c3f > > Diff: https://reviews.apache.org/r/7177/diff/ > > > Testing > ------- > > New unit tests and old unit tests pass. > > > Thanks, > > Brock Noland > >
