> On Sept. 21, 2012, 7:42 p.m., Brock Noland wrote: > > Thinking about this more, I think we should put some kind of limit on how > > fast we roll due to an exception. If the file system was full, it looks > > we'd end up creating a lot of log-* files which could cause us to run out > > of file ids (2^31) much faster than expected. For example if we threw an > > exception right after creating the file (due to writing to it), we could > > see a scenario where: > > > > a) write fails > > b) we roll > > c) roll creates an empty log-* file but is unable to write to it, exception > > thrown > > d) repeat > > > > all of this happening quite quickly since none of it would actually hit > > disk. > > > > Another option is to only increment the file id if we are able to > > successfully write out the metadata file and sync it to disk. > > Hari Shreedharan wrote: > I will file another jira to handle this issue. This is very exceptional > condition, so I won't consider this urgent.
agreed - Brock ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7177/#review11797 ----------------------------------------------------------- On Sept. 20, 2012, 4:17 p.m., Brock Noland wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7177/ > ----------------------------------------------------------- > > (Updated Sept. 20, 2012, 4:17 p.m.) > > > Review request for Flume. > > > Description > ------- > > This resolves the issue by introducing a special exception > LogFileClosedException which the caller catches when the file has closed and > retries. Additionally LogFileExceededMaxSizeException is introduced. Instead > of making the asynchronous check, an exception is thrown when the max size is > exceeded. The caller than rolls the log to a new writer. > > Additionally, the error handling logic has been rolled into one method. > > > This addresses bug FLUME-1593. > https://issues.apache.org/jira/browse/FLUME-1593 > > > Diffs > ----- > > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java > e36eafb > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java > 8071140 > > flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileRetryableIOException.java > PRE-CREATION > > flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java > 8baf8fe > > flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java > 87d9c3f > > Diff: https://reviews.apache.org/r/7177/diff/ > > > Testing > ------- > > New unit tests and old unit tests pass. > > > Thanks, > > Brock Noland > >
