-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7267/
-----------------------------------------------------------
(Updated Oct. 16, 2012, 3:54 p.m.)
Review request for Flume.
Changes
-------
Updated patch disallows writes unless there is 10MB + the buffer size.
Description (updated)
-------
Updated patch is very simple, simply disables writes when the file system has
less than 10MB + the buffer size. There is a race condition in that it's
possible for the file system to fill after we do this check, but that should be
fine because it would just cause a roll and then the next operation this check
would fail the write prior to actually doing the write.
This addresses bug FLUME-1609.
https://issues.apache.org/jira/browse/FLUME-1609
Diffs (updated)
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
64725dd
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java
a2c790c
Diff: https://reviews.apache.org/r/7267/diff/
Testing
-------
Manual testing is required as we do not have a file system abstraction.
To test this, I first created a 100MB in memory file system.
I then filled the file system to ensure that file channel would refuse to
replay with the file system full.
Creating the big file:
{noformat}
# dd if=/dev/zero of=/mnt/tmpfs/bigfile
dd: writing to `/mnt/tmpfs/bigfile': No space left on device
204393+0 records in
204392+0 records out
104648704 bytes (105 MB) copied, 13.6317 seconds, 7.7 MB/s
{noformat}
The channel refused to start:
{noformat}
2012-09-25 13:13:51,111 (lifecycleSupervisor-1-0) [ERROR -
org.apache.flume.channel.file.FileChannel.start(FileChannel.java:269)] Failed
to start the file channel [channel=channel1]
java.lang.IllegalStateException: No available free space
at
com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at org.apache.flume.channel.file.Log.replay(Log.java:285)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:258)
at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I then deleted the file, stopped the channel and created a 40MB file. In this
case, the agent was going to start, do work, and then after some period of time
the file system would become full. I expected to see an error message that this
occured:
Creating the file:
{noformat}
# dd if=/dev/zero of=/mnt/tmpfs/bigfile count=80000
{noformat}
The channel stopped allowing writes:
{noformat}
2012-09-25 13:17:40,572 (Log-BackgroundWorker-channel1) [ERROR -
org.apache.flume.channel.file.Log.checkFreeSpace(Log.java:898)] Log is being
disabled because [/tmp/flume/data1-1, /tmp/flume/data1-2, /tmp/flume/data1-3]
have exceeded their available space. [channel=channel1]
2012-09-25 13:17:40,573 (Log-BackgroundWorker-channel1) [ERROR -
org.apache.flume.channel.file.Log.checkFreeSpace(Log.java:905)] Directory
/tmp/flume/data1-1 has 51179520 bytes available which is less than the required
minimum of 52428800 bytes
2012-09-25 13:17:40,575 (Log-BackgroundWorker-channel1) [ERROR -
org.apache.flume.channel.file.Log.checkFreeSpace(Log.java:905)] Directory
/tmp/flume/data1-2 has 51179520 bytes available which is less than the required
minimum of 52428800 bytes
2012-09-25 13:17:40,576 (Log-BackgroundWorker-channel1) [ERROR -
org.apache.flume.channel.file.Log.checkFreeSpace(Log.java:905)] Directory
/tmp/flume/data1-3 has 51179520 bytes available which is less than the required
minimum of 52428800 bytes
2012-09-25 13:17:40,581
(PollableSourceRunner-SequenceGeneratorSource-source1-2) [ERROR -
org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:156)]
Unhandled exception, logging and sleeping for 5000ms
java.lang.IllegalStateException: No available free space
at
com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at org.apache.flume.channel.file.Log.rollback(Log.java:503)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:530)
at
org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
at
org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:269)
at
org.apache.flume.source.SequenceGeneratorSource.process(SequenceGeneratorSource.java:68)
at
org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I then deleted the file and expected the channel to continue working after a
period of time. It did.
{noformat}
2012-09-25 13:18:26,028 (Log-BackgroundWorker-channel1) [INFO -
org.apache.flume.channel.file.Log.checkFreeSpace(Log.java:893)] Log was
disabled because a log directory has too little free space. However, the space
has cleared and the log will be enabled. [channel=channel1]
...
2012-09-25 13:18:37,179
(PollableSourceRunner-SequenceGeneratorSource-source1-2) [DEBUG -
org.apache.flume.channel.file.LogFile$Writer.preallocate(LogFile.java:253)]
Preallocating at position 5242788
2012-09-25 13:18:37,189
(PollableSourceRunner-SequenceGeneratorSource-source1-1) [INFO -
org.apache.flume.channel.file.Log.roll(Log.java:736)] Roll start
/tmp/flume/data1-2
2012-09-25 13:18:37,190
(PollableSourceRunner-SequenceGeneratorSource-source1-1) [INFO -
org.apache.flume.channel.file.LogFile$Writer.<init>(LogFile.java:138)] Opened
/tmp/flume/data1-2/log-5
2012-09-25 13:18:37,192
(PollableSourceRunner-SequenceGeneratorSource-source1-1) [INFO -
org.apache.flume.channel.file.LogFile$Writer.close(LogFile.java:236)] Closing
/tmp/flume/data1-2/log-2
2012-09-25 13:18:37,193
(PollableSourceRunner-SequenceGeneratorSource-source1-2) [DEBUG -
org.apache.flume.channel.file.LogFile$Writer.preallocate(LogFile.java:253)]
Preallocating at position 0
2012-09-25 13:18:37,193
(PollableSourceRunner-SequenceGeneratorSource-source1-1) [INFO -
org.apache.flume.channel.file.Log.roll(Log.java:751)] Roll end
{noformat}
Thanks,
Brock Noland
