-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7267/
-----------------------------------------------------------
Review request for Flume.
Description
-------
Disables writing to the log when there is less than 50MB free in any of the log
directories. 50MB is not configurable but it seems like a trivial minimum
amount to support considering most disks are now several terabytes.
It might make sense to make this configurable and increase the default to 100MB
or something?
Also changes the background worker to ensure this the space available is
checked once per second.
This addresses bug FLUME-1609.
https://issues.apache.org/jira/browse/FLUME-1609
Diffs
-----
flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java
64725dd
Diff: https://reviews.apache.org/r/7267/diff/
Testing
-------
Manual testing is required as we do not have a file system abstraction.
To test this, I first created a 100MB in memory file system.
I then filled the file system to ensure that file channel would refuse to
replay with the file system full.
Creating the big file:
{noformat}
# dd if=/dev/zero of=/mnt/tmpfs/bigfile
dd: writing to `/mnt/tmpfs/bigfile': No space left on device
204393+0 records in
204392+0 records out
104648704 bytes (105 MB) copied, 13.6317 seconds, 7.7 MB/s
{noformat}
The channel refused to start:
{noformat}
2012-09-25 13:13:51,111 (lifecycleSupervisor-1-0) [ERROR -
org.apache.flume.channel.file.FileChannel.start(FileChannel.java:269)] Failed
to start the file channel [channel=channel1]
java.lang.IllegalStateException: No available free space
at
com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at org.apache.flume.channel.file.Log.replay(Log.java:285)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:258)
at
org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:236)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I then deleted the file, stopped the channel and created a 40MB file. In this
case, the agent was going to start, do work, and then after some period of time
the file system would become full. I expected to see an error message that this
occured:
Creating the file:
{noformat}
# dd if=/dev/zero of=/mnt/tmpfs/bigfile count=80000
{noformat}
The channel stopped allowing writes:
{noformat}
2012-09-25 13:17:40,572 (Log-BackgroundWorker-channel1) [ERROR -
org.apache.flume.channel.file.Log.checkFreeSpace(Log.java:898)] Log is being
disabled because [/tmp/flume/data1-1, /tmp/flume/data1-2, /tmp/flume/data1-3]
have exceeded their available space. [channel=channel1]
2012-09-25 13:17:40,573 (Log-BackgroundWorker-channel1) [ERROR -
org.apache.flume.channel.file.Log.checkFreeSpace(Log.java:905)] Directory
/tmp/flume/data1-1 has 51179520 bytes available which is less than the required
minimum of 52428800 bytes
2012-09-25 13:17:40,575 (Log-BackgroundWorker-channel1) [ERROR -
org.apache.flume.channel.file.Log.checkFreeSpace(Log.java:905)] Directory
/tmp/flume/data1-2 has 51179520 bytes available which is less than the required
minimum of 52428800 bytes
2012-09-25 13:17:40,576 (Log-BackgroundWorker-channel1) [ERROR -
org.apache.flume.channel.file.Log.checkFreeSpace(Log.java:905)] Directory
/tmp/flume/data1-3 has 51179520 bytes available which is less than the required
minimum of 52428800 bytes
2012-09-25 13:17:40,581
(PollableSourceRunner-SequenceGeneratorSource-source1-2) [ERROR -
org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:156)]
Unhandled exception, logging and sleeping for 5000ms
java.lang.IllegalStateException: No available free space
at
com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at org.apache.flume.channel.file.Log.rollback(Log.java:503)
at
org.apache.flume.channel.file.FileChannel$FileBackedTransaction.doRollback(FileChannel.java:530)
at
org.apache.flume.channel.BasicTransactionSemantics.rollback(BasicTransactionSemantics.java:168)
at
org.apache.flume.channel.ChannelProcessor.processEvent(ChannelProcessor.java:269)
at
org.apache.flume.source.SequenceGeneratorSource.process(SequenceGeneratorSource.java:68)
at
org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I then deleted the file and expected the channel to continue working after a
period of time. It did.
{noformat}
2012-09-25 13:18:26,028 (Log-BackgroundWorker-channel1) [INFO -
org.apache.flume.channel.file.Log.checkFreeSpace(Log.java:893)] Log was
disabled because a log directory has too little free space. However, the space
has cleared and the log will be enabled. [channel=channel1]
...
2012-09-25 13:18:37,179
(PollableSourceRunner-SequenceGeneratorSource-source1-2) [DEBUG -
org.apache.flume.channel.file.LogFile$Writer.preallocate(LogFile.java:253)]
Preallocating at position 5242788
2012-09-25 13:18:37,189
(PollableSourceRunner-SequenceGeneratorSource-source1-1) [INFO -
org.apache.flume.channel.file.Log.roll(Log.java:736)] Roll start
/tmp/flume/data1-2
2012-09-25 13:18:37,190
(PollableSourceRunner-SequenceGeneratorSource-source1-1) [INFO -
org.apache.flume.channel.file.LogFile$Writer.<init>(LogFile.java:138)] Opened
/tmp/flume/data1-2/log-5
2012-09-25 13:18:37,192
(PollableSourceRunner-SequenceGeneratorSource-source1-1) [INFO -
org.apache.flume.channel.file.LogFile$Writer.close(LogFile.java:236)] Closing
/tmp/flume/data1-2/log-2
2012-09-25 13:18:37,193
(PollableSourceRunner-SequenceGeneratorSource-source1-2) [DEBUG -
org.apache.flume.channel.file.LogFile$Writer.preallocate(LogFile.java:253)]
Preallocating at position 0
2012-09-25 13:18:37,193
(PollableSourceRunner-SequenceGeneratorSource-source1-1) [INFO -
org.apache.flume.channel.file.Log.roll(Log.java:751)] Roll end
{noformat}
Thanks,
Brock Noland
