Guy, What version of flume is this?
-Jeff On Fri, Nov 7, 2014 at 1:19 AM, Needham, Guy <[email protected]> wrote: > Hi all, > > I have a configuration with a file channel configured such that: > > a1.channels.ch1.type = file > a1.channels.ch1.checkpointDir = /hadoop/user/flume/channels/checkpoint > a1.channels.ch1.dataDirs = /hadoop/user/flume/channels/data > a1.channels.ch1.capacity = 100000 > a1.channels.ch1.transactionCapacity = 5000 > > It's been running since October 28th with no issues, but when I looked > today in /hadoop/user/flume/channels/data I saw that the file channel was > building up large files which had been processed and not deleting them: > > [rdd@hadoop-kn-p2-m01 flume]$ ls -lh channels/data/ > total 1.6G > -rw-r----- 1 rdd rdd 1.5G Oct 28 16:10 log-1 > -rw-r----- 1 rdd rdd 47 Oct 28 16:10 log-1.meta > -rw-r----- 1 rdd rdd 72M Oct 31 16:28 log-2 > -rw-r----- 1 rdd rdd 47 Oct 31 16:29 log-2.meta > It seems like for each day that data landed (we're still in testing so > data not landing constantly) a data file has been created but not deleted > when reading was completed. > Is this expected behaviour? Is there a way to stop large files building up > and still use the file channel? > Regards, > Guy Needham | Data Discovery > Virgin Media | Enterprise Data, Design & Management > Bartley Wood Business Park, Hook, Hampshire RG27 9UP > D 01256 75 3362 > I welcome VSRE emails. Learn more at *http://vsre.info/* > <http://vsre.info/> > > > > > -------------------------------------------------------------------- > Save Paper - Do you really need to print this e-mail? > > Visit www.virginmedia.com for more information, and more fun. > > This email and any attachments are or may be confidential and legally > privileged > and are sent solely for the attention of the addressee(s). If you have > received this > email in error, please delete it from your system: its use, disclosure or > copying is > unauthorised. Statements and opinions expressed in this email may not > represent > those of Virgin Media. Any representations or commitments in this email are > subject to contract. > > Registered office: Media House, Bartley Wood Business Park, Hook, > Hampshire, RG27 9UP > Registered in England and Wales with number 2591237 >
