You should check whether your version of Flume supports the ignorePattern configuration param. The latest version on trunk does.
Mike On Wed, May 22, 2013 at 12:43 AM, ZORAIDA HIDALGO SANCHEZ <[email protected]>wrote: > Oh! I see, then was a miss understanding. Ok, so we will need to find a > workaround. Thanks a lot. > > De: Mike Percy <[email protected]> > Responder a: Flume User List <[email protected]> > Fecha: miércoles, 22 de mayo de 2013 09:35 > Para: Flume User List <[email protected]> > Asunto: Re: Spooling fileSuffix attribute ignored > > Hi Phil, > Nice approach. How is the spooling directory source working for you? Any > thoughts on how it could be improved? > > Mike > > > On Tue, May 21, 2013 at 8:17 AM, Phil Scala <[email protected]>wrote: > >> Hi,**** >> >> ** ** >> >> Based on my use and understanding that setting “fileSuffix” is simpy the >> extension to the file to be added once the file was consumed and placed >> onto the channel. I don’t think it was intended to be used to indicate a >> completely uploaded file. In the dev newsgroup there was a discussion >> about having the spooler ”wait” for a little while before ingesting the >> file. An ignore pattern may also be a good idea.**** >> >> ** ** >> >> Currently I am using an upload directory that I monitor and when lsof >> reports a file is not in use/open I then move it to my spool directory. >> **** >> >> ** ** >> >> HTH**** >> >> Phil**** >> >> ** ** >> >> ** ** >> >> ** ** >> >> Phil Scala**** >> >> Software Developer / Architect >> Global Relay >> >> [email protected] >> >> *866.484.6630* | >> [email protected]<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=mailto%3ainfo%40globalrelay.net> >> | >> globalrelay.com<https://ex1.globalrelay.net/owa/redir.aspx?C=01df0579a4cc42feb9f7e2801368f729&URL=http%3a%2f%2fwww.globalrelay.com%2f> >> **** >> >> ** ** >> >> *From:* ZORAIDA HIDALGO SANCHEZ [mailto:[email protected]] >> *Sent:* Tuesday, May 21, 2013 9:21 AM >> *To:* Flume User List >> *Subject:* Spooling fileSuffix attribute ignored**** >> >> ** ** >> >> Dear all, **** >> >> I am using the Spooling attribute "fileSuffix" to ensure that only when a >> file has been completely upload into the Spooling dir, is consumed by the >> source. However, files with no suffix are also appended to the channel and >> then processed by the sink. **** >> >> My configuration:**** >> >> ** ** >> >> tier1.sources = s1**** >> >> tier1.channels = c1**** >> >> tier1.sinks = s1**** >> >> ** ** >> >> # For each source, channel, and sink, set**** >> >> # standard properties.**** >> >> tier1.sources.s1.type = spooldir**** >> >> tier1.sources.s1.spoolDir = /home/user/flume/data**** >> >> tier1.sources.s1.deletePolicy = immediate**** >> >> tier1.sources.s1.batchSize = 1000**** >> >> tier1.sources.s1.bufferMaxLines = 3000**** >> >> tier1.sources.s1.fileHeader = true**** >> >> tier1.sources.s1.fileSuffix=.COMPLETED**** >> >> ** ** >> >> Is that ok? **** >> >> ** ** >> >> Thanks.**** >> >> ** ** >> ------------------------------ >> >> >> Este mensaje se dirige exclusivamente a su destinatario. Puede consultar >> nuestra política de envío y recepción de correo electrónico en el enlace >> situado más abajo. >> This message is intended exclusively for its addressee. We only send and >> receive email on the basis of the terms set out at: >> http://www.tid.es/ES/PAGINAS/disclaimer.aspx**** >> > > > ------------------------------ > > Este mensaje se dirige exclusivamente a su destinatario. Puede consultar > nuestra política de envío y recepción de correo electrónico en el enlace > situado más abajo. > This message is intended exclusively for its addressee. We only send and > receive email on the basis of the terms set out at: > http://www.tid.es/ES/PAGINAS/disclaimer.aspx >
