This could be of help https://issues.apache.org/jira/browse/FLUME-2413 (you need this at Sink level but this should work as well)
HTH! On Mon, Feb 16, 2015 at 4:01 PM, mahendran m <[email protected]> wrote: > I am moving the IIS logs into remote hdfs using spooling directory and I > need to update how much data (size not no of events) has been moved > successfully in real time in UI > > Thanks, > ------------------------------ > Date: Mon, 16 Feb 2015 12:59:06 +0530 > Subject: Re: No threads used in Spooling Directory > From: [email protected] > To: [email protected] > > > What Flume Dev's meant was, Flume is not intended to do File transfers as > a whole. A File shall be read and an Event created out of the content based > on how an Event needs to be created (each line as an Event by default). > > To answer you question (to the best of my knowledge) > > Q1. I would like to know how much file has been processed in parallel by > SpoolDir ? > Nope > > Q2. Is it possible to how much* size* of file has been moved by SpoolDir > to sink at any point using built-in API? > Nope > If you can share what you want to achieve, we may be able to provide some > pointers. > thanks > ashish > > > > On Fri, Feb 13, 2015 at 5:43 PM, mahendran m <[email protected]> > wrote: > > I need to move IIS logs to hdfs , then which is suitable for this purpose ? > > Thanks , > > ------------------------------ > From: [email protected] > To: [email protected] > Subject: RE: No threads used in Spooling Directory > Date: Fri, 13 Feb 2015 11:00:48 +0000 > > > I had posted for similar requirement few days back , but from core team > members got to know that spoolDir is not suitable for fileBased handling, > its more of event based content transfer that is being done. > > *From:* mahendran m [mailto:[email protected]] > *Sent:* Friday, February 13, 2015 11:02 AM > *To:* FlumeMaling List > *Subject:* No threads used in Spooling Directory > > HI , > > Q1. I would like to know how much file has been processed in parallel > by SpoolDir ? > > Q2. Is it possible to how much* size* of file has been moved by SpoolDir > to sink at any point using built-in API? > > Thanks , > > > > Please consider the environment before printing this e-mail > > > Disclaimer: This communication is for the exclusive use of the intended > recipient(s) and shall not attach any liability on the originator or ITC > Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group > Companies. If you are the addressee, the contents of this e-mail are intended > for your use only and it shall not be forwarded to any third party, without > first obtaining written authorization from the originator or ITC Infotech > India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It > may contain information which is confidential and legally privileged and the > same shall not be used or dealt with by any third party in any manner > whatsoever without the specific consent of ITC Infotech India Ltd./ its > Holding company/ its Subsidiaries/ its Group Companies. > > > > > -- > thanks > ashish > > Blog: http://www.ashishpaliwal.com/blog > My Photo Galleries: http://www.pbase.com/ashishpaliwal > -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal
