Yes, depends on a E2E chain. Stop the agent, go to /tmp/name-of-collector and remove all directories there ( should be sent, sending, tmp and another one ). Then echo > your-file (erase them) and restart flume. You will lost the logs from, but I didnt know another solution yet. Hope it helps.
Alex sent via my mobile device On Dec 1, 2011, at 8:43 PM, Mark Lewandowski <mark.e.lewandow...@gmail.com> wrote: > Hmmm, the only problem is that these boxes are EC2 instances, and because > flume is writing such large files I'm filling 100% of my disk space before > retransmission would even occur. The files are being put in the sent > directory, which I believe means that they've been successfully sent to the > collector. > > I tried stopping the flume process, truncating the log file I'm writing to, > and wiping the sent/, writing/, logged/, sending/ directories in tmp (where > flume writes while doing E2E), and restarting the process that writes the > original log. Even after all that, when I restart flume it starts trying to > send over these huge chunks of data. > > Is there something else I can try blowing away that would reset the WAL? > > -Mark > > On Thu, Dec 1, 2011 at 11:01 AM, Eric Sammer <esam...@cloudera.com> wrote: > Mark: > > This is due to some kind of wacky state in the WAL (I'm assuming this is E2E) > if I remember correctly. The issue is that events keep piling up in the WAL > and it never gets drained properly. I don't have the details in front of me > but I vaguely remember this being related to downstream ACK issues (or could > be a remanent thereof). You could try taking a look at the logs on the errant > host to confirm the retransmission and lack of ACK. > > Let us know. > > > On Thu, Dec 1, 2011 at 10:53 AM, Mark Lewandowski > <mark.e.lewandow...@gmail.com> wrote: > Hi all, > > I've got 9 identical boxes configured in my flume cluster. 8 of them are > working correctly, with exactly the same config (source: tail, sink: > AutoE2EChain). 1 box is trying to send way too much data through my > collector. I'm tailing a log file that's currently ~200k, but when I start > the flume agent, and I the writing and sending directories I see that flume > is trying to send files of ~1GB. A new 1GB file is created in this directory > about every 10s. > > Any ideas? > > -Mark > > > > -- > Eric Sammer > twitter: esammer > data: www.cloudera.com >
