Did everyone go on vacation?
Ralph
On Sep 21, 2011, at 8:49 PM, Ralph Goers wrote:
>
> On Sep 20, 2011, at 3:02 PM, Derek Deeter wrote:
>
>> We have set up a configuration that uses the collector sink with our own
>> sink (called GlobalLogSink). Configuration is:
>>
>> collector (15000) { GlobalLogSink(...) }
>>
>> The GlobalLogSink connects to Cassandra which we have not fully tuned yet.
>> We are running into an exception in the RollSink code that causes
>> DirectDriver to close down, which occurs when Cassandra does a Garbage
>> Collection or some other time intensive action that causes a response to be
>> longer than necessary. I have pasted two tracebacks at the end of this
>> email, one from the collector and the other from jstack.
>>
>> The code at RollSink:296 does a write with a wait set to 1000 millis; if
>> the write takes more than that, an exception occurs that shuts the sink
>> down.
>>
>> When testing in our development environment, we found that this problem went
>> away, but there we were using the first flume-0.9.4 release, which has the
>> 'synchronize' functionality before the ReentrantReadWriteLock was
>> implemented, so we had to back down a release or two from flume-0.9.4+25.6,
>> but then we ran into Ack problems.
>>
>> There are a couple fixes needed to correct this issue:
>> We would like to add a fix to RollSink.java to make the 1000ms delay
>> configurable.
>> Also, when the timeout does occur, the Sink should fail gracefully and not
>> just hang/close down.
>> I would like to open a JIRA on this.
>>
>> To fix the problem I need to know what source/branch to use - this error
>> occurs in the source associated with the flume-0.9.4+25 RPM file where we
>> found source at 0.9.4-cdh3u1 that corresponds to that. Is there a branch
>> label or tag that we should use to fix this?
>>
>> A couple of related questions we'd appreciate answers to:
>> 1) Is there a way to set up a collector sink that doesn't use RollSink?
>> 2) Is there a way to get acknowledgments without using RollSink?
>>
>> Ideally, we would like to have something like an 'AckSink' that's lets us
>> utilize E2E with out having the RollSink functionality.
>>
>> Thanks in advance,
>> Derek Deeter
>
> I'd really like some information as to why the CollectorSink is hard coded to
> have a RollSink. Although the use described above wants acknowledgements it
> isn't clear why closing and reopening is necessary. I would think that in
> this use case the locking wouldn't be necessary just to send the
> acknowledgements.
>
> Can someone explain why the code works this way?
>
> Ralph
