[
https://issues.apache.org/jira/browse/FLUME-533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080595#comment-13080595
]
Jonathan Hsieh commented on FLUME-533:
--------------------------------------
I ported patch and created a test that is similar to the ruby script mentioned
above. The entire event seems to have gotten dropped in this case. I'm
editing to make it take the first maxeventsize bytes and then truncate the rest
until a new event delimiter is reached.
> TailSource line length limitation
> ---------------------------------
>
> Key: FLUME-533
> URL: https://issues.apache.org/jira/browse/FLUME-533
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v0.9.3
> Reporter: Disabled imported user
> Assignee: Disabled imported user
> Labels: TailSource, line_length, sink
> Fix For: v0.9.5
>
> Attachments: 0001-Fix-line-length-limitation-in-TailSource.patch
>
>
> It appears that having a line with length of 32767 bytes or more
> throws the TailSource into a, um, tail spin. I've set
> flume.event.max.size.bytes to 1000000, so that shouldn't be the
> limitation. Simple steps to reproduce:
> flume-0.9.3rc1> ruby -e 'puts "x"*32767' > /tmp/log
> flume-0.9.3rc1> bin/flume dump 'tail("/tmp/log")'
> It appears that the source sends an incomplete message to the sink,
> and then continuously retries to send it.
> I've attached a patch that creates a buffer of size
> flume.event.max.size.bytes,
> and if it detects a line longer than this, it gets thrown away. It's been
> working
> for our systems for a week, but I haven't had time to write a unit test and
> test
> the edge cases.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
