[
https://issues.apache.org/jira/browse/FLUME-2251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14130406#comment-14130406
]
Hari Shreedharan commented on FLUME-2251:
-----------------------------------------
There a couple of things which need to be fixed in the -2 patch (-3 does not
apply):
- The channel.getTransaction should be inside the try, since the channel
stashes the transaction in a thread-local. So if the getTransaction throws, the
next time you call it, you'd get the same transaction.
- The first one means, there should be a null check before rollback and close
calls.
- transaction.rollback() can throw, for example - if the file channel cannot
write to disk for some reason. That should be in try catch (see other sinks) -
with the exception being logged.
- The most important issue is the lack of batching in the sink. A batch size of
1 is going to give performance that is pretty much unusable. We need to fix
this. This is fairly simple to do - add a config param and put the process
method's code within the try in a loop (of course, one transaction per process
method call, so the tx,begin etc is outside the loop).
- Why not make the "MessagePreprocessor" configurable and pass the context in
just when the sink starts up - the way HDFS and HBase sinks do it?
> Add support for Kafka Sink
> --------------------------
>
> Key: FLUME-2251
> URL: https://issues.apache.org/jira/browse/FLUME-2251
> Project: Flume
> Issue Type: Sub-task
> Components: Sinks+Sources
> Affects Versions: v1.4.0
> Reporter: Ashish Paliwal
> Priority: Minor
> Labels: feature, patch
> Attachments: FLUME-2251-0.patch, FLUME-2251-2.patch,
> FLUME-2251-3.patch, FLUME-2251.patch, FLUME-2251.patch, Flume-2251-1.patch
>
>
> Add support for Kafka Sink
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
