[
https://issues.apache.org/jira/browse/FLUME-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15097032#comment-15097032
]
Roshan Naik commented on FLUME-2799:
------------------------------------
My comments:
- Needs update to documentation wrt Converter, converter config and custom
converters.
- The default converter here is applying 4 headers (2 old, plus 2 new ones).
Adding headers to every event is expensive in terms of memory (and also some
cpu due to added GC pressure). Which headers to apply should be user selectable
with the default settings preserving existing behavior.
- The need for introducing Convertor for adding additional headers may be a bit
overkill, but acceptable.
> Kafka Source - Message Offset and Partition add to headers
> ----------------------------------------------------------
>
> Key: FLUME-2799
> URL: https://issues.apache.org/jira/browse/FLUME-2799
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.6.0
> Reporter: Michael Andre Pearce (IG)
> Priority: Minor
> Labels: easyfix, patch
> Fix For: v1.7.0
>
> Attachments: FLUME-2799-0.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Currently Kafka source only persists the original kafka message's topic into
> the Flume event headers.
> For downstream interceptors and sinks that may want to have available to them
> the partition and the offset , we need to add these.
> Also it is noted that the conversion from MessageAndMetaData to FlumeEvent is
> not configurable unlike other sources such as JMS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
