[
https://issues.apache.org/jira/browse/FLUME-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16883682#comment-16883682
]
Nicolas Laduguie commented on FLUME-2580:
-----------------------------------------
Hi, what about this feature request ?
We would appreciate it because we have the use case where we need to consume
events from Kafka and directly write them to HDFS, without passing through a
channel which would either not prevent data loss (file channel, memory
channel), or duplicate data to guarantee data integrity (jdbc channel, kafka
channel).
Our last need is to compute HDFS path according to Kafka messages specific
fields like date or anything else, but we can't do it because we need an
interceptor either on the channel side or on the sink side.
Thank you for your support.
Regards
> Sink side interceptors
> ----------------------
>
> Key: FLUME-2580
> URL: https://issues.apache.org/jira/browse/FLUME-2580
> Project: Flume
> Issue Type: New Feature
> Reporter: Hari Shreedharan
> Assignee: Hari Shreedharan
> Priority: Major
> Attachments: SinkSideInterceptors.pdf
>
>
> Currently, we only have source-side interceptors that help routing. But if we
> use something like Kafka Channel, having a sink-side interceptor can help us
> modify events as they come in. We could also do validation on event schemas
> and drop them before they hit the sink, rather than have an infinite loop due
> to such events.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
