[
https://issues.apache.org/jira/browse/FLUME-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15221121#comment-15221121
]
Li Ye edited comment on FLUME-2848 at 4/1/16 4:48 AM:
------------------------------------------------------
This issue is similar to Flume-2789.
An optinal property called ignoreTopicInHeader is added for Kafka Sink. Its
default value is false, so it is compatible with Flume 1.6.0. If you want to
ignore topic in header and write events to the topic you specified in
properties file, you can set ignoreTopicInHeader to true.
Besides, three optinal properties topicHeader, keyHeader, timestampHeader are
added for Kafka Source. They are similar to fileHeader and basenameHeader for
Spooling Directory Source. Their default value are true, so they are compatible
with Flume 1.6.0. If you do not want to add headers storing topic, key or
timestamp, you can set them to false. It is also helpful for performance of
Kafka Source.
was (Author: l90006322):
This issue is similar to Flume-2789.
An optinal property called ignoreTopicInHeader is added for Kafka Sink. Its
default value is false, so it is compatible with Flume 1.6.0. If you want to
ignore topic in header and write events to the topic you specified in
properties file, you can set ignoreTopicInHeader to true.
Besides, three optinal properties topicHeader, keyHeader, timestampHeader are
added for Kafka Sink. They are similar to fileHeader and basenameHeader for
Spooling Directory Source.
Their default value are true, so they are compatible with Flume 1.6.0. If you
do not want to add headers storing topic, key or timestamp, you can set them to
false.
> Kafka Source could not work with Kafka Sink
> -------------------------------------------
>
> Key: FLUME-2848
> URL: https://issues.apache.org/jira/browse/FLUME-2848
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.6.0
> Environment: IMHO any
> Reporter: Arkadiusz Firus
> Labels: easyfix
> Attachments: FLUME-2789-1.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> KafkaSource stores information from which topic a record was read. This
> information is stored in an event header with the key "topic" (see
> org.apache.flume.source.kafka.KafkaSource line 108).
> KafkaSink can be instructed to store record in a different topic than the one
> in configuration (dynamic destination). To do that "topic" header have to be
> set in an event (see org.apache.flume.sink.kafka.KafkaSink line 107).
> When this two features are combined we have infinite loop - KafkaSink writes
> records to the topic which is read by KafkaSource.
> Two solve the problem fields:
> org.apache.flume.sink.kafka.KafkaSink.TOPIC_HDR
> org.apache.flume.source.kafka.KafkaSourceConstants.TOPIC
> need to have different values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
