[
https://issues.apache.org/jira/browse/FLUME-2848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arkadiusz Firus updated FLUME-2848:
-----------------------------------
Description:
KafkaSource stores information from which topic a record was read. This
information is stored in an event header with the key "topic" (see
org.apache.flume.source.kafka.KafkaSource line 108).
KafkaSink can be instructed to store record in a different topic than the one
in configuration (dynamic destination). To do that "topic" header have to be
set in an event (see org.apache.flume.sink.kafka.KafkaSink line 107).
When this two features are combined we have infinite loop - KafkaSink writes
records to the topic which is read by KafkaSource.
Two solve the problem fields:
org.apache.flume.sink.kafka.KafkaSink.TOPIC_HDR
org.apache.flume.source.kafka.KafkaSourceConstants.TOPIC
need to have different values.
was:
KafkaSource stores information from which topic record was read. This
information is stored in event header with key "topic" (see
org.apache.flume.source.kafka.KafkaSource line 108).
KafkaSink can be instructed to store record in a different topic than the one
in configuration (dynamic destination). To do that "topic" header have to be
set in event (see org.apache.flume.sink.kafka.KafkaSink line 107).
When this two features are combined we have infinite loop - KafkaSink writes
record to topic which is read by KafkaSource.
Two solve the problem fields:
org.apache.flume.sink.kafka.KafkaSink.TOPIC_HDR
org.apache.flume.source.kafka.KafkaSourceConstants.TOPIC
need to have different values.
> Kafka Source could not work with Kafka Sink
> -------------------------------------------
>
> Key: FLUME-2848
> URL: https://issues.apache.org/jira/browse/FLUME-2848
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.6.0
> Environment: IMHO any
> Reporter: Arkadiusz Firus
> Labels: easyfix
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> KafkaSource stores information from which topic a record was read. This
> information is stored in an event header with the key "topic" (see
> org.apache.flume.source.kafka.KafkaSource line 108).
> KafkaSink can be instructed to store record in a different topic than the one
> in configuration (dynamic destination). To do that "topic" header have to be
> set in an event (see org.apache.flume.sink.kafka.KafkaSink line 107).
> When this two features are combined we have infinite loop - KafkaSink writes
> records to the topic which is read by KafkaSource.
> Two solve the problem fields:
> org.apache.flume.sink.kafka.KafkaSink.TOPIC_HDR
> org.apache.flume.source.kafka.KafkaSourceConstants.TOPIC
> need to have different values.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
