[
https://issues.apache.org/jira/browse/BEAM-4038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16458325#comment-16458325
]
Raghu Angadi commented on BEAM-4038:
------------------------------------
The interfaces with function call backs are problematic since we need to decide
what context to provide for the function. E.g.
{{KafkaPublishTimestampFunction}} provides {{elementTimestamp}}. Why? since we
think that is probably what user wants to know along with KV. Recently we
deprecated old timestamp functions for reader in order to support watermarks
better. At least in the case of reader, there is no alternative to having a
function callback since we need to set watermark/timestamp _before_ the user
gets to see the record.
Long story short, I think it is better to avoid another function to set
headers. It will be similar story when more fields are added to
{{KafkaRecord}}. In fact I think we should remove
{{KafkaPublishTimestampFunction}}.
How about adding {{KafkaIO.writeRecords()}} which is a
{{PCollection<{ProducerRecord<K, V>, PDone>}}? This way user builds the
ProducerRecord anyway they see fit. We can provide Avro coder for Kafka's
{{ProducerRecord}}. We can handle older kafka versions by ignoring fields that
are not present in old versions. We can add a coder very similar to
{{KafkaRecordCoder}}.
This is more work than adding a function, but I think it improve flexibility of
writer now and for future.
> Support Kafka Headers in KafkaIO
> --------------------------------
>
> Key: BEAM-4038
> URL: https://issues.apache.org/jira/browse/BEAM-4038
> Project: Beam
> Issue Type: New Feature
> Components: io-java-kafka
> Reporter: Geet Kumar
> Assignee: Geet Kumar
> Priority: Minor
> Fix For: 2.5.0
>
> Time Spent: 9h 10m
> Remaining Estimate: 0h
>
> Headers have been added to Kafka Consumer/Producer records (KAFKA-4208). The
> purpose of this JIRA is to support this feature in KafkaIO.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
