[
https://issues.apache.org/jira/browse/NIFI-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16176144#comment-16176144
]
sundaramoorthy M commented on NIFI-4380:
----------------------------------------
Team,
I would recommend to have the schema ID and VERSION Number also sent as a part
of the flow file attribute when consuming data from Kafka.
This will aid to help the schema evolution strategy OR to identify the schema
change.
Hope this can be accommodated in a single jira.
Thanks,
Sundar.
> Add Kafka Key attribute to outgoing FF for ConsumeKafkaRecord
> -------------------------------------------------------------
>
> Key: NIFI-4380
> URL: https://issues.apache.org/jira/browse/NIFI-4380
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Andrew Psaltis
>
> Often times during processing of data having access to the Key used for the
> message in Kafka is important. For example, some CDC tools that use Kafka as
> the destination for changes will use the primary key for the table as the key
> when writing data to Kafka. The value of this key becomes import after the is
> consumed from Kakfa, as many times the final destination is another data
> store, for which you need to know which column(s) of data represents the key.
> When you introduce a Schema Registry into the picture then many times the key
> is written into Kafka is often encoded, for example, if Avro is being used
> for data serialization into Kafka then the key may have its own schema that
> is serialized with. Thus, this change would need to ensure that the key is
> deserialized before being written as an attribute.
> Not sure if it gets overly complex or if it is required to have Reader/Writer
> configuration for the key as well as the value. It is certainly to be
> expected that is the key and value are stored in Kafka in Avro that the
> schema for each will be unique.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
