[
https://issues.apache.org/jira/browse/KAFKA-20709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kravii updated KAFKA-20709:
---------------------------
Description:
Routing messages from a single source topic to multiple destination topics
based on configurable Kafka record headers, with priority ordering when
multiple headers are present, is not available natively in Apache Kafka today.
*Requirement:*
A built in SMT under mirror maker, one that reads a record header's value and
uses it to determine the destination topic.
*Motivation:*
Modern data platforms increasingly need to route records based on metadata
carried on the record itself, rather than on topic names or timestamps alone.
Required capabilities -
* Multi-tenant Kafka platforms that need per-tenant topic isolation driven by
tenant-id headers
* Data protection regulations, requiring data segregation by classification,
sensitivity, or residency.
* Content classification pipelines where an upstream classifier writes its
decision into a header and downstream routing must honor it.
* Tiered processing pipelines that route records to different topics based on
priority or SLA metadata.
was:
Routing messages from a single source topic to multiple destination topics
based on configurable Kafka record headers, with priority ordering when
multiple headers are present, is not available natively in Apache Kafka today.
Plugin required in the mirror maker to handle this.
*Requirement:*
A built in SMT, one that reads a record header's value and uses it to determine
the destination topic.
*Motivation:*
Modern data platforms increasingly need to route records based on metadata
carried on the record itself, rather than on topic names or timestamps alone.
Required capabilities -
* Multi-tenant Kafka platforms that need per-tenant topic isolation driven by
tenant-id headers
* Data protection regulations, requiring data segregation by classification,
sensitivity, or residency.
* Content classification pipelines where an upstream classifier writes its
decision into a header and downstream routing must honor it.
* Tiered processing pipelines that route records to different topics based on
priority or SLA metadata.
> Add built-in SMT for header-based topic routing in Kafka Connect /
> MirrorMaker 2 with configurable priority across multiple headers
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20709
> URL: https://issues.apache.org/jira/browse/KAFKA-20709
> Project: Kafka
> Issue Type: New Feature
> Components: connect, mirrormaker
> Affects Versions: 3.7.2
> Reporter: kravii
> Priority: Major
> Labels: connect-transformation, mirror-maker
>
> Routing messages from a single source topic to multiple destination topics
> based on configurable Kafka record headers, with priority ordering when
> multiple headers are present, is not available natively in Apache Kafka today.
> *Requirement:*
> A built in SMT under mirror maker, one that reads a record header's value and
> uses it to determine the destination topic.
> *Motivation:*
> Modern data platforms increasingly need to route records based on metadata
> carried on the record itself, rather than on topic names or timestamps alone.
> Required capabilities -
> * Multi-tenant Kafka platforms that need per-tenant topic isolation driven
> by tenant-id headers
> * Data protection regulations, requiring data segregation by classification,
> sensitivity, or residency.
> * Content classification pipelines where an upstream classifier writes its
> decision into a header and downstream routing must honor it.
> * Tiered processing pipelines that route records to different topics based
> on priority or SLA metadata.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
