[
https://issues.apache.org/jira/browse/NIFI-11259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann reassigned NIFI-11259:
---------------------------------------
Assignee: Paul Grey (was: David Handermann)
> Redesign Kafka Component Integration
> ------------------------------------
>
> Key: NIFI-11259
> URL: https://issues.apache.org/jira/browse/NIFI-11259
> Project: Apache NiFi
> Issue Type: Epic
> Components: Extensions
> Reporter: David Handermann
> Assignee: Paul Grey
> Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> NiFi has supported integration with Kafka across a number of major release
> versions. As Kafka has continued to release new major versions, NiFi
> integration has followed a general pattern of duplicating existing code,
> updating Kafka libraries, and making small modifications. Although this
> approach has allowed NiFi to support a range of Kafka versions, it has made
> Kafka components difficult to maintain.
> A new approach should be designed and implemented that abstracts access to
> Kafka library components. This approach should provide more maintainable
> Kafka Processors, and also define a better path to support new major Kafka
> releases.
> The general approach should consist of a NiFi Controller Service and
> supporting classes that abstract Kafka operations for publishing and
> consuming messages. The Controller Service interface should not have any
> dependencies on Kafka libraries, making it independent of Kafka versions. The
> interface should define a contract for accessing Kafka Brokers so that
> Controller Service implementations will be responsible for connection and
> authentication settings. Controller Service implementations should be aligned
> to major Kafka versions, allowing minor dependency version updates without
> changing Controller Service implementations.
> New Kafka Processors should be designed and implemented to use the Controller
> Service interface, without requiring knowledge of Kafka connection details.
> New Processors should support the same basic production and consumption
> capabilities, with configurable strategies for Kafka Record handling. These
> Processors should support the same capabilities as current Processors,
> including treating entire FlowFiles as Kafka Records, as well as processing
> FlowFiles using a configurable delimiter or using NiFi Record services.
> New Kafka components should build on the pattern of the nifi-kafka-shared
> module to promote code reuse across major Kafka versions.
> The new decoupled Kafka Processors and Controller Services should enable
> better unit testing. The Testcontainers library should considered for
> integration testing with different Kafka runtime versions.
> Documentation should be written to describe common migration scenarios from
> existing Kafka Processors to new components.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
