[
https://issues.apache.org/jira/browse/NIFI-14878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18016843#comment-18016843
]
ASF subversion and git services commented on NIFI-14878:
--------------------------------------------------------
Commit 3097a90f0519f023cc640053eb89055a30de0d81 in nifi's branch
refs/heads/main from lkuchars
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=3097a90f05 ]
NIFI-14878 Added StandardProtobufReader for Protobuf Record Processing (#10214)
Co-authored-by: ForThoseWhoComeAfter <[email protected]>
Signed-off-by: David Handermann <[email protected]>
> Implement StandardProtobufReader Controller Service for Protobuf Record
> Processing
> ----------------------------------------------------------------------------------
>
> Key: NIFI-14878
> URL: https://issues.apache.org/jira/browse/NIFI-14878
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
> Affects Versions: 2.6.0
> Reporter: Lukas Kucharski
> Assignee: Lukas Kucharski
> Priority: Major
> Time Spent: 9h
> Remaining Estimate: 0h
>
> h2. Summary
> Implement StandardProtobufReader Controller Service for Protobuf Record
> Processing
> h2. Description
> This enhancement completes the Protobuf support implementation by introducing
> the *StandardProtobufReader* controller service. This represents Phase 2 of
> the Confluent Protobuf integration, building upon the foundational services
> delivered in Phase 1 (linked).
> The new reader addresses limitations of the existing *ProtobufReader*
> component, which requires manual proto file management and cannot effectively
> operate in clustered environments without manual schema distribution across
> nodes. Also, the existing reader is designed to work with filesystem source
> only, as the 'Proto Directory' property is always required, even when the
> schema access strategy is set to 'Text Property'
> h2. Key Components Implemented
> *StandardProtobufReader* Controller Service
> - New Record Reader Implementation: Controller service extending
> *SchemaRegistryService* for Protobuf record processing
> - Multiple Schema Access Strategies: Supports flexible schema resolution
> methods without requiring local proto files
> h3. Schema Access Strategy Support
> The service supports the following schema access strategies:
> 1. Schema Text: Direct schema definition input
> 2. Schema Name from Registry: Reference schemas by name from the configured
> registry
> 3. Schema Reference Reader: Use the schema id embedded in the protobuf
> payload header. Then use the schema registry to fetch the schema by the the
> id.
> h2. Key points
> - {*}ProtobufSchemaCompiler{*}: component for compiling and managing
> Protobuf schemas
> - Schema Caching
> - Reference Resolution: Automatic resolution of imported/referenced schemas
> from registry
> - Compiles raw Protobuf schema definitions using wire library
> - Handles complex schema dependencies and imports
> - Schema Resolution: Determines schema source based on configured strategy
> - Message Name Resolution: Identifies target message type from payload or
> configuration
> - Record Deserialization: Converts Protobuf binary data to NiFi records
>
> h3. Design Decisions
> - New Implementation: Created as separate component rather than extending
> existing ProtobufReader to avoid breaking changes
> - Focused Scope: Does not support "Generate From Proto File" strategy,
> focusing on registry-based approaches
> - Clean API: Streamlined property configuration without legacy constraints
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
