Lukas Kucharski created NIFI-14878:
--------------------------------------
Summary: Implement StandardProtobufReader Controller Service for
Protobuf Record Processing
Key: NIFI-14878
URL: https://issues.apache.org/jira/browse/NIFI-14878
Project: Apache NiFi
Issue Type: Improvement
Components: Extensions
Affects Versions: 2.6.0
Reporter: Lukas Kucharski
Assignee: Lukas Kucharski
h2. Summary
Implement StandardProtobufReader Controller Service for Protobuf Record
Processing
h2. Description
This enhancement completes the Protobuf support implementation by introducing
the *StandardProtobufReader* controller service. This represents Phase 2 of the
Confluent Protobuf integration, building upon the foundational services
delivered in Phase 1 (linked).
The new reader addresses limitations of the existing *ProtobufReader*
component, which requires manual proto file management and cannot effectively
operate in clustered environments without manual schema distribution across
nodes. Also, the existing reader is designed to work with filesystem source
only, as the 'Proto Directory' property is always required, even when the
schema access strategy is set to 'Text Property'
h2. Key Components Implemented
*StandardProtobufReader* Controller Service
- New Record Reader Implementation: Controller service extending
*SchemaRegistryService* for Protobuf record processing
- Multiple Schema Access Strategies: Supports flexible schema resolution
methods without requiring local proto files
h3. Schema Access Strategy Support
The service supports the following schema access strategies:
1. Schema Text: Direct schema definition input
2. Schema Name from Registry: Reference schemas by name from the configured
registry
3. Schema Reference Reader: Use the schema id embedded in the protobuf payload
header. Then use the schema registry to fetch the schema by the the id.
h2. Key points
- {*}ProtobufSchemaCompiler{*}: component for compiling and managing Protobuf
schemas
- Schema Caching: (200 schemas, 1-hour expiration)
- Reference Resolution: Automatic resolution of imported/referenced schemas
from registry
- Compiles raw Protobuf schema definitions using wire library
- Handles complex schema dependencies and imports
- Schema Resolution: Determines schema source based on configured strategy
- Message Name Resolution: Identifies target message type from payload or
configuration
- Record Deserialization: Converts Protobuf binary data to NiFi records
h3. Design Decisions
- New Implementation: Created as separate component rather than extending
existing ProtobufReader to avoid breaking changes
- Focused Scope: Does not support "Generate From Proto File" strategy,
focusing on registry-based approaches
- Clean API: Streamlined property configuration without legacy constraints
Use Case Scenarios
Enterprise Kafka Integration
- Process Protobuf messages from Kafka topics using Confluent Schema Registry
- Support for complex message hierarchies and nested schemas
- Dynamic message type resolution from wire format headers
Multi-Schema Environments
- Handle topics containing multiple Protobuf message types
- Automatic schema evolution support through registry integration
- Efficient processing of high-volume Protobuf data streams
Configuration Properties
- Schema Access Strategy: Selection of schema resolution method
- Schema Registry: Reference to configured schema registry service
- Schema Reference Reader: Service for reading embedded schema references
- Message Name Resolver: Service for dynamic message name resolution
- Cache Configuration: Tunable caching parameters for performance optimization
Business Value
- Simplified Protobuf Processing: Eliminates manual proto file management in
production environments
- Enterprise-Ready: Full support for clustered deployments and schema registry
integration
- Performance Optimized: Intelligent caching reduces schema compilation
overhead
- Future-Proof: Extensible design supporting evolving Protobuf use cases
Verification
- Successfully built and tested with Maven and JDK 21
- Integration tested with Confluent Schema Registry environments
- Performance validated with schema caching mechanisms
- Cluster deployment scenarios verified
- Requires foundational services from NIFI-14424 Phase 1
(ConfluentProtobufMessageNameResolver, SchemaDefinition, MessageNameResolver
API)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
