prankstrisse opened a new pull request, #11499: URL: https://github.com/apache/nifi/pull/11499
<!-- Licensed to the Apache Software Foundation (ASF) under one or more --> <!-- contributor license agreements. See the NOTICE file distributed with --> <!-- this work for additional information regarding copyright ownership. --> <!-- The ASF licenses this file to You under the Apache License, Version 2.0 --> <!-- (the "License"); you may not use this file except in compliance with --> <!-- the License. You may obtain a copy of the License at --> <!-- http://www.apache.org/licenses/LICENSE-2.0 --> <!-- Unless required by applicable law or agreed to in writing, software --> <!-- distributed under the License is distributed on an "AS IS" BASIS, --> <!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. --> <!-- See the License for the specific language governing permissions and --> <!-- limitations under the License. --> # Summary [NIFI-14210](https://issues.apache.org/jira/browse/NIFI-14210) ## Changes - **`StandardProtobufWriter`** (`nifi-protobuf-services`) — new `SchemaRegistryService` + `RecordSetWriterFactory`. Resolves the target Proto message either statically via a `Message Name` property or dynamically via a `MessageNameResolver` service, compiles the schema with the existing `ProtobufSchemaCompiler`, and serializes each Record with a new `ProtobufDataSerializer`. A single message is written per FlowFile, since concatenated Protobuf messages cannot be delimited on read. - **`ProtobufDataSerializer`** — maps NiFi Record fields onto Wire `Schema`/`MessageType` descriptors and encodes the binary payload. `google.protobuf.Any`-typed fields are written as plain nested messages rather than re-wrapped as `Any`. - **`WriteProtobufResultWithExternalSchema`** — orchestrates header + message-index + payload sequencing, mirroring the existing `WriteAvroResultWithExternalSchema` pattern: invokes `SchemaReferenceWriter.writeHeader(...)` when a Schema Reference Writer is configured, then an optional `MessageIndexWriter.writeMessageIndex(...)` step, then the Protobuf payload. - **`MessageIndexWriter`** (`nifi-schema-registry-service-api`) — new shared interface, the write-side inverse of `MessageNameResolver`: given a resolved message name and schema definition, encodes the path to that message and writes it to the output stream so a reader can resolve the same name back from the encoded path. - **`ConfluentProtobufMessageIndexWriter`** + **`ProtobufMessageIndexEncoder`** (`nifi-confluent-protobuf-message-index-writer`, new module) — Confluent implementation of `MessageIndexWriter`. Walks nested message declarations to find the target message's declaration-order path and zigzag-varint encodes it (including the `[0]` single-message optimization), mirroring the decoding done by `ConfluentProtobufMessageNameResolver`. Together with the unchanged, already-generic `ConfluentEncodedSchemaReferenceWriter`, this reproduces the Confluent wire format (5-byte header + message-index array + payload) on write. - **`VarintUtils`** — relocated from the message-name-resolver module to `org.apache.nifi.confluent.schema` and made non-`package-private` so it can be shared between the resolver (read) and index writer (write) modules. - Minor visibility/refactoring changes to `ProtobufSchemaCompiler` and `ProtobufSchemaValidator` to support reuse from the writer. ## Testing - `ProtobufMessageIndexEncoderTest`, `ConfluentProtobufMessageIndexWriterTest` — unit tests for index-path encoding, including nested messages and the single-index optimization. - `TestProtobufDataSerializer` — Record-to-Protobuf field mapping and encoding. - `TestStandardProtobufWriter` — controller service property/validation behavior. - `TestStandardProtobufWriterConfluentRoundTrip` — end-to-end round trip: `StandardProtobufWriter` output (header + index + payload) is read back by `StandardProtobufReader`/`ConfluentProtobufMessageNameResolver` and compared against the original Records. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000` - [x] Pull request contains [commits signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) with a registered key indicating `Verified` status ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `./mvnw clean install -P contrib-check` - [ ] JDK 21 - [ ] JDK 25 ### Licensing - [x] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [x] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
