[
https://issues.apache.org/jira/browse/NIFI-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paweł Rzepiński reassigned NIFI-16350:
--------------------------------------
Assignee: Paweł Rzepiński
> ConsumeKafka RECORD strategy and verification treat Kafka tombstones as parse
> failures
> --------------------------------------------------------------------------------------
>
> Key: NIFI-16350
> URL: https://issues.apache.org/jira/browse/NIFI-16350
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Wojciech Trefon
> Assignee: Paweł Rzepiński
> Priority: Major
>
> h2. Summary
> A Kafka tombstone is a record with a non-null key and a null value. When
> ConsumeKafka uses the RECORD processing strategy, tombstones are passed to
> the configured Record Reader as zero-length content.
> With StandardProtobufReader and ConfluentEncodedSchemaReferenceReader, this
> causes tombstones to be treated as malformed Protobuf records. The same
> record also causes ConsumeKafka verification to fail in the Parse Records
> step.
> h2. Steps to reproduce
> # Publish a Kafka record with a non-null key and a null value.
> # Configure ConsumeKafka with Processing Strategy set to RECORD.
> # Configure StandardProtobufReader with ConfluentEncodedSchemaReferenceReader.
> # Stop ConsumeKafka and run Verify.
> # Observe that the Parse Records verification step fails on the tombstone.
> # Start ConsumeKafka normally.
> # Observe that the tombstone is routed as a parse failure while the consumer
> offset advances.
> h2. Actual behavior
> Verify reports an error similar to:
> {code}
> Parse Records
> Failed to parse Record number 9:
> org.apache.nifi.schema.access.SchemaNotFoundException: Failed to read header
> in first 5 bytes from stream
> {code}
> During normal RECORD processing, the zero-length value reaches
> StandardProtobufReader and is routed to the parse failure relationship. If
> that relationship is auto-terminated, the tombstone is silently discarded
> while consumption continues and the offset advances.
> This differs from FLOW_FILE processing, where a tombstone can be represented
> by an empty FlowFile with the kafka.tombstone attribute set to true and
> routed through success.
> h2. Expected behavior
> A valid Kafka tombstone should not be classified as a malformed Protobuf
> record solely because it has no schema header. Verification and normal
> processing should handle the same tombstone consistently and preserve the
> distinction between a null Kafka value and a non-null empty value.
> h2. Code observation
> Kafka3ConsumerService converts a null ConsumerRecord value to a zero-length
> byte array before RECORD processing. At that point, downstream record parsing
> cannot distinguish a tombstone from a non-null empty value.
> h2. Related issues
> * [NIFI-12371|https://issues.apache.org/jira/browse/NIFI-12371] added
> tombstone support for non-record Kafka processors and noted that record-based
> processors might require separate handling.
> * [NIFI-14122|https://issues.apache.org/jira/browse/NIFI-14122] and
> [NIFI-14123|https://issues.apache.org/jira/browse/NIFI-14123] restored
> tombstone support in Kafka publish paths.
> * [NIFI-14247|https://issues.apache.org/jira/browse/NIFI-14247] covers
> ConsumeKafka tombstones in FLOW_FILE processing.
> The related issues do not cover ConsumeKafka RECORD processing or the Verify
> Parse Records step.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
