[ 
https://issues.apache.org/jira/browse/NIFI-16350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18141268#comment-18141268
 ] 

ASF subversion and git services commented on NIFI-16350:
--------------------------------------------------------

Commit bb9e46c779d29094c2027687bcf418e28372124a in nifi's branch 
refs/heads/main from Paweł Rzepiński
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=bb9e46c779d ]

NIFI-16350 Skip Kafka tombstones in RECORD mode instead of routing them to 
parse failure (#11857)

* NIFI-16350 Skip Kafka tombstones in ConsumeKafka RECORD mode

Null values were indistinguishable from empty payloads, so RECORD mode and 
Verify treated tombstones as parse failures. Keep the distinction and skip only 
real tombstones while still committing their offsets.

Co-authored-by: Cursor <[email protected]>

* NIFI-16350 Address review feedback on ConsumeKafka tombstone handling

Remove the ByteRecord constructor that defaulted tombstone to false.
Callers now pass the tombstone flag explicitly. Consolidate the
tombstone integration and converter tests, and rename test helpers
for clarity.

Co-authored-by: Cursor <[email protected]>

---------

Co-authored-by: Cursor <[email protected]>

> ConsumeKafka RECORD strategy and verification treat Kafka tombstones as parse 
> failures
> --------------------------------------------------------------------------------------
>
>                 Key: NIFI-16350
>                 URL: https://issues.apache.org/jira/browse/NIFI-16350
>             Project: Apache NiFi
>          Issue Type: Bug
>            Reporter: Wojciech Trefon
>            Assignee: Paweł Rzepiński
>            Priority: Major
>          Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> h2. Summary
> A Kafka tombstone is a record with a non-null key and a null value. When 
> ConsumeKafka uses the RECORD processing strategy, tombstones are passed to 
> the configured Record Reader as zero-length content.
> With StandardProtobufReader and ConfluentEncodedSchemaReferenceReader, this 
> causes tombstones to be treated as malformed Protobuf records. The same 
> record also causes ConsumeKafka verification to fail in the Parse Records 
> step.
> h2. Steps to reproduce
> # Publish a Kafka record with a non-null key and a null value.
> # Configure ConsumeKafka with Processing Strategy set to RECORD.
> # Configure StandardProtobufReader with ConfluentEncodedSchemaReferenceReader.
> # Stop ConsumeKafka and run Verify.
> # Observe that the Parse Records verification step fails on the tombstone.
> # Start ConsumeKafka normally.
> # Observe that the tombstone is routed as a parse failure while the consumer 
> offset advances.
> h2. Actual behavior
> Verify reports an error similar to:
> {code}
> Parse Records
> Failed to parse Record number 9: 
> org.apache.nifi.schema.access.SchemaNotFoundException: Failed to read header 
> in first 5 bytes from stream
> {code}
> During normal RECORD processing, the zero-length value reaches 
> StandardProtobufReader and is routed to the parse failure relationship. If 
> that relationship is auto-terminated, the tombstone is silently discarded 
> while consumption continues and the offset advances.
> This differs from FLOW_FILE processing, where a tombstone can be represented 
> by an empty FlowFile with the kafka.tombstone attribute set to true and 
> routed through success.
> h2. Expected behavior
> A valid Kafka tombstone should not be classified as a malformed Protobuf 
> record solely because it has no schema header. Verification and normal 
> processing should handle the same tombstone consistently and preserve the 
> distinction between a null Kafka value and a non-null empty value.
> h2. Code observation
> Kafka3ConsumerService converts a null ConsumerRecord value to a zero-length 
> byte array before RECORD processing. At that point, downstream record parsing 
> cannot distinguish a tombstone from a non-null empty value.
> h2. Related issues
> * [NIFI-12371|https://issues.apache.org/jira/browse/NIFI-12371] added 
> tombstone support for non-record Kafka processors and noted that record-based 
> processors might require separate handling.
> * [NIFI-14122|https://issues.apache.org/jira/browse/NIFI-14122] and 
> [NIFI-14123|https://issues.apache.org/jira/browse/NIFI-14123] restored 
> tombstone support in Kafka publish paths.
> * [NIFI-14247|https://issues.apache.org/jira/browse/NIFI-14247] covers 
> ConsumeKafka tombstones in FLOW_FILE processing.
> The related issues do not cover ConsumeKafka RECORD processing or the Verify 
> Parse Records step.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to