Wojciech Trefon created NIFI-16350:
--------------------------------------
Summary: ConsumeKafka RECORD strategy and verification treat Kafka
tombstones as parse failures
Key: NIFI-16350
URL: https://issues.apache.org/jira/browse/NIFI-16350
Project: Apache NiFi
Issue Type: Bug
Reporter: Wojciech Trefon
h2. Summary
A Kafka tombstone is a record with a non-null key and a null value. When
ConsumeKafka uses the RECORD processing strategy, tombstones are passed to the
configured Record Reader as zero-length content.
With StandardProtobufReader and ConfluentEncodedSchemaReferenceReader, this
causes tombstones to be treated as malformed Protobuf records. The same record
also causes ConsumeKafka verification to fail in the Parse Records step.
h2. Steps to reproduce
# Publish a Kafka record with a non-null key and a null value.
# Configure ConsumeKafka with Processing Strategy set to RECORD.
# Configure StandardProtobufReader with ConfluentEncodedSchemaReferenceReader.
# Stop ConsumeKafka and run Verify.
# Observe that the Parse Records verification step fails on the tombstone.
# Start ConsumeKafka normally.
# Observe that the tombstone is routed as a parse failure while the consumer
offset advances.
h2. Actual behavior
Verify reports an error similar to:
{code}
Parse Records
Failed to parse Record number 9:
org.apache.nifi.schema.access.SchemaNotFoundException: Failed to read header in
first 5 bytes from stream
{code}
During normal RECORD processing, the zero-length value reaches
StandardProtobufReader and is routed to the parse failure relationship. If that
relationship is auto-terminated, the tombstone is silently discarded while
consumption continues and the offset advances.
This differs from FLOW_FILE processing, where a tombstone can be represented by
an empty FlowFile with the kafka.tombstone attribute set to true and routed
through success.
h2. Expected behavior
A valid Kafka tombstone should not be classified as a malformed Protobuf record
solely because it has no schema header. Verification and normal processing
should handle the same tombstone consistently and preserve the distinction
between a null Kafka value and a non-null empty value.
h2. Code observation
Kafka3ConsumerService converts a null ConsumerRecord value to a zero-length
byte array before RECORD processing. At that point, downstream record parsing
cannot distinguish a tombstone from a non-null empty value.
h2. Related issues
* [NIFI-12371|https://issues.apache.org/jira/browse/NIFI-12371] added tombstone
support for non-record Kafka processors and noted that record-based processors
might require separate handling.
* [NIFI-14122|https://issues.apache.org/jira/browse/NIFI-14122] and
[NIFI-14123|https://issues.apache.org/jira/browse/NIFI-14123] restored
tombstone support in Kafka publish paths.
* [NIFI-14247|https://issues.apache.org/jira/browse/NIFI-14247] covers
ConsumeKafka tombstones in FLOW_FILE processing.
The related issues do not cover ConsumeKafka RECORD processing or the Verify
Parse Records step.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
