Daniel Urban created NIFI-10199:
-----------------------------------
Summary: PublishKafkaRecord should not flood the logs when a batch
times out
Key: NIFI-10199
URL: https://issues.apache.org/jira/browse/NIFI-10199
Project: Apache NiFi
Issue Type: Improvement
Reporter: Daniel Urban
Currently, PublishKafkaRecord reports errors on the record level. This means
that when an error occurs on the batch level, the same error is reported for
each record of said batch.
For example, in case of a delivery timeout, a transactional Kafka producer logs
a message like this:
{code:java}
2022-06-29 13:16:20,913 INFO
org.apache.kafka.clients.producer.internals.TransactionManager: [Producer
clientId=producer-XYZ, transactionalId=XYZ] Transiting to abortable error state
due to org.apache.kafka.common.errors.TimeoutException: Expiring 63 record(s)
for TOPIC-01:302002 ms has passed since batch creation {code}
Then, following this, we get N lines, where N is the number of records in the
batch. Also note that the TimeoutException is printed twice in this message.
{code:java}
2022-06-29 13:16:20,913 ERROR
org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_2_6:
PublishKafkaRecord_2_6[id=91ba351d-96ad-1d0d-934c-cfd07e0bacd5] Failed to send
StandardFlowFileRecord[uuid=349b4d38-451d-4e62-8ea9-636f7c666fc1,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1656453186665-7724448,
container=default, section=416], offset=0,
length=132021252],offset=0,name=NAME,size=132021252] to Kafka:
org.apache.kafka.common.errors.TimeoutException: Expiring 63 record(s) for
PROD_trn_lba_icode_vld_v2-38:302002 ms has passed since batch creation
org.apache.kafka.common.errors.TimeoutException: Expiring 63 record(s) for
PROD_trn_lba_icode_vld_v2-38:302002 ms has passed since batch creation{code}
Experienced this with PublishKafkaRecord_2_6, but other versions might be
affected as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
