[
https://issues.apache.org/jira/browse/NIFI-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17575995#comment-17575995
]
ASF subversion and git services commented on NIFI-10199:
--------------------------------------------------------
Commit 387ae7d32318c1cd765c1794aff70f4b80a40b28 in nifi's branch
refs/heads/main from Daniel Urban
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=387ae7d323 ]
NIFI-10199: PublishKafka InFlightMessageTracker should not log batch level
errors on each record
Currently, when a batch level error occurs (e.g. delivery timeout), the same
error is logged for each record of the batch, needlessly flooding the logs.
InFlightMessageTracker now only logs an exception when a flow file encounters
it for the first time.
This closes #6185
Signed-off-by: Paul Grey <[email protected]>
> PublishKafkaRecord should not flood the logs when a batch times out
> -------------------------------------------------------------------
>
> Key: NIFI-10199
> URL: https://issues.apache.org/jira/browse/NIFI-10199
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Daniel Urban
> Assignee: Daniel Urban
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently, PublishKafkaRecord reports errors on the record level. This means
> that when an error occurs on the batch level, the same error is reported for
> each record of said batch.
> For example, in case of a delivery timeout, a transactional Kafka producer
> logs a message like this:
> {code:java}
> 2022-06-29 13:16:20,913 INFO
> org.apache.kafka.clients.producer.internals.TransactionManager: [Producer
> clientId=producer-XYZ, transactionalId=XYZ] Transiting to abortable error
> state due to org.apache.kafka.common.errors.TimeoutException: Expiring 63
> record(s) for TOPIC-01:302002 ms has passed since batch creation {code}
> Then, following this, we get N lines, where N is the number of records in the
> batch. Also note that the TimeoutException is printed twice in this message.
> {code:java}
> 2022-06-29 13:16:20,913 ERROR
> org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_2_6:
> PublishKafkaRecord_2_6[id=91ba351d-96ad-1d0d-934c-cfd07e0bacd5] Failed to
> send
> StandardFlowFileRecord[uuid=349b4d38-451d-4e62-8ea9-636f7c666fc1,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1656453186665-7724448,
> container=default, section=416], offset=0,
> length=132021252],offset=0,name=NAME,size=132021252] to Kafka:
> org.apache.kafka.common.errors.TimeoutException: Expiring 63 record(s) for
> PROD_trn_lba_icode_vld_v2-38:302002 ms has passed since batch creation
> org.apache.kafka.common.errors.TimeoutException: Expiring 63 record(s) for
> PROD_trn_lba_icode_vld_v2-38:302002 ms has passed since batch creation{code}
> Experienced this with PublishKafkaRecord_2_6, but other versions might be
> affected as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
