[
https://issues.apache.org/jira/browse/NIFI-10199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joe Witt updated NIFI-10199:
----------------------------
Fix Version/s: 1.18.0
> PublishKafkaRecord should not flood the logs when a batch times out
> -------------------------------------------------------------------
>
> Key: NIFI-10199
> URL: https://issues.apache.org/jira/browse/NIFI-10199
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Daniel Urban
> Assignee: Daniel Urban
> Priority: Major
> Fix For: 1.18.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Currently, PublishKafkaRecord reports errors on the record level. This means
> that when an error occurs on the batch level, the same error is reported for
> each record of said batch.
> For example, in case of a delivery timeout, a transactional Kafka producer
> logs a message like this:
> {code:java}
> 2022-06-29 13:16:20,913 INFO
> org.apache.kafka.clients.producer.internals.TransactionManager: [Producer
> clientId=producer-XYZ, transactionalId=XYZ] Transiting to abortable error
> state due to org.apache.kafka.common.errors.TimeoutException: Expiring 63
> record(s) for TOPIC-01:302002 ms has passed since batch creation {code}
> Then, following this, we get N lines, where N is the number of records in the
> batch. Also note that the TimeoutException is printed twice in this message.
> {code:java}
> 2022-06-29 13:16:20,913 ERROR
> org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_2_6:
> PublishKafkaRecord_2_6[id=91ba351d-96ad-1d0d-934c-cfd07e0bacd5] Failed to
> send
> StandardFlowFileRecord[uuid=349b4d38-451d-4e62-8ea9-636f7c666fc1,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1656453186665-7724448,
> container=default, section=416], offset=0,
> length=132021252],offset=0,name=NAME,size=132021252] to Kafka:
> org.apache.kafka.common.errors.TimeoutException: Expiring 63 record(s) for
> PROD_trn_lba_icode_vld_v2-38:302002 ms has passed since batch creation
> org.apache.kafka.common.errors.TimeoutException: Expiring 63 record(s) for
> PROD_trn_lba_icode_vld_v2-38:302002 ms has passed since batch creation{code}
> Experienced this with PublishKafkaRecord_2_6, but other versions might be
> affected as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
