[jira] [Commented] (KAFKA-8021) KafkaProducer.flush() can show unexpected behavior when a batch is split
[ https://issues.apache.org/jira/browse/KAFKA-8021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16781850#comment-16781850 ] Abhishek Mendhekar commented on KAFKA-8021: --- [~sriharsha] the issue is in 0.11 and higher version (including latest trunk). Since this is a very small race I am trying to reproduce this in an integration test but I was able to verify it via unit tests. > KafkaProducer.flush() can show unexpected behavior when a batch is split > > > Key: KAFKA-8021 > URL: https://issues.apache.org/jira/browse/KAFKA-8021 > Project: Kafka > Issue Type: Bug > Components: clients, producer >Affects Versions: 0.11.0.0 >Reporter: Abhishek Mendhekar >Assignee: Abhishek Mendhekar >Priority: Major > > KafkaProducer.flush() marks the flush in progress and then waits for all > incomplete batches to be completed (waits on the producer batch futures to > finish). > The behavior is seen when a batch is split due to MESSAGE_TOO_LARGE exception. > The large batch is split into smaller batches (2 or more) but > ProducerBatch.split() marks the large batch future as complete before adding > the new batches in the incomplete list of batches. At this time if the > KafkaProducer.flush() is called then it'll make a copy of existing incomplete > list of batches and waits for them to complete while ignoring the large batch > that was split into smaller batches. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8021) KafkaProducer.flush() can show unexpected behavior when a batch is split
Abhishek Mendhekar created KAFKA-8021: - Summary: KafkaProducer.flush() can show unexpected behavior when a batch is split Key: KAFKA-8021 URL: https://issues.apache.org/jira/browse/KAFKA-8021 Project: Kafka Issue Type: Bug Components: clients, producer Affects Versions: 0.11.0.0 Reporter: Abhishek Mendhekar Assignee: Abhishek Mendhekar KafkaProducer.flush() marks the flush in progress and then waits for all incomplete batches to be completed (waits on the producer batch futures to finish). The behavior is seen when a batch is split due to MESSAGE_TOO_LARGE exception. The large batch is split into smaller batches (2 or more) but ProducerBatch.split() marks the large batch future as complete before adding the new batches in the incomplete list of batches. At this time if the KafkaProducer.flush() is called then it'll make a copy of existing incomplete list of batches and waits for them to complete while ignoring the large batch that was split into smaller batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-5461) KIP-168: Add GlobalTopicCount metric per cluster
[ https://issues.apache.org/jira/browse/KAFKA-5461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Mendhekar updated KAFKA-5461: -- Fix Version/s: (was: 0.11.0.1) 0.11.1.0 > KIP-168: Add GlobalTopicCount metric per cluster > > > Key: KAFKA-5461 > URL: https://issues.apache.org/jira/browse/KAFKA-5461 > Project: Kafka > Issue Type: Improvement >Reporter: Abhishek Mendhekar >Assignee: Abhishek Mendhekar > Fix For: 0.11.1.0 > > > See KIP > [here|https://cwiki.apache.org/confluence/display/KAFKA/KIP-168%3A+Add+TopicCount+metric+per+cluster] > Email discussion > [here|http://mail-archives.apache.org/mod_mbox/kafka-dev/201706.mbox/%3CCAMcwe-ugep-UiSn9TkKEMwwTM%3DAzGC4jPro9LnyYRezyZg_NKA%40mail.gmail.com%3E] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5461) KIP-168: Add GlobalTopicCount metric per cluster
[ https://issues.apache.org/jira/browse/KAFKA-5461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Mendhekar updated KAFKA-5461: -- Summary: KIP-168: Add GlobalTopicCount metric per cluster (was: KIP-168: Add TotalTopicCount metric per cluster) > KIP-168: Add GlobalTopicCount metric per cluster > > > Key: KAFKA-5461 > URL: https://issues.apache.org/jira/browse/KAFKA-5461 > Project: Kafka > Issue Type: Improvement >Reporter: Abhishek Mendhekar >Assignee: Abhishek Mendhekar > Fix For: 0.11.0.1 > > > See KIP > [here|https://cwiki.apache.org/confluence/display/KAFKA/KIP-168%3A+Add+TopicCount+metric+per+cluster] > Email discussion > [here|http://mail-archives.apache.org/mod_mbox/kafka-dev/201706.mbox/%3CCAMcwe-ugep-UiSn9TkKEMwwTM%3DAzGC4jPro9LnyYRezyZg_NKA%40mail.gmail.com%3E] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KAFKA-5461) KIP-168: Add TotalTopicCount metric per cluster
[ https://issues.apache.org/jira/browse/KAFKA-5461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Mendhekar updated KAFKA-5461: -- Description: See KIP [here|https://cwiki.apache.org/confluence/display/KAFKA/KIP-168%3A+Add+TotalTopicCount+metric+per+cluster] Email discussion [here|http://mail-archives.apache.org/mod_mbox/kafka-dev/201706.mbox/%3CCAMcwe-ugep-UiSn9TkKEMwwTM%3DAzGC4jPro9LnyYRezyZg_NKA%40mail.gmail.com%3E] was:See KIP [here|https://cwiki.apache.org/confluence/display/KAFKA/KIP-168%3A+Add+TotalTopicCount+metric+per+cluster] > KIP-168: Add TotalTopicCount metric per cluster > --- > > Key: KAFKA-5461 > URL: https://issues.apache.org/jira/browse/KAFKA-5461 > Project: Kafka > Issue Type: Improvement >Reporter: Abhishek Mendhekar >Assignee: Abhishek Mendhekar > Fix For: 0.11.0.1 > > > See KIP > [here|https://cwiki.apache.org/confluence/display/KAFKA/KIP-168%3A+Add+TotalTopicCount+metric+per+cluster] > Email discussion > [here|http://mail-archives.apache.org/mod_mbox/kafka-dev/201706.mbox/%3CCAMcwe-ugep-UiSn9TkKEMwwTM%3DAzGC4jPro9LnyYRezyZg_NKA%40mail.gmail.com%3E] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
