[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16800656#comment-16800656 ] Spyridon Ninos commented on KAFKA-3539: --- Hi guys, any solutions proposed? I've hit a similar issue with [~tu...@avast.com] too, but by studying the code I am not confident that any solution will be that much better than the current one, either semantically or technically. Having said that, some weeks ago I took a look at how to solve the blocking nature of the producer - I'd like to know what others have thought as probable solutions. Any suggestions? Thanks > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798792#comment-16798792 ] Spyridon Ninos commented on KAFKA-3539: --- Hi [~tu...@avast.com], indeed the producer does not comply with the Future contract, however I'm not sure that this should be considered critical because it violates the contract only when the system has issues (i.e. the producer cannot communicate with the cluster). Should we modify the producer to strictly comply with the Future contract under any and all circumstances, we may hit a practical wall when it comes to code complexity. One rough analysis showed that in order for the user thread not to block, we should at least remove the partitioner from the user thread and push the functionality somewhere between the user thread and the sender thread (or, worse, the sender thread itself). Unblocking the user thread when the sender thread cannot send the messages to kafka, means that the messages should be stored somewhere locally (i.e. producer side) until the sender thread can push them to the cluster. This, in turn, changes the memory requirements of the producer, and increases the risk to lose more messages than the amount you can lose with the current implementation, in case the producer dies unexpectedly. I understand the concern people have about the contract - the question is, though, do we really need the Future contract at all? What are your thoughts on it? As a side note, my comment targeted mainly the status of the ticket; since no work is done currently, having it at the "in progress" status is misleading. Thanks > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794322#comment-16794322 ] Spyridon Ninos commented on KAFKA-3539: --- Hi [~ozhurakousky], given that the PR you created for this change is closed, and it seems that there is no intention to work on the idea, should we also close the ticket? Thanks > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
