[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17383368#comment-17383368 ] Moses Nakamura commented on KAFKA-3539: --- Looks like KafkaProducerTest is the main one that breaks. Right now there are only two failing tests . . . KafkaProducerTest > shouldCloseProperlyAndThrowIfInterrupted() FAILED KafkaProducerTest > testFlushCompleteSendOfInflightBatches() FAILED I don't think I've made any changes to the public API, it's only behavioral changes so far. They're runtime failures, I was able to fix any compilation issues pretty easily. Do you know who someone more active in this part of the codebase is? > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > Labels: needs-discussion, needs-kip > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380730#comment-17380730 ] Ewen Cheslack-Postava commented on KAFKA-3539: -- [~moses.nakamura] I spent a bunch of time on the clients in the past, but I've barely been involved for the past few years. So not really even sure of the current state of implementation and tests (e.g. I wasn't working on the clients when EoS was implemented). What tests fail due to minor changes? If they are unit tests, that should be unexpected unless you are changing public API, which would require a KIP anyway. You might also just be seeing flakiness in integration tests, which unfortunately is expected. Providing a list of the tests that break and whether it's compilation or runtime issues would probably help, but someone more active can probably provide better guidance. > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > Labels: needs-discussion, needs-kip > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380148#comment-17380148 ] Moses Nakamura commented on KAFKA-3539: --- So I know what I want to do and I've been trying to put together a prototype in advance of updating the KIP document, but even relatively modest changes seem to break a lot of tests. [~ewencp] are you an expert in KafkaProducer? Would you mind taking some time to chat and give me advice on how to proceed? > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > Labels: needs-discussion, needs-kip > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17353465#comment-17353465 ] Moses Nakamura commented on KAFKA-3539: --- I've gotten lots of good feedback on the discussion thread so far. [~cmccabe] mentioned that there is a "fix" for the metadata case that means we don't need to block, queue, or fail fast. That sounds better, so I'd like to know more about that, but I think we have enough information to start winnowing down the KIP to a more specific recommendation once I understand that idea better. > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > Labels: needs-discussion, needs-kip > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344023#comment-17344023 ] Moses Nakamura commented on KAFKA-3539: --- [~ewencp] thanks, I started the KIP process, it's [KIP-739|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181306446] and the discussion thread is [here|https://lists.apache.org/x/thread.html/r0cc154d47903bc9f905bce790c4a492761d7cc54076b1c320c68f122@%3Cdev.kafka.apache.org%3E]. Please let me know what you think! > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > Labels: needs-discussion, needs-kip > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343043#comment-17343043 ] Ewen Cheslack-Postava commented on KAFKA-3539: -- [~mjsax] They aren't really duplicates. It's possible to also just have races between whatever creates a topic and a process that tries to produce to it. I think 3450 implicitly assumes a simple sequential model where create should complete, and only then processes are deployed. In practice, there are lots of convergence based systems that can do without this with the right preflight checks (topic creation may happen in parallel to deploying an app that uses it, but does not have permissions to create topics). They may be related, but at least the high level description of 3450 wrt topic auto creation doesn't quite align with this one. [~moses.nakamura] this is a very tricky thing to get right. At first glance we should make anything blocking (or with simple timeout) just be tied to the producer constructor, but at that point we don't know what info to get. Adding a queue can help, but adds overhead and really just delays the issue. Configurable timeouts allow you to set your tolerance for blocking operations (you're going to spend at least some time in the send() call anyway). Alternatively, we could not wait on metadata at all and only check if it's available, request it if it is not, and then bail with a RetriableException. Pretty much any solution proposed so far is going to need a KIP (it's at a minimum a behavioral change, for your option 1 it's an API addition). A good starting point if you want to address this is to enumerate the variety of options and their pros/cons/compatibility issues. > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > Labels: needs-discussion, needs-kip > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17342739#comment-17342739 ] Moses Nakamura commented on KAFKA-3539: --- We've been wrestling with this problem for a while. I think my company would like to start the process to try to patch it. However, it's definitely thorny, so we had some issues we wanted to talk about! The main questions seem to be: # Which thread should wait for the partition metadata? # How can we preserve the ordering semantics that send gives us today? I think the ideal answer to (1) would be, "no thread" and that we should do it asynchronously. This would be a pretty invasive change though, so I think it's probably more expedient to pass in a threadpool and do it there. How would you feel about optionally passing in a threadpool to KafkaProducer#send, or else on construction to KafkaProducer? For (2), it seems like we have two plausible approaches, either pass async data to the RecordAccumulator, or else layer another queue on top. I think layering another queue on top would be less invasive, so I would lean toward that approach. We could have a central queue that orders all of the work, conditioned on the metadata that we're waiting for being ready. Does that sound good to you? Should I put up a patch so we can talk about it? > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > Labels: needs-discussion, needs-kip > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013111#comment-17013111 ] Matthias J. Sax commented on KAFKA-3539: Seems KAFKA-3450 is a duplicate? > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > Labels: needs-discussion, needs-kip > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921598#comment-16921598 ] alex gabriel commented on KAFKA-3539: - [~tu...@avast.com] [~stevenz3wu] I think that you can avoid additional ExecutorService creation to make #send fully non-blocking by specifying max.block.ms to 0. But you still need to catch delivery exception until metadata comes. In your current solutions(if I got it right) you still have the chance to lose all the events that were allocated inside your ExecutorService queue( that is not persistent) since you only add events to the persistence storage only after rejection exceptions. > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > Labels: needs-discussion, needs-kip > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852428#comment-16852428 ] Richard Yu commented on KAFKA-3539: --- [~radai] I don't know if this problem is resolved or not. Issue 6705 was closed because it was thought any changes made would be too complex to fix this behavioral issue. > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > Labels: needs-discussion, needs-kip > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16800714#comment-16800714 ] radai rosenblatt commented on KAFKA-3539: - IIUC, the root of the problem is the kafka producer stores compressed batches of msgs, in a map keyed by the partition these msgs are intended for. since without metadata there's no knowing the layout of a topic the producer cant tell where to "place" a msg, which is why it blocks on no metadata. one possible solution would be to have an "unknown" msg bucket (with some finite capacity) where msgs of unknown destination go. the biggest issue with this is that those msgs cannot be compressed (as kafka compresses batches, not individual msgs, and there's no guarantee that everything in the unknown bucket will go into the same batch). once metadata is obtained the "unknown bucket" would need to be iterated over, and the msgs deposited (and compressed) into the correct queues. this would need to happen when metadata arrives and before any new msgs are allowed into the producer (to not violate order) > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16800656#comment-16800656 ] Spyridon Ninos commented on KAFKA-3539: --- Hi guys, any solutions proposed? I've hit a similar issue with [~tu...@avast.com] too, but by studying the code I am not confident that any solution will be that much better than the current one, either semantically or technically. Having said that, some weeks ago I took a look at how to solve the blocking nature of the producer - I'd like to know what others have thought as probable solutions. Any suggestions? Thanks > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799776#comment-16799776 ] Steven Zhen Wu commented on KAFKA-3539: --- just to echo last two comments. There are many applications that just want best effort delivery to kafka and can't tolerate blocking behavior at all. Now many people have to re-invent the wheels to work around this problem. E.g. we implemented exactly the same things that [~tu...@avast.com] mentioned. Improved documentation can help. But an intuitive API is definitely much better. > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798975#comment-16798975 ] Christophe Jolif commented on KAFKA-3539: - I think the main issue, is that reading the API people think they can write code that won't block... So when it suddenly block you realize all your assumptions were wrong and you might end up blocking critical code that should be running. And you usually just learn in the hard way during pre-production tests (ideally) or worst... At the very least the documentation should be clearer because when you read it you don't imagine this can block. > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[
https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798916#comment-16798916
]
Michal Turek commented on KAFKA-3539:
-
Hi [~spyridon.ninos],
I have never said the fix will be simple and I fully understand you. I saw this
code while debugging the issue and other related ones and agree it will be
challenging to fix the producer correctly and backward compatible. Also your
original message I was commenting spoke not about "in progress" status but
about the ticket closing with no action. If you would change the "in progress"
status to "open" I wouldn't say anything.
To describe our use case and the workarounds... We have REST-like services with
high request rates (thousands or tens thousands req/s) implemented async way to
be as effective as possible. Blocking on any level would cause full blocking
the whole application including HTTP handler threads. If the threads are
blocked, it's even impossible to return 500 Internal server error to the
clients, there are only TCP/IP timeouts of the connections.
- We use home-made {{class NonBlockingKafkaProducer implements
Producer}} wrapper that writes to std. producer through ExecutorService
that immediately rejects tasks if producer blocks all the provided threads and
the queue in executor is full (async non-blocking API). The configured buffer
can handle blocking that takes max. few seconds.
- Unrelated to this ticket: This wrapper is wrapped by second one
{{PersistingKafkaProducer implements Producer}} (also home-made)
that is able to persist the failed events to a local disk and re-produce them
after Kafka is available again - for unexpected longer outages.
- There are also issues during *every* application startup. Kafka producer
reads topics metadata lazily on the first incoming message... and blocks the
callers threads. Every application startup is not a rare situation and one can
consider critical to behave correctly during the whole process life cycle. The
I/O operation to get the metadata takes some time, the first N requests
immediately block all N threads. We use *Kafka consumer API* get list of topics
and warm-up *producer* before binding the listening socket and making the
instance active.
- In special cases we also manage and periodically update a white list of
topics that exist in Kafka based on the aforementioned consumer API to prevent
writes to the non-existing ones. Each such write again triggers metadata update
and blocks that thread. The white list doesn't solve deletion of a topic, but
we typically don't delete.
- Are we really so special or are these issues common in the whole industry? We
built this know how about Kafka during last five years - mainly during
production issues. I remember we started with Kafka 0.8.1.something.
My 10 cents...
> KafkaProducer.send() may block even though it returns the Future
>
>
> Key: KAFKA-3539
> URL: https://issues.apache.org/jira/browse/KAFKA-3539
> Project: Kafka
> Issue Type: Bug
> Components: producer
>Reporter: Oleg Zhurakousky
>Priority: Critical
>
> You can get more details from the us...@kafka.apache.org by searching on the
> thread with the subject "KafkaProducer block on send".
> The bottom line is that method that returns Future must never block, since it
> essentially violates the Future contract as it was specifically designed to
> return immediately passing control back to the user to check for completion,
> cancel etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798792#comment-16798792 ] Spyridon Ninos commented on KAFKA-3539: --- Hi [~tu...@avast.com], indeed the producer does not comply with the Future contract, however I'm not sure that this should be considered critical because it violates the contract only when the system has issues (i.e. the producer cannot communicate with the cluster). Should we modify the producer to strictly comply with the Future contract under any and all circumstances, we may hit a practical wall when it comes to code complexity. One rough analysis showed that in order for the user thread not to block, we should at least remove the partitioner from the user thread and push the functionality somewhere between the user thread and the sender thread (or, worse, the sender thread itself). Unblocking the user thread when the sender thread cannot send the messages to kafka, means that the messages should be stored somewhere locally (i.e. producer side) until the sender thread can push them to the cluster. This, in turn, changes the memory requirements of the producer, and increases the risk to lose more messages than the amount you can lose with the current implementation, in case the producer dies unexpectedly. I understand the concern people have about the contract - the question is, though, do we really need the Future contract at all? What are your thoughts on it? As a side note, my comment targeted mainly the status of the ticket; since no work is done currently, having it at the "in progress" status is misleading. Thanks > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794810#comment-16794810 ] Michal Turek commented on KAFKA-3539: - Hi [~spyridon.ninos], ticket close won't fix the critical bug, so please no. Our code base is full of multiple ugly workarounds that allow us live with this broken behavior of Kafka producer without daily headache, but the situation isn't good. Everyone with little higher request rate will soon or later surprisingly hit this "Future must never block, since it essentially violates the Future contract" bug. The producer implementation is just broken. > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16794322#comment-16794322 ] Spyridon Ninos commented on KAFKA-3539: --- Hi [~ozhurakousky], given that the PR you created for this change is closed, and it seems that there is no intention to work on the idea, should we also close the ticket? Thanks > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Oleg Zhurakousky >Priority: Critical > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-3539) KafkaProducer.send() may block even though it returns the Future
[ https://issues.apache.org/jira/browse/KAFKA-3539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088823#comment-16088823 ] ASF GitHub Bot commented on KAFKA-3539: --- Github user evis closed the pull request at: https://github.com/apache/kafka/pull/3478 > KafkaProducer.send() may block even though it returns the Future > > > Key: KAFKA-3539 > URL: https://issues.apache.org/jira/browse/KAFKA-3539 > Project: Kafka > Issue Type: Bug >Reporter: Oleg Zhurakousky >Assignee: Evgeny Veretennikov >Priority: Critical > > You can get more details from the us...@kafka.apache.org by searching on the > thread with the subject "KafkaProducer block on send". > The bottom line is that method that returns Future must never block, since it > essentially violates the Future contract as it was specifically designed to > return immediately passing control back to the user to check for completion, > cancel etc. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
