Hi Nathan, I have a couple of remarks/questions about your mail, if I may.
First of all, the javadoc for the pause operation of KafkaConsumer states: "Suspend fetching from the requested partitions. Future calls to poll(Duration) <https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll-java.time.Duration-> will not return any records from these partitions until they have been resumed using resume(Collection) <https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#resume-java.util.Collection->. Note that this method does not affect partition subscription. In particular, it does not cause a group rebalance when automatic assignment is used." [1] You mentioned that "those operations" cause a rebalance, can you perhaps elaborate on that some more? Second, you state that "it would try to finish the current transactions", which confuses me a little as well, since the consumer is not really aware of transactions in a meaningful way. Or does "transaction" in this case refer to your last call to poll()? Have you looked into splitting your subscription across two consumers, one for high priority topics, one for low(er) priority topics? Unless you are looking for a dynamic, multi-tier priority system across many topics, that might be your best bet. This works quite well for scenarios where you have one topic that acts as a control plane (think start,stop processing type of messages) and a second topic contains the actual data. Best regards, Sönke [1] https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#pause-java.util.Collection- On Sun, Mar 24, 2019 at 2:41 AM nathank...@gmail.com <nathank...@gmail.com> wrote: > > > On 2019/01/28 02:26:31, n...@afshartous.com wrote: > > Hi Sönke, > > > > Thanks for taking the time to review. I’ve put KIP-349 into > hibernation. > > > > Thanks also to everyone who participated in the discussion. > > > > Best regards, > > -- > > Nick > > > > > On Jan 25, 2019, at 5:51 AM, Sönke Liebau > > > <soenke.lie...@opencore.com.INVALID> > wrote: > > > > > > a bit late to the party, sorry. I recently spent some time looking > > > into this / a similar issue [1]. > > > After some investigation and playing around with settings I think that > > > the benefit that could be gained from this is somewhat limited and > > > probably outweighed by the implementation effort. > > > > > > The consumer internal are already geared towards treating partitions > > > fairly so that no partition has to wait an undue amount of time and > > > this can be further tuned for latency over throughput. Additionally, > > > if this is a large issue for someone, there is always the option of > > > having a dedicated consumer reading only from the control topic, which > > > would mean that messages from that topic are received "immediately". > > > For a Kafka Streams job it would probably make sense to create two > > > input streams and then merging those as a first step. > > > > > > I think with these knobs a fairly large amount of flexibility can be > > > achieved so that there is no urgent need to implement priorities. > > > > > > So my personal preference would be to set this KIP to dormant for now. > > > > > > > > > > > > > Hello Nick, > > I'm extremely new to Kafka, but I was attempting to set up a per-topic > priority application, and ended up finding this thread. I'm having > difficulty seeing how one can implement it with pause/resume. Would you > elaborate? > > Since those operations are per-partition, and when you stop a partition, > it attempts to re-balance, I would need to stop all partitions. Even then, > it would try to finish the current transactions instead of immediately > putting it on hold and processing other topics. > > It also looks like in order to determine if I had received messages from > the pri-1 topic, I would need to loop through all records, and ignore those > that weren't pri-1 until a poll failed to retrieve any, which seems like it > would screw up the other topics. > > Thank you, > > Nathan > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
