Marc, thanks much for documenting the guts!
There is one correction for Fetch Request handling:
When is it satisfied?
The fetch size requested is reached - ie. the amount of data the consumer
wishes to receive in one response
Consumer configuration: *fetch.message.max.bytes*
As per the code:
I've just updated the wiki with most of your comments hoping to make it
more clear. I stole much of Joel's comment so all the credit goes to him if
it works! The line between documenting the code and describing it is always
thin. My intention was not to make anything official at first either,
Subject: Re: Purgatory
Excellent - thanks for putting that together! Will review it more carefully
tomorrow and suggest some minor edits if required.
On Thu, Nov 07, 2013 at 10:45:40PM -0500, Marc Labbe wrote:
I've just added a page for purgatory, feel free to comment/modify at will.
I hope I
to add a summary
at the beginning about what it is and how it may impact a user.
Regards,
Libo
-Original Message-
From: Joel Koshy [mailto:jjkosh...@gmail.com]
Sent: Friday, November 08, 2013 2:01 AM
To: users@kafka.apache.org
Subject: Re: Purgatory
Excellent - thanks for putting
: Joel Koshy [mailto:jjkosh...@gmail.com]
Sent: Friday, November 08, 2013 2:01 AM
To: users@kafka.apache.org
Subject: Re: Purgatory
Excellent - thanks for putting that together! Will review it more
carefully tomorrow and suggest some minor edits if required.
On Thu, Nov 07, 2013 at 10:45
Marc - thanks again for doing this. Couple of suggestions:
- I would suggest removing the disclaimer and email quotes since this
can become a stand-alone clean document on what the purgatory is and
how it works.
- A diagram would be helpful - it could say, show the watcher map and
the
I've just added a page for purgatory, feel free to comment/modify at will.
I hope I didn't misinterpret too much of the code.
https://cwiki.apache.org/confluence/display/KAFKA/Request+Purgatory+(0.8)
I added a few questions of my own.
On Fri, Nov 1, 2013 at 9:43 PM, Joe Stein
Excellent - thanks for putting that together! Will review it more
carefully tomorrow and suggest some minor edits if required.
On Thu, Nov 07, 2013 at 10:45:40PM -0500, Marc Labbe wrote:
I've just added a page for purgatory, feel free to comment/modify at will.
I hope I didn't misinterpret too
Guozhang,
The documentation is not very clear.
Marc's response for producer purgatory makes sense.
I am not entirely clear on fetch purgatory.
How does broker use purgatory? Is it a temporary holding area? What happens
to the messages if purge interval is exceeded in case of either/both
producer
Priya, if you want you can look at RequestPurgatory.scala for some more
details.
The config is the size of the atomic requestCounter.
Basically the purge in the purgatory is a way to check if the request has
been satisfied and delayed and can get removed. It is a background scan
when the size
Guozhang,
I have to agree with Priya the doc isn't very clear. Although the
configuration is documented, it is simply rewording the name of the config,
which isn't particularly useful if you want more information about what the
purgatory is. I searched the whole wiki and doc and could not find
Marc, thanks for writing that up. I think it is worth adding some
details on the request-purgatory on a wiki (Jay had started a wiki
page for kafka internals [1] a while ago, but we have not had time to
add much to it since.) Your write-up could be reviewed and added
there. Do you have edit
Hi Priya
my understanding is producer requests will be delayed (and put in request
purgatory) only if your producer uses ack=-1. It will be in the purgatory
(delayed) until all brokers have acknowledged the messages to be
replicated. The documentation suggests to monitor the
Hello Priya,
You can find the definitions of these two configs here:
http://kafka.apache.org/documentation.html#brokerconfigs
Guozhang
On Thu, Oct 31, 2013 at 11:20 AM, Marc Labbe mrla...@gmail.com wrote:
Hi Priya
my understanding is producer requests will be delayed (and put in request
14 matches
Mail list logo
