[
https://issues.apache.org/jira/browse/KAFKA-1430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13990829#comment-13990829
]
Jay Kreps commented on KAFKA-1430:
----------------------------------
A couple of random thoughts.
- Not sure if I have fully groked the implications of this.
- In general I feel we are not building on a firm foundation with the purgatory
code and we should somehow find a way to refactor the apis to make code that
uses it more readable. I think in the absence of this it is impossible to write
correct code because it is too hard to understand the flow of things. This is
somewhat a separate thing from this proposal, and could be a separate ticket or
effort.
- Do we actually need to do anything for requests not in the last segment? My
concern is that trying to maintain the full set of physical sizes is going to
be very error prone and will impact the recovery logic too. One thing about
these apis is that it is always acceptable to return early. So what if we
maintain physical information about the last segment which we maintain through
the append api and for all previous segments we just return immediately (after
all you will never wait on older segments).
> Purgatory redesign
> ------------------
>
> Key: KAFKA-1430
> URL: https://issues.apache.org/jira/browse/KAFKA-1430
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Affects Versions: 0.8.2
> Reporter: Jun Rao
>
> We have seen 2 main issues with the Purgatory.
> 1. There is no atomic checkAndWatch functionality. So, a client typically
> first checks whether a request is satisfied or not and then register the
> watcher. However, by the time the watcher is registered, the registered item
> could already be satisfied. This item won't be satisfied until the next
> update happens or the delayed time expires, which means the watched item
> could be delayed.
> 2. FetchRequestPurgatory doesn't quite work. This is because the current
> design tries to incrementally maintain the accumulated bytes ready for fetch.
> However, this is difficult since the right time to check whether a fetch (for
> regular consumer) request is satisfied is when the high watermark moves. At
> that point, it's hard to figure out how many bytes we should incrementally
> add to each pending fetch request.
> The problem has been reported in KAFKA-1150 and KAFKA-703.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
