For testing purposes, we previously used a Set to record the results in IQv2StoreIntegrationTest. Let's take an example where we now have two partitions and four key-value pairs: <0,0> in p0, <1,1> in p1, <2,2> in p0, and <3,3> in p1.
If we execute withRange(1,3), it will return a Set of <1, 2, 3>. However, if we run withRange(1,3).withDescendingKeys(), and still use a Set, the result will again be a Set of <1,2,3>. This means we won't be able to determine whether the results have been reversed. To resolve this ambiguity, I've switched to using a List to record the results, ensuring the order of retrieval from partitions p0 and p1. So, withRange(1,3) would yield a List of [2, 1, 3], whereas withRange(1,3).withDescendingKeys() would produce a List of [2,3,1]. This ordering makes sense since RocksDB sorts its keys, and InMemoryStore uses a TreeMap structure, which means the keys are already sorted. Sincerely, Hanyu On Wed, Oct 4, 2023 at 9:25 AM Hanyu (Peter) Zheng <pzh...@confluent.io> wrote: > Hi, Bruno > > Thank you for your suggestions, I will update them soon. > Sincerely, > > Hanyu > > On Wed, Oct 4, 2023 at 9:25 AM Hanyu (Peter) Zheng <pzh...@confluent.io> > wrote: > >> Hi, Lucas, >> >> Thank you for your suggestions. >> I will update the KIP and code together. >> >> Sincerely, >> Hanyu >> >> On Tue, Oct 3, 2023 at 8:16 PM Hanyu (Peter) Zheng <pzh...@confluent.io> >> wrote: >> >>> If we use WithDescendingKeys() to generate a RangeQuery to do the >>> reveseQuery, how do we achieve the methods like withRange, withUpperBound, >>> and withLowerBound only in this method? >>> >>> On Tue, Oct 3, 2023 at 8:01 PM Hanyu (Peter) Zheng <pzh...@confluent.io> >>> wrote: >>> >>>> I believe there's no need to introduce a method like >>>> WithDescendingKeys(). Instead, we can simply add a reverse flag to >>>> RangeQuery. Each method within RangeQuery would then accept an additional >>>> parameter. If the reverse is set to true, it would indicate the results >>>> should be reversed. >>>> >>>> Initially, I introduced a reverse variable. When set to false, the >>>> RangeQuery class behaves normally. However, when reverse is set to true, >>>> the RangeQuery essentially takes on the functionality of ReverseRangeQuery. >>>> Further details can be found in the "Rejected Alternatives" section. >>>> >>>> In my perspective, RangeQuery is a class responsible for creating a >>>> series of RangeQuery objects. It offers methods such as withRange, >>>> withUpperBound, and withLowerBound, allowing us to generate objects >>>> representing different queries. I'm unsure how adding a >>>> withDescendingOrder() method would be compatible with the other methods, >>>> especially considering that, based on KIP 969, WithDescendingKeys() doesn't >>>> appear to take any input variables. And if withDescendingOrder() doesn't >>>> accept any input, how does it return a RangeQuery? >>>> >>>> On Tue, Oct 3, 2023 at 4:37 PM Hanyu (Peter) Zheng <pzh...@confluent.io> >>>> wrote: >>>> >>>>> Hi, Colt, >>>>> The underlying structure of inMemoryKeyValueStore is treeMap. >>>>> Sincerely, >>>>> Hanyu >>>>> >>>>> On Tue, Oct 3, 2023 at 4:34 PM Hanyu (Peter) Zheng < >>>>> pzh...@confluent.io> wrote: >>>>> >>>>>> Hi Bill, >>>>>> 1. I will update the KIP in accordance with the PR and synchronize >>>>>> their future updates. >>>>>> 2. I will use that name. >>>>>> 3. you mean add something about ordering at the motivation section? >>>>>> >>>>>> Sincerely, >>>>>> Hanyu >>>>>> >>>>>> >>>>>> On Tue, Oct 3, 2023 at 4:29 PM Hanyu (Peter) Zheng < >>>>>> pzh...@confluent.io> wrote: >>>>>> >>>>>>> Hi, Walker, >>>>>>> >>>>>>> 1. I will update the KIP in accordance with the PR and synchronize >>>>>>> their future updates. >>>>>>> 2. I will use that name. >>>>>>> 3. I'll provide additional details in that section. >>>>>>> 4. I intend to utilize rangeQuery to achieve what we're referring to >>>>>>> as reverseQuery. In essence, reverseQuery is merely a term. To clear up >>>>>>> any >>>>>>> ambiguity, I'll make necessary adjustments to the KIP. >>>>>>> >>>>>>> Sincerely, >>>>>>> Hanyu >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Tue, Oct 3, 2023 at 4:09 PM Hanyu (Peter) Zheng < >>>>>>> pzh...@confluent.io> wrote: >>>>>>> >>>>>>>> Ok, I will change it back to following the code, and update them >>>>>>>> together. >>>>>>>> >>>>>>>> On Tue, Oct 3, 2023 at 2:27 PM Walker Carlson >>>>>>>> <wcarl...@confluent.io.invalid> wrote: >>>>>>>> >>>>>>>>> Hello Hanyu, >>>>>>>>> >>>>>>>>> Looking over your kip things mostly make sense but I have a couple >>>>>>>>> of >>>>>>>>> comments. >>>>>>>>> >>>>>>>>> >>>>>>>>> 1. You have "withDescandingOrder()". I think you mean >>>>>>>>> "descending" :) >>>>>>>>> Also there are still a few places in the do where its called >>>>>>>>> "setReverse" >>>>>>>>> 2. Also I like "WithDescendingKeys()" better >>>>>>>>> 3. I'm not sure of what ordering guarantees we are offering. >>>>>>>>> Perhaps we >>>>>>>>> can add a section to the motivation clearly spelling out the >>>>>>>>> current >>>>>>>>> ordering and the new offering? >>>>>>>>> 4. When you say "use unbounded reverseQuery to achieve >>>>>>>>> reverseAll" do >>>>>>>>> you mean "use unbounded RangeQuery to achieve reverseAll"? as >>>>>>>>> far as I can >>>>>>>>> tell we don't have a reverseQuery as a named object? >>>>>>>>> >>>>>>>>> >>>>>>>>> Looking good so far >>>>>>>>> >>>>>>>>> best, >>>>>>>>> Walker >>>>>>>>> >>>>>>>>> On Tue, Oct 3, 2023 at 2:13 PM Colt McNealy <c...@littlehorse.io> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>> > Hello Hanyu, >>>>>>>>> > >>>>>>>>> > Thank you for the KIP. I agree with Matthias' proposal to keep >>>>>>>>> the naming >>>>>>>>> > convention consistent with KIP-969. I favor the >>>>>>>>> `.withDescendingKeys()` >>>>>>>>> > name. >>>>>>>>> > >>>>>>>>> > I am curious about one thing. RocksDB guarantees that records >>>>>>>>> returned >>>>>>>>> > during a range scan are lexicographically ordered by the bytes >>>>>>>>> of the keys >>>>>>>>> > (either ascending or descending order, as specified in the >>>>>>>>> query). This >>>>>>>>> > means that results within a single partition are indeed >>>>>>>>> ordered.** My >>>>>>>>> > reading of KIP-805 suggests to me that you don't need to specify >>>>>>>>> the >>>>>>>>> > partition number you are querying in IQv2, which means that you >>>>>>>>> can have a >>>>>>>>> > valid reversed RangeQuery over a store with "multiple >>>>>>>>> partitions" in it. >>>>>>>>> > >>>>>>>>> > Currently, IQv1 does not guarantee order of keys in this >>>>>>>>> scenario. Does >>>>>>>>> > IQv2 support ordering across partitions? Such an implementation >>>>>>>>> would >>>>>>>>> > require opening a rocksdb range scan** on multiple rocksdb >>>>>>>>> instances (one >>>>>>>>> > per partition), and polling the first key of each. Whether or >>>>>>>>> not this is >>>>>>>>> > ordered, could we please add that to the documentation? >>>>>>>>> > >>>>>>>>> > **(How is this implemented/guaranteed in an >>>>>>>>> `inMemoryKeyValueStore`? I >>>>>>>>> > don't know about that implementation). >>>>>>>>> > >>>>>>>>> > Colt McNealy >>>>>>>>> > >>>>>>>>> > *Founder, LittleHorse.dev* >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > On Tue, Oct 3, 2023 at 1:35 PM Hanyu (Peter) Zheng >>>>>>>>> > <pzh...@confluent.io.invalid> wrote: >>>>>>>>> > >>>>>>>>> > > ok, I will update it. Thank you Matthias >>>>>>>>> > > >>>>>>>>> > > Sincerely, >>>>>>>>> > > Hanyu >>>>>>>>> > > >>>>>>>>> > > On Tue, Oct 3, 2023 at 11:23 AM Matthias J. Sax < >>>>>>>>> mj...@apache.org> >>>>>>>>> > wrote: >>>>>>>>> > > >>>>>>>>> > > > Thanks for the KIP Hanyu! >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > I took a quick look and it think the proposal makes sense >>>>>>>>> overall. >>>>>>>>> > > > >>>>>>>>> > > > A few comments about how to structure the KIP. >>>>>>>>> > > > >>>>>>>>> > > > As you propose to not add `ReverseRangQuery` class, the code >>>>>>>>> example >>>>>>>>> > > > should go into "Rejected Alternatives" section, not in the >>>>>>>>> "Proposed >>>>>>>>> > > > Changes" section. >>>>>>>>> > > > >>>>>>>>> > > > For the `RangeQuery` code example, please omit all existing >>>>>>>>> methods >>>>>>>>> > etc, >>>>>>>>> > > > and only include what will be added/changed. This make it >>>>>>>>> simpler to >>>>>>>>> > > > read the KIP. >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > nit: typo >>>>>>>>> > > > >>>>>>>>> > > > > the fault value is false >>>>>>>>> > > > >>>>>>>>> > > > Should be "the default value is false". >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > Not sure if `setReverse()` is the best name. Maybe >>>>>>>>> > `withDescandingOrder` >>>>>>>>> > > > (or similar, I guess `withReverseOrder` would also work) >>>>>>>>> might be >>>>>>>>> > > > better? Would be good to align to KIP-969 proposal that >>>>>>>>> suggest do use >>>>>>>>> > > > `withDescendingKeys` methods for "reverse key-range"; if we >>>>>>>>> go with >>>>>>>>> > > > `withReverseOrder` we should change KIP-969 accordingly. >>>>>>>>> > > > >>>>>>>>> > > > Curious to hear what others think about naming this >>>>>>>>> consistently across >>>>>>>>> > > > both KIPs. >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > -Matthias >>>>>>>>> > > > >>>>>>>>> > > > >>>>>>>>> > > > On 10/3/23 9:17 AM, Hanyu (Peter) Zheng wrote: >>>>>>>>> > > > > >>>>>>>>> > > > >>>>>>>>> > > >>>>>>>>> > >>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-985%3A+Add+reverseRange+and+reverseAll+query+over+kv-store+in+IQv2 >>>>>>>>> > > > > >>>>>>>>> > > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > -- >>>>>>>>> > > >>>>>>>>> > > [image: Confluent] <https://www.confluent.io> >>>>>>>>> > > Hanyu (Peter) Zheng he/him/his >>>>>>>>> > > Software Engineer Intern >>>>>>>>> > > +1 (213) 431-7193 <+1+(213)+431-7193> >>>>>>>>> > > Follow us: [image: Blog] >>>>>>>>> > > < >>>>>>>>> > > >>>>>>>>> > >>>>>>>>> https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog >>>>>>>>> > > >[image: >>>>>>>>> > > Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] >>>>>>>>> > > <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] >>>>>>>>> > > <https://slackpass.io/confluentcommunity>[image: YouTube] >>>>>>>>> > > <https://youtube.com/confluent> >>>>>>>>> > > >>>>>>>>> > > [image: Try Confluent Cloud for Free] >>>>>>>>> > > < >>>>>>>>> > > >>>>>>>>> > >>>>>>>>> https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound&utm_source=gmail&utm_medium=organic >>>>>>>>> > > > >>>>>>>>> > > >>>>>>>>> > >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> [image: Confluent] <https://www.confluent.io> >>>>>>>> Hanyu (Peter) Zheng he/him/his >>>>>>>> Software Engineer Intern >>>>>>>> +1 (213) 431-7193 <+1+(213)+431-7193> >>>>>>>> Follow us: [image: Blog] >>>>>>>> <https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image: >>>>>>>> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] >>>>>>>> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] >>>>>>>> <https://slackpass.io/confluentcommunity>[image: YouTube] >>>>>>>> <https://youtube.com/confluent> >>>>>>>> >>>>>>>> [image: Try Confluent Cloud for Free] >>>>>>>> <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound&utm_source=gmail&utm_medium=organic> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> [image: Confluent] <https://www.confluent.io> >>>>>>> Hanyu (Peter) Zheng he/him/his >>>>>>> Software Engineer Intern >>>>>>> +1 (213) 431-7193 <+1+(213)+431-7193> >>>>>>> Follow us: [image: Blog] >>>>>>> <https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image: >>>>>>> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] >>>>>>> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] >>>>>>> <https://slackpass.io/confluentcommunity>[image: YouTube] >>>>>>> <https://youtube.com/confluent> >>>>>>> >>>>>>> [image: Try Confluent Cloud for Free] >>>>>>> <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound&utm_source=gmail&utm_medium=organic> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> [image: Confluent] <https://www.confluent.io> >>>>>> Hanyu (Peter) Zheng he/him/his >>>>>> Software Engineer Intern >>>>>> +1 (213) 431-7193 <+1+(213)+431-7193> >>>>>> Follow us: [image: Blog] >>>>>> <https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image: >>>>>> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] >>>>>> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] >>>>>> <https://slackpass.io/confluentcommunity>[image: YouTube] >>>>>> <https://youtube.com/confluent> >>>>>> >>>>>> [image: Try Confluent Cloud for Free] >>>>>> <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound&utm_source=gmail&utm_medium=organic> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> [image: Confluent] <https://www.confluent.io> >>>>> Hanyu (Peter) Zheng he/him/his >>>>> Software Engineer Intern >>>>> +1 (213) 431-7193 <+1+(213)+431-7193> >>>>> Follow us: [image: Blog] >>>>> <https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image: >>>>> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] >>>>> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] >>>>> <https://slackpass.io/confluentcommunity>[image: YouTube] >>>>> <https://youtube.com/confluent> >>>>> >>>>> [image: Try Confluent Cloud for Free] >>>>> <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound&utm_source=gmail&utm_medium=organic> >>>>> >>>> >>>> >>>> -- >>>> >>>> [image: Confluent] <https://www.confluent.io> >>>> Hanyu (Peter) Zheng he/him/his >>>> Software Engineer Intern >>>> +1 (213) 431-7193 <+1+(213)+431-7193> >>>> Follow us: [image: Blog] >>>> <https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image: >>>> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] >>>> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] >>>> <https://slackpass.io/confluentcommunity>[image: YouTube] >>>> <https://youtube.com/confluent> >>>> >>>> [image: Try Confluent Cloud for Free] >>>> <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound&utm_source=gmail&utm_medium=organic> >>>> >>> >>> >>> -- >>> >>> [image: Confluent] <https://www.confluent.io> >>> Hanyu (Peter) Zheng he/him/his >>> Software Engineer Intern >>> +1 (213) 431-7193 <+1+(213)+431-7193> >>> Follow us: [image: Blog] >>> <https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image: >>> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] >>> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] >>> <https://slackpass.io/confluentcommunity>[image: YouTube] >>> <https://youtube.com/confluent> >>> >>> [image: Try Confluent Cloud for Free] >>> <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound&utm_source=gmail&utm_medium=organic> >>> >> >> >> -- >> >> [image: Confluent] <https://www.confluent.io> >> Hanyu (Peter) Zheng he/him/his >> Software Engineer Intern >> +1 (213) 431-7193 <+1+(213)+431-7193> >> Follow us: [image: Blog] >> <https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image: >> Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] >> <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] >> <https://slackpass.io/confluentcommunity>[image: YouTube] >> <https://youtube.com/confluent> >> >> [image: Try Confluent Cloud for Free] >> <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound&utm_source=gmail&utm_medium=organic> >> > > > -- > > [image: Confluent] <https://www.confluent.io> > Hanyu (Peter) Zheng he/him/his > Software Engineer Intern > +1 (213) 431-7193 <+1+(213)+431-7193> > Follow us: [image: Blog] > <https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image: > Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] > <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] > <https://slackpass.io/confluentcommunity>[image: YouTube] > <https://youtube.com/confluent> > > [image: Try Confluent Cloud for Free] > <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound&utm_source=gmail&utm_medium=organic> > -- [image: Confluent] <https://www.confluent.io> Hanyu (Peter) Zheng he/him/his Software Engineer Intern +1 (213) 431-7193 <+1+(213)+431-7193> Follow us: [image: Blog] <https://www.confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog>[image: Twitter] <https://twitter.com/ConfluentInc>[image: LinkedIn] <https://www.linkedin.com/in/hanyu-peter-zheng/>[image: Slack] <https://slackpass.io/confluentcommunity>[image: YouTube] <https://youtube.com/confluent> [image: Try Confluent Cloud for Free] <https://www.confluent.io/get-started?utm_campaign=tm.fm-apac_cd.inbound&utm_source=gmail&utm_medium=organic>
