Re: SessionKeySchema#segmentsToSearch()

[Damian Guy]

w.r.t `SessionKeySchema` MAX_VALUE is correct. This is because we use the
end time (rather than the start time) of the session to define which
segment the session is in. So it could be in any segment starting from
`from` time.

On Sun, 19 Nov 2017 at 20:27 Ted Yu <yuzhih...@gmail.com> wrote:

> For `getMinSegmentGreaterThanEqualToTimestamp` , the email was indeed
> meant for #4162.
>
> Pardon.
>
> On Sun, Nov 19, 2017 at 11:55 AM, Guozhang Wang <wangg...@gmail.com>
> wrote:
>
>> For `SessionKeySchema#segmentsToSearch`: for session store, multiple
>> sessions may merge together when receiving late arrived records. When I
>> looked at the code, it seems that we have merged the sessions during
>> aggregations to effectively move the sessions between segments. So I'm not
>> 100% certain why we still need to enforce MAX_VALUE. @Damian?
>>
>> For `getMinSegmentGreaterThanEqualToTimestamp` and `
>> getMaxSegmentLessThanEqualToTimestamp`: I think you meant to leave it as a
>> comment on https://github.com/apache/kafka/pull/4162? This is only added
>> in
>> that PR.
>>
>>
>> Guozhang
>>
>>
>> On Sat, Nov 18, 2017 at 11:16 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>> >
>> > This code:
>> >
>> > final Segment minSegment = segments
>> > .getMinSegmentGreaterThanEqualToTimestamp(timeFrom);
>> >
>> > final Segment maxSegment = segments
>> > .getMaxSegmentLessThanEqualToTimestamp(timeTo);
>> >
>> > Can be replaced with:
>> >
>> > final List searchSpace = keySchema.segmentsToSearch(
>> > segments, from, to);
>> >
>> > The minSegment would be first in List and maxSegment would be last in
>> List.
>> >
>> > On Sat, Nov 18, 2017 at 11:09 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>> >
>> > > Hi,
>> > > I was reading code for SessionKeySchema#segmentsToSearch() where:
>> > >
>> > > public List segmentsToSearch(final Segments segments,
>> final
>> > > long from, final long to) {
>> > > return segments.segments(from, Long.MAX_VALUE);
>> > >
>> > > I wonder why the parameter to is ignored.
>> > > WindowKeySchema#segmentsToSearch() passes parameter to
>> > > to segments.segments().
>> > >
>> > > Cheers
>> > >
>>
>>
>>
>>
>> --
>> -- Guozhang
>>
>
>

Re: SessionKeySchema#segmentsToSearch()

[Ted Yu]

For `getMinSegmentGreaterThanEqualToTimestamp` , the email was indeed meant
for #4162.

Pardon.

On Sun, Nov 19, 2017 at 11:55 AM, Guozhang Wang <wangg...@gmail.com> wrote:

> For `SessionKeySchema#segmentsToSearch`: for session store, multiple
> sessions may merge together when receiving late arrived records. When I
> looked at the code, it seems that we have merged the sessions during
> aggregations to effectively move the sessions between segments. So I'm not
> 100% certain why we still need to enforce MAX_VALUE. @Damian?
>
> For `getMinSegmentGreaterThanEqualToTimestamp` and `
> getMaxSegmentLessThanEqualToTimestamp`: I think you meant to leave it as a
> comment on https://github.com/apache/kafka/pull/4162? This is only added
> in
> that PR.
>
>
> Guozhang
>
>
> On Sat, Nov 18, 2017 at 11:16 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
> > This code:
> >
> > final Segment minSegment = segments
> > .getMinSegmentGreaterThanEqualToTimestamp(timeFrom);
> >
> > final Segment maxSegment = segments
> > .getMaxSegmentLessThanEqualToTimestamp(timeTo);
> >
> > Can be replaced with:
> >
> > final List searchSpace = keySchema.segmentsToSearch(
> > segments, from, to);
> >
> > The minSegment would be first in List and maxSegment would be last in
> List.
> >
> > On Sat, Nov 18, 2017 at 11:09 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> >
> > > Hi,
> > > I was reading code for SessionKeySchema#segmentsToSearch() where:
> > >
> > > public List segmentsToSearch(final Segments segments,
> final
> > > long from, final long to) {
> > > return segments.segments(from, Long.MAX_VALUE);
> > >
> > > I wonder why the parameter to is ignored.
> > > WindowKeySchema#segmentsToSearch() passes parameter to
> > > to segments.segments().
> > >
> > > Cheers
> > >
>
>
>
>
> --
> -- Guozhang
>

Re: SessionKeySchema#segmentsToSearch()

[Guozhang Wang]

For `SessionKeySchema#segmentsToSearch`: for session store, multiple
sessions may merge together when receiving late arrived records. When I
looked at the code, it seems that we have merged the sessions during
aggregations to effectively move the sessions between segments. So I'm not
100% certain why we still need to enforce MAX_VALUE. @Damian?

For `getMinSegmentGreaterThanEqualToTimestamp` and `
getMaxSegmentLessThanEqualToTimestamp`: I think you meant to leave it as a
comment on https://github.com/apache/kafka/pull/4162? This is only added in
that PR.


Guozhang


On Sat, Nov 18, 2017 at 11:16 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> This code:
>
> final Segment minSegment = segments
> .getMinSegmentGreaterThanEqualToTimestamp(timeFrom);
>
> final Segment maxSegment = segments
> .getMaxSegmentLessThanEqualToTimestamp(timeTo);
>
> Can be replaced with:
>
> final List searchSpace = keySchema.segmentsToSearch(
> segments, from, to);
>
> The minSegment would be first in List and maxSegment would be last in
List.
>
> On Sat, Nov 18, 2017 at 11:09 AM, Ted Yu <yuzhih...@gmail.com> wrote:
>
> > Hi,
> > I was reading code for SessionKeySchema#segmentsToSearch() where:
> >
> > public List segmentsToSearch(final Segments segments, final
> > long from, final long to) {
> > return segments.segments(from, Long.MAX_VALUE);
> >
> > I wonder why the parameter to is ignored.
> > WindowKeySchema#segmentsToSearch() passes parameter to
> > to segments.segments().
> >
> > Cheers
> >




--
-- Guozhang

Re: SessionKeySchema#segmentsToSearch()

[Ted Yu]

This code:

final Segment minSegment = segments
.getMinSegmentGreaterThanEqualToTimestamp(timeFrom);

final Segment maxSegment = segments
.getMaxSegmentLessThanEqualToTimestamp(timeTo);

Can be replaced with:

final List searchSpace = keySchema.segmentsToSearch(
segments, from, to);

The minSegment would be first in List and maxSegment would be last in List.

On Sat, Nov 18, 2017 at 11:09 AM, Ted Yu <yuzhih...@gmail.com> wrote:

> Hi,
> I was reading code for SessionKeySchema#segmentsToSearch() where:
>
> public List segmentsToSearch(final Segments segments, final
> long from, final long to) {
> return segments.segments(from, Long.MAX_VALUE);
>
> I wonder why the parameter to is ignored.
> WindowKeySchema#segmentsToSearch() passes parameter to
> to segments.segments().
>
> Cheers
>

SessionKeySchema#segmentsToSearch()

[Ted Yu]

Hi,
I was reading code for SessionKeySchema#segmentsToSearch() where:

public List segmentsToSearch(final Segments segments, final
long from, final long to) {
return segments.segments(from, Long.MAX_VALUE);

I wonder why the parameter to is ignored.
WindowKeySchema#segmentsToSearch() passes parameter to
to segments.segments().

Cheers