OK, I'll try On 9/13/16, ShaoFeng Shi <[email protected]> wrote: > sorry one correction: > > > " if some messages are arrived late than the margin, it will not be lost" > should be: > " if some messages are arrived late than the margin, it will be lost" > > Mario, you can try to set a bigger margin value to reduce the possibility. > > 2016-09-13 22:05 GMT+08:00 ShaoFeng Shi <[email protected]>: > >> In 1.5.x streaming OLAP, kylin uses a timestamp range to seek the >> start/end offset in kafka, which is binary search; It allows a margin >> window, but if some messages are arrived late than the margin, it will >> not >> be lost; >> >> Now we're working on a new implementation, which will strictly use offset >> to fetch the new messages each time, so there will not be message lost. >> >> >> 2016-09-13 15:53 GMT+08:00 Billy(Yiming) Liu <[email protected]>: >> >>> The current design is still an experimental approach. Kafka could not >>> guarantee the global order, so we have to find other solution. The new >>> design Streaming OLAP solution will relay on the Kafka partition order, >>> instead of app timestamp. The code is under KYLIN-1726 branch still. >>> >>> 2016-09-13 15:46 GMT+08:00 Mario Copperfield <[email protected]>: >>> >>> > OK, Thank you >>> > >>> > On Tue, Sep 13, 2016 at 3:27 PM, Sarnath K <[email protected]> wrote: >>> > >>> > > Yes, that's true. If you are looking at an app timestamp(event >>> origin >>> > > time), then We can't binary search on it. Though Binary search may >>> be a >>> > > good approximation for the common case. >>> > > Not sure what Kylin is designed for. Let's wait to hear from the >>> experts! >>> > > >>> > > On Sep 13, 2016 12:49, "Mario Copperfield" <[email protected]> >>> wrote: >>> > > >>> > > > It's true that data appears in order in Kafka, but it can't assert >>> that >>> > > the >>> > > > timestamp of data is ordered, in fact, in real time it always >>> > > > appear >>> > > > without order >>> > > > >>> > > > On Tue, Sep 13, 2016 at 3:14 PM, Sarnath K <[email protected]> >>> wrote: >>> > > > >>> > > > > I am not sure about what Kylin does. But I know that data >>> appears in >>> > > > order >>> > > > > in Kafka broker. But the consumer can consume in any order that >>> > > > > it >>> > > likes. >>> > > > > So, offsets are more driven by Consumers and Kafka does not >>> > > > > have >>> a >>> > say >>> > > > on >>> > > > > it. >>> > > > > Sharing this based on my preliminary understanding of how Kafka >>> > works. >>> > > > > Best, >>> > > > > Sarnath >>> > > > > >>> > > > > On Sep 13, 2016 12:41, "Mario Copperfield" >>> > > > > <[email protected]> >>> > > wrote: >>> > > > > >>> > > > > > Dear all, >>> > > > > > I am using kylin streaming build, and when i read the >>> code >>> > > about >>> > > > > > this module, i found that kylin use binary search to find the >>> > offset >>> > > > > which >>> > > > > > is the closest adjust to the starttamp. I doubt that is that >>> work >>> > if >>> > > > the >>> > > > > > data in kafka is not order? >>> > > > > > Thanks and waits. >>> > > > > > >>> > > > > > >>> > > > > > -- >>> > > > > > Best regards, >>> > > > > > Amuro Copperfield >>> > > > > > >>> > > > > >>> > > > >>> > > > >>> > > > >>> > > > -- >>> > > > Best regards, >>> > > > Amuro Copperfield >>> > > > >>> > > >>> > >>> > >>> > >>> > -- >>> > Best regards, >>> > Amuro Copperfield >>> > >>> >>> >>> >>> -- >>> With Warm regards >>> >>> Yiming Liu (刘一鸣) >>> >> >> >> >> -- >> Best regards, >> >> Shaofeng Shi >> >> > > > -- > Best regards, > > Shaofeng Shi >
-- Best regards, Amuro Copperfield
