Re: Fwd: About Kafka in Kylin

OK, I'll try On 9/13/16, ShaoFeng Shi wrote: > sorry one correction: > > > " if some messages are arrived late than the margin, it will not be lost" > should be: > " if some messages are arrived late than the margin, it will be lost" > > Mario, you can try to set a bigger

Re: Fwd: About Kafka in Kylin

sorry one correction: " if some messages are arrived late than the margin, it will not be lost" should be: " if some messages are arrived late than the margin, it will be lost" Mario, you can try to set a bigger margin value to reduce the possibility. 2016-09-13 22:05 GMT+08:00 ShaoFeng Shi

Re: Fwd: About Kafka in Kylin

The current design is still an experimental approach. Kafka could not guarantee the global order, so we have to find other solution. The new design Streaming OLAP solution will relay on the Kafka partition order, instead of app timestamp. The code is under KYLIN-1726 branch still. 2016-09-13

Re: Fwd: About Kafka in Kylin

Yes, that's true. If you are looking at an app timestamp(event origin time), then We can't binary search on it. Though Binary search may be a good approximation for the common case. Not sure what Kylin is designed for. Let's wait to hear from the experts! On Sep 13, 2016 12:49, "Mario

Re: Fwd: About Kafka in Kylin

I am not sure about what Kylin does. But I know that data appears in order in Kafka broker. But the consumer can consume in any order that it likes. So, offsets are more driven by Consumers and Kafka does not have a say on it. Sharing this based on my preliminary understanding of how Kafka

Fwd: About Kafka in Kylin

Dear all, I am using kylin streaming build, and when i read the code about this module, i found that kylin use binary search to find the offset which is the closest adjust to the starttamp. I doubt that is that work if the data in kafka is not order? Thanks and waits. -- Best