Re: Ordering of stream from different kafka partitions

2018-06-20 Thread Andrey Zagrebin
Hi Amol, > In above code also it will sort the records in specific time window only. All windows will be emitted as watermark passes the end of the window. The watermark only increases. So the non-overlapping windows should be also sorted by time and as a consequence the records across windows

Re: Ordering of stream from different kafka partitions

2018-06-20 Thread Andrey Zagrebin
Hi, Good point, sorry for confusion, BoundedOutOfOrdernessTimestampExtractor of course does not buffer records, you need to apply windowing (e.g. TumblingEventTimeWindows) for that and then sort the window output by time and emit records in sorted order. You can also use windowAll which alread

Re: Ordering of stream from different kafka partitions

2018-06-20 Thread sihua zhou
Hi, I think a global ordering is a bit impractical on production, but in theroy, you still can do that. You need to - Firstly fix the operate's parallelism to 1(except the source node). - If you want to sort the records within a bouned time, then you can keyBy() a constant and window it,

Re: Ordering of stream from different kafka partitions

2018-06-19 Thread Andrey Zagrebin
Hi Amol, I think you could try (based on your stack overflow code) org.apache.flink.streaming.api.functions.timestamps.BoundedOutOfOrdernessTimestampExtractor like this: DataStream streamSource = env .addSource(kafkaConsumer) .setParallelism(4) .assignTimestampsAndWatermarks( new