Re: Multiple windows with large number of partitions

[Christopher Santiago]

Hi Aljoscha,


Aljoscha Krettek wrote
>>is there are reason for keying on both the "date only" field and the
"userid". I think you should be fine by just specifying that you want 1-day
windows on your timestamps.

My mistake, this was from earlier tests that I had performed.  I removed it
and went to keyBy(2) and I am still experiencing the same issues.


Aljoscha Krettek wrote
>>Also, do you have a timestamp extractor in place that takes the timestamp
from your data and sets it as the internal timestamp field. 

Yes there is, it is from the BoundedOutOfOrdernessGenerator example:

    public static class BoundedOutOfOrdernessGenerator implements
AssignerWithPeriodicWatermarks<Tuple3&lt;DateTime, String, String>> {
        private static final long serialVersionUID = 1L;
        private final long maxOutOfOrderness = Time.days(2).toMilliseconds();
        private long currentMaxTimestamp;

        @Override
        public long extractTimestamp(Tuple3<DateTime, String, String>
element, long previousElementTimestamp) {
            long timestamp = element.f0.getMillis(); 
            currentMaxTimestamp = Math.max(timestamp, currentMaxTimestamp);
            return timestamp;
        }

        @Override
        public Watermark getCurrentWatermark() {
            return new Watermark(currentMaxTimestamp - maxOutOfOrderness);
        }
    }

Thanks,
Chris



--
View this message in context: 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Multiple-windows-with-large-number-of-partitions-tp6521p6562.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at 
Nabble.com.