Hi,
what do you mean by "still experiencing the same issues"? Is the key count
still very hight, i.e. 500k windows?
For the watermark generation, specifying a lag of 2 days is very
conservative. If the watermark is this conservative I guess there will
never arrive elements that are behind the watermark, thus you wouldn't need
the late-element handling in your triggers. The late-element handling in
Triggers is only required to compensate for the fact that the watermark can
be a heuristic and not always correct.
Cheers,
Aljoscha
On Thu, 28 Apr 2016 at 21:24 Christopher Santiago <ch...@ninjametrics.com>
wrote:
> Hi Aljoscha,
>
>
> Aljoscha Krettek wrote
> >>is there are reason for keying on both the "date only" field and the
> "userid". I think you should be fine by just specifying that you want 1-day
> windows on your timestamps.
>
> My mistake, this was from earlier tests that I had performed. I removed it
> and went to keyBy(2) and I am still experiencing the same issues.
>
>
> Aljoscha Krettek wrote
> >>Also, do you have a timestamp extractor in place that takes the timestamp
> from your data and sets it as the internal timestamp field.
>
> Yes there is, it is from the BoundedOutOfOrdernessGenerator example:
>
> public static class BoundedOutOfOrdernessGenerator implements
> AssignerWithPeriodicWatermarks<Tuple3<DateTime, String, String>> {
> private static final long serialVersionUID = 1L;
> private final long maxOutOfOrderness =
> Time.days(2).toMilliseconds();
> private long currentMaxTimestamp;
>
> @Override
> public long extractTimestamp(Tuple3<DateTime, String, String>
> element, long previousElementTimestamp) {
> long timestamp = element.f0.getMillis();
> currentMaxTimestamp = Math.max(timestamp, currentMaxTimestamp);
> return timestamp;
> }
>
> @Override
> public Watermark getCurrentWatermark() {
> return new Watermark(currentMaxTimestamp - maxOutOfOrderness);
> }
> }
>
> Thanks,
> Chris
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Multiple-windows-with-large-number-of-partitions-tp6521p6562.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
