Re: watermark trigger doesn't check whether element's timestamp is passed

[aj heller]

Hi Manu, Aljoscha,

I had been interested in implementing FLIP-2, but I haven't been able to
make time for it. There is no implementation yet that I'm aware of, and
I'll gladly step aside (or help out how I can) if you or anyone is
interested to take charge of it.

That said, I'm also not sure if discussions are ongoing. I had hoped to
prototype the proposal as is, to have something more concrete to discuss.

Cheers,
aj
On Nov 1, 2016 3:24 PM, "Manu Zhang"  wrote:

> Thanks.  The ideal case is to fire after watermark past each element from
> the window but that requires a custom trigger and FLIP-2 as well. The
> enhanced window evictor will help to avoid the last firing.
>
> Are the discussions on FLIP-2 still going on ?
> Are there any opening JIRAs or PRs ? (The proposed `ProcessWindowFunction`
> will be sufficient for my case)
> Is there a workaround now for my case ?
>
> Thanks again for following through this.
> Manu
>
> On Wed, Nov 2, 2016 at 1:07 AM Aljoscha Krettek 
> wrote:
>
>> Ah, I finally understand it. You would a way to query the current
>> watermark in the window function to only emit those elements where the
>> timestamp is lower than the watermark.
>>
>> When the window fires again, do you want to emit elements that you
>> emitted during the last firing again? If not, I think you also need to use
>> an evictor to evict the elements from the window where the timestamp is
>> lower than the watermark. With this FLIP https://cwiki.apache.org/
>> confluence/display/FLINK/FLIP-2+Extending+Window+Function+Metadata we
>> should be able to extend the WindowFunction Context to also provide the
>> current watermark. With this recent PR https://github.com/apache/
>> flink/pull/2736 you would be able to evict elements from the window
>> state after the window function was called.
>>
>> Cheers,
>> Aljoscha
>>
>> On Tue, 1 Nov 2016 at 02:27 Manu Zhang  wrote:
>>
>> Yes, here's the example https://github.com/manuzhang/flink/blob/pv/flink-
>> examples/flink-examples-streaming/src/main/scala/org/
>> apache/flink/streaming/scala/examples/session/
>> PageViewSessionWindowing.scala
>>
>> If you print and compare the timestamp of timer with that of "PageView"
>> in the outputs, you could see what I mean.
>>
>> I think the recently introduced TimelyFlatMapFunction is close to what I
>> want to achieve. It will be great if we can query time information in the
>> window function so I filed https://issues.apache.
>> org/jira/browse/FLINK-4953
>>
>> Thanks for your time.
>>
>> Manu
>>
>> On Mon, Oct 31, 2016 at 10:29 PM Aljoscha Krettek 
>> wrote:
>>
>> Hmm, I don't completely understand what's going on. Could you maybe post
>> an example, with the trigger code that shows this behaviour?
>>
>> Cheers,
>> Aljoscha
>>
>> On Thu, 27 Oct 2016 at 17:12 Manu Zhang  wrote:
>>
>> Hi,
>>
>> It's what I'm seeing. If timers are not fired at the end of window, a
>> state (in the window) whose timestamp is *after *the timer will also be
>> emitted. That's a problem for event-time trigger.
>>
>> Thanks,
>> Manu
>>
>>
>> On Thu, Oct 27, 2016 at 10:29 PM Aljoscha Krettek 
>> wrote:
>>
>> Hi,
>> is that example input/output what you would like to achieve or what you
>> are currently seeing with Flink? I think for your use case a custom Trigger
>> would be required that works like the event-time trigger but additionally
>> registers timers for each element where you want to emit.
>>
>> Cheers,
>> Aljoscha
>>
>> On Wed, 26 Oct 2016 at 04:04 Manu Zhang  wrote:
>>
>> Hi Aljoscha,
>>
>> Thanks for your response.  My use case is to track user trajectory based
>> on page view event when they visit a website.  The input would be like a
>> list of PageView(userId, url, eventTimestamp) with watermarks (=
>> eventTimestamp - duration). I'm trying SessionWindows with some event time
>> trigger. Note we can't wait for the end of session window due to latency.
>> Instead, we want to emit the user trajectories whenever a buffered
>> PageView's event time is passed by watermark. I tried
>> ContinuousEventTimeTrigger and a custom trigger which sets timer on each
>> element's timestamp. For both triggers I've witnessed a problem like the
>> following (e.g. a session gap of 5)
>>
>> PageView("user1", "http://foo;, 1)
>> PageView("user1", "http://foo/bar;, 2)
>> Watermark(1) => emit UserTrajectory("user1", "http://foo -> *http://foo/bar
>> *", [1,6])
>> PageView("user1", "http://foo/bar/foobar;, 5)
>> Watermark(4) => emit UserTrajectory("user1", "http://foo ->
>> http://foo/bar -> *http://foo/bar/foobar *", [1,
>> 10])
>>
>> The urls in bold should be included since there could be events before
>> them not arrived yet.
>>
>>
>> Thanks,
>> Manu
>>
>>
>> On Tue, Oct 25, 2016 at 1:36 AM Aljoscha Krettek 
>> wrote:
>>
>> Hi,
>> with 

Re: Merging N parallel/partitioned WindowedStreams together, one-to-one, into a global window stream

[AJ Heller]

Thank you Fabian, I think that solves it. I'll need to rig up some tests to
verify, but it looks good.

I used a RichMapFunction to assign ids incrementally to windows (mapping
STREAM_OBJECT to Tuple2<Long, STREAM_OBJECT> using a private long value in
the mapper that increments on every map call). It works, but by any chance
is there a more succinct way to do it?

On Thu, Oct 6, 2016 at 1:50 PM, Fabian Hueske <fhue...@gmail.com> wrote:

> Maybe this can be done by assigning the same window id to each of the N
> local windows, and do a
>
> .keyBy(windowId)
> .countWindow(N)
>
> This should create a new global window for each window id and collect all
> N windows.
>
> Best, Fabian
>
> 2016-10-06 22:39 GMT+02:00 AJ Heller <a...@drfloob.com>:
>
>> The goal is:
>>  * to split data, random-uniformly, across N nodes,
>>  * window the data identically on each node,
>>  * transform the windows locally on each node, and
>>  * merge the N parallel windows into a global window stream, such that
>> one window from each parallel process is merged into a "global window"
>> aggregate
>>
>> I've achieved all but the last bullet point, merging one window from each
>> partition into a globally-aggregated window output stream.
>>
>> To be clear, a rolling reduce won't work because it would aggregate over
>> all previous windows in all partitioned streams, and I only need to
>> aggregate over one window from each partition at a time.
>>
>> Similarly for a fold.
>>
>> The closest I have found is ParallelMerge for ConnectedStreams, but I
>> have not found a way to apply it to this problem. Can flink achieve this?
>> If so, I'd greatly appreciate a point in the right direction.
>>
>> Cheers,
>> -aj
>>
>
>

Re: ResourceManager not using correct akka URI in standalone cluster (?)

[AJ Heller]

Thank you Till. I was in a time crunch, and rebuilt my cluster from the
ground up with hadoop installed. All works fine now, `netstat -pn | grep
6123` shows flink's pid. Hadoop may be irrelevant, I can't rule out PEBKAC
yet :-). Sorry, when I have time I'll attempt to reproduce the scenario, on
the off chance there's a bug in there I can help dig up.

Best,
aj

Re: ResourceManager not using correct akka URI in standalone cluster (?)

[AJ Heller]

More information:

>From the master node, I cannot `telnet localhost 6123` nor `telnet  6123` while the cluster is apparently running. Connection refused
immediately. `netstat -n | grep 6123` is empty. There's no server
listening. But the processes are running on all machines.

Does it matter that I don't have hadoop or HDFS installed? It is optional,
right? To be clear, this fails at startup, long before I'm able to run any
job.

On Amazon EC2, the machines know of their private IPs, but not their public
IPs. I've instructed the cluster to operate over the public network because
I couldn't get the private IP scenario working.

Running `./bin/start-local.sh` shows non-zero counts in the Flink
Dashboard. Cluster setups show zero-counts all around.

-aj

On Thu, Sep 15, 2016 at 12:41 PM, AJ Heller <a...@drfloob.com> wrote:

> I'm running a standalone cluster on Amazon EC2. Leader election is
> happening according to the logs, and the Flink Dashboard is up and running,
> accessible remotely. The issue I'm having is that the SocketWordCount
> example is not working, the local connection is being refused!
>
> In the Flink Dashboard, 0 task managers are being reported. And in the
> jobmanager logs, the last line indicates "leader session null". All other
> akka URIs in the log file begin "akka.tcp://flink@PUBLIC_IP/...", but the
> Resourse Manager URI indicated "akka://flink/...".
>
>
> jobmanager log:
> http://pastebin.com/VWJM8XvW
>
> client log:
> http://pastebin.com/ZrWsbcwa
>
> flink-conf.yaml:
> http://pastebin.com/xy2tz7WS
>
> master and slave files are populated with public ips as well.
>

ResourceManager not using correct akka URI in standalone cluster (?)

[AJ Heller]

I'm running a standalone cluster on Amazon EC2. Leader election is
happening according to the logs, and the Flink Dashboard is up and running,
accessible remotely. The issue I'm having is that the SocketWordCount
example is not working, the local connection is being refused!

In the Flink Dashboard, 0 task managers are being reported. And in the
jobmanager logs, the last line indicates "leader session null". All other
akka URIs in the log file begin "akka.tcp://flink@PUBLIC_IP/...", but the
Resourse Manager URI indicated "akka://flink/...".


jobmanager log:
http://pastebin.com/VWJM8XvW

client log:
http://pastebin.com/ZrWsbcwa

flink-conf.yaml:
http://pastebin.com/xy2tz7WS

master and slave files are populated with public ips as well.