Re: Package multiple jobs in a single jar

2015-05-19 Thread Matthias J. Sax
; /** Set up the flink job in the passed ExecutionEnvironment */ ExecutionEnvironment config(ExecutionEnvironment env); } What do you think? On Sun, May 17, 2015 at 10:38 PM, Matthias J. Sax mj...@informatik.hu-berlin.de wrote: Hi, I like the idea that Flink's WebClient can show

Re: Package multiple jobs in a single jar

2015-05-17 Thread Matthias J. Sax
Hi, I like the idea that Flink's WebClient can show different plans for different jobs within a single jar file. I prepared a prototype for this feature. You can find it here: https://github.com/mjsax/flink/tree/multipleJobsWebUI To test the feature, you need to prepare a jar file, that

Re: About Interplay of Merged Streams, Output Selectors and Checkpoint Barriers (and Watermarks)

2015-05-12 Thread Matthias J. Sax
Hi, I don't understand why we need the same machnism twice in the code... Could checkpoing barrieres and low watermarks be unified (or one build on-top/by-using the other) -Matthias On 05/12/2015 02:47 PM, Gyula Fóra wrote: Hi, Checkpoint barriers are handled directly on top of the network

Re: [DISCUSS] Merging Storm compatibility to Flink-contrib

2015-05-12 Thread Matthias J. Sax
(by Matthias J. Sax) [1] raises the issue as it is way more code to maintain and is more complex in general that how the community would like to handle these in terms of distribution. Do we want to have it in the Flink repository or maybe in a separate one. I am personally really for having the Storm

Re: [DISCUSS] Behaviour of Streaming Sources

2015-05-08 Thread Matthias J. Sax
Did you consider the Storm way to handle this? Storm offers a method void next() that uses a collector object to emit new tuples. Using this interface, next() can loop internally as long as tuples are available and return if there is (currently) no input. What I have seen, people tend to emit a

Re: how can implemented subquery in flink

2015-05-05 Thread Matthias J. Sax
If the sub-query is not correlated, you can (always) transform it into a simple join query. However, the query you show is no valid SQL... Did you mean: SELECT C_CUSTKEY,C_NAME FROM Customers WHERE C_MKTSEGMENT = (SELECT C_MKTSEGMENT FROM Customers WHERE C_ADDRESS=MG9kdTD2WBHm) In this case,

Re: Rework of the window-join semantics

2015-04-24 Thread Matthias J. Sax
will be generated after all the outputs. Cheers, Asterios [1] http://www-01.ibm.com/support/knowledgecenter/#!/SSCRJU_3.2.1/com.ibm.swg.im.infosphere.streams.spl-standard-toolkit-reference.doc/doc/join.html On Thu, Apr 9, 2015 at 1:30 PM, Matthias J. Sax mj...@informatik.hu-berlin.de wrote: Hi

Re: Rework of the window-join semantics

2015-04-09 Thread Matthias J. Sax
/naiad_final.pdf On 07 Apr 2015, at 11:50, Matthias J. Sax mj...@informatik.hu-berlin.demailto:mj...@informatik.hu-berlin.de wrote: Hi @all, please keep me in the loop for this work. I am highly interested and I want to help on it. My initial thoughts are as follows: 1) Currently, system

Re: Rework of the window-join semantics

2015-04-08 Thread Matthias J. Sax
https://users.soe.ucsc.edu/~abadi/Papers/naiad_final.pdf https://users.soe.ucsc.edu/~abadi/Papers/naiad_final.pdf On 07 Apr 2015, at 11:50, Matthias J. Sax mj...@informatik.hu-berlin.demailto:mj...@informatik.hu-berlin.de wrote: Hi @all, please keep me in the loop for this work. I am

Canceling a Cluster Job from Java

2015-04-02 Thread Matthias J. Sax
Hi, I want to cancel a running job from a Java program. However, I don't know how to do it. The Client class (org.apache.flink.client.program.Client) that is used to submit a job, does not provide a method for it (a Client.cancel(JobId) would be nice). There is also the Scala JobClient classm

Re: Canceling a Cluster Job from Java

2015-04-02 Thread Matthias J. Sax
(askTimeout)); Let me know if this approach is not working for you. On Thu, Apr 2, 2015 at 10:57 AM, Matthias J. Sax mj...@informatik.hu-berlin.de wrote: Hi, I want to cancel a running job from a Java program. However, I don't know how to do it. The Client class

Storm compatibility layer for Flink (first beta available)

2015-04-02 Thread Matthias J. Sax
Hi @all, I started to work on an compatibility layer to run Storm Topologies on Flink. I just pushed a first beta: https://github.com/mjsax/flink/tree/flink-storm-compatibility Please check it out, and let me know how you like it. In this first version, I tried to code without changing too many

Re: Question about Infinite Streaming Job on Mini Cluster and ITCase

2015-03-31 Thread Matthias J. Sax
is shutting down before the cancel method, that probably a bug. Robert On Fri, Mar 27, 2015 at 10:07 PM, Matthias J. Sax mj...@informatik.hu-berlin.de wrote: Hi, I am trying to run an infinite streaming job (ie, one that does not terminate because it is generating output date randomly

Question about Infinite Streaming Job on Mini Cluster and ITCase

2015-03-27 Thread Matthias J. Sax
Hi, I am trying to run an infinite streaming job (ie, one that does not terminate because it is generating output date randomly on the fly). I kill this job with .stop() or .shutdown() method of ForkableFlinkMiniCluster. I did not find any example using a similar setup. In the provided examples,

Re: Question about Flink Streaming

2015-03-24 Thread Matthias J. Sax
24, 2015 at 5:05 PM, Matthias J. Sax mj...@informatik.hu-berlin.de wrote: Hi, as I get more familiar with Flink streaming and do some coding, I hit a few points which I want do discuss about because I find them contra-intuitive. Please tell me, what you think about it or clarify what I

Re: Problem mvn install

2015-03-02 Thread Matthias J. Sax
in Flink's master branch. I don't think they are put there as part of the build process. Best, Fabian 2015-03-02 15:09 GMT+01:00 Matthias J. Sax mj...@informatik.hu-berlin.de : Hi, if I start mvn-Dmaven.test.skip=true clean install, the goal fails and I get the following error

Re: Questions about flink-streaming-examples

2015-02-26 Thread Matthias J. Sax
are talking is about a week old, I wrote it. Kind regards, Marton On Thu, Feb 26, 2015 at 9:04 AM, Matthias J. Sax mj...@informatik.hu-berlin.de wrote: Hi, I just build flink-streaming and avoid the problem. I guess, that the issue is related to the module structure and dependencies

Flink Master broken...

2015-02-24 Thread Matthias J. Sax
Hi, I just pulled the latest version flink master and I got a compilation error. I tracked the bug down. It seems that the following commit moved a class without changing the package statement... Cheers, Matthias commit 354efec0f9da0fa03ea9b337b02a1a2a03a9ac16 Author: Robert Metzger

Re: Flink Master broken...

2015-02-24 Thread Matthias J. Sax
build it multiple times today, also our Travis says everything is fine: https://travis-ci.org/apache/flink/builds On Tue, Feb 24, 2015 at 5:29 PM, Matthias J. Sax mj...@informatik.hu-berlin.de wrote: Hi, I just pulled the latest version flink master and I got a compilation error. I

Re: Flink Master broken...

2015-02-24 Thread Matthias J. Sax
and do a mvn -DskipTests clean install? 2015-02-24 18:26 GMT+01:00 Matthias J. Sax mj...@informatik.hu-berlin.de: Hi, I build on command line: mjsax@T420s-dbis-mjsax:~/workspace_flink/flink$ git pull flink master From https://github.com/apache/flink * branchmaster

<    1   2   3   4