Re: Integration Alluxio and Flink

Hi Till, I tried to downgrade the Alluxio's netty version from 4.0.28.Final to 4.0.27.Final to align Flink and Alluxio dependencies. First of all, Flink 1.0.0 uses 4.0.27.Final, is it correct? Btw it doesn't work, same error as above. BR, Andrea 2016-03-14 15:30 GMT+01:00 Till Rohrmann

Re: Memory ran out PageRank

Hi I think this is the same issue we had before on the list [1]. Stephan recommended the following workaround: A possible workaround is to use the option "setSolutionSetUnmanaged(true)" on the iteration. That will eliminate the fragmentation issue, at least. Unfortunately, you cannot set

Re: Memory ran out PageRank

Thank you for this alternative. I don’t understand how the workaround will fix this on systems with limited memory and maybe larger graph. Running Connected Components on the same graph gives the same problem. IterationHead(Unnamed Delta Iteration)(82/88) switched to FAILED

Memory ran out PageRank

Hi, While running PageRank on a synthetic graph I run into this problem: Any advice on how should I proceed to overcome this memory issue? IterationHead(Vertex-centric iteration (org.apache.flink.graph.library.PageRank$VertexRankUpdater@7712cae0 |

Re: Behavior of SlidingProessingTimeWindow with CountTrigger

Hi Aijoscha, Wow, great illustration. That was very clear explanation. Yes, I did enter the elements fast for case b and I was seeing more of case As. Also, sometimes I have seen a window getting triggered when I enter 1 or 2 elements, I believe that is expansion of case A, w.r.t to window 2.

Re: Behavior of SlidingProessingTimeWindow with CountTrigger

Hi, sure, the evictors are a bit confusing (especially the fact that they are called evictors). They should more correctly called “Keepers”. The process is the following: 1. Trigger Fires 2. Evictor decides what elements to keep, so a CountEvictor.of(3) says, keep only three elements, all

Re: Integration Alluxio and Flink

Hi Andrea, the problem won’t be netty-all but netty, I suspect. Flink is using version 3.8 whereas alluxio-core-client uses version 3.2.2. I think you have to exclude or shade this dependency away. Cheers, Till ​ On Mon, Mar 14, 2016 at 5:12 PM, Andrea Sella wrote:

Re: Passing two value to the ConvergenceCriterion function

Hi, take a look at the "Record" class. That one implements the Value interface and can have multiple values. On Fri, Mar 11, 2016 at 6:01 PM, Riccardo Diomedi < riccardo.diomed...@gmail.com> wrote: > Hi > > I want to send two value to the ConvergenceCriterion function, so i > decided to use an

Re: Passing two value to the ConvergenceCriterion function

Ok! On 14 Mar 2016, at 10:41, Robert Metzger wrote: > Hi, > > take a look at the "Record" class. That one implements the Value interface > and can have multiple values. > > On Fri, Mar 11, 2016 at 6:01 PM, Riccardo Diomedi > wrote: > Hi >

Re: kafka.javaapi.consumer.SimpleConsumer class not found

Can you send me the full build file to further investigate the issue? On Fri, Mar 11, 2016 at 4:56 PM, Balaji Rajagopalan < balaji.rajagopa...@olacabs.com> wrote: > Robert, > That did not fix it ( using flink and connector same version) . Tried > with scala version 2.11, so will try to see

Re: Kafka integration error

Hi Stefanos, this looks like an issue with Kafka. Which version does your broker have? Can you check the logs of the broker you are trying to connect to? On Fri, Mar 11, 2016 at 5:27 PM, Stefanos Antaris < antaris.stefa...@gmail.com> wrote: > Hi to all, > > i am trying to make Flink to work

Re: kafka.javaapi.consumer.SimpleConsumer class not found

Yes figured that out, thanks for point that, my bad. I have put back 0.10.2 as flink version, will try to reproduce the problem again, this time I have explicitly called out the scala version as 2.11. On Mon, Mar 14, 2016 at 4:14 PM, Robert Metzger wrote: > Hi, > >

Re: kafka.javaapi.consumer.SimpleConsumer class not found

Hi, flink-connector-kafka_ doesn't exist for 1.0.0. You have to use either flink-connector-kafka-0.8_ or flink-connector-kafka-0.9_ On Mon, Mar 14, 2016 at 11:17 AM, Balaji Rajagopalan < balaji.rajagopa...@olacabs.com> wrote: > What I noticied was that, if I remove the dependency on >

Re: kafka.javaapi.consumer.SimpleConsumer class not found

Robert, I have moved on to latest version of flink of 1.0.0 hoping that will solve my problem with kafka connector . Here is my pom.xml but now I cannot get the code compiled. [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.1:compile (scala-compile-first) on project

Re: kafka.javaapi.consumer.SimpleConsumer class not found

What I noticied was that, if I remove the dependency on flink-connector-kafka so it is clearly to do something with that dependency. On Mon, Mar 14, 2016 at 3:46 PM, Balaji Rajagopalan < balaji.rajagopa...@olacabs.com> wrote: > Robert, >I have moved on to latest version of flink of 1.0.0

Re: Checkpoint

Hi, I’m not aware of a problem where pending files are not moved to their final locations. So if you have such a behavior it would indicate a bug. Also, the "trigger checkpoint” does not yet indicate that the checkpoint is happening. If you have a very long sleep interval in some of your

Re: Flink and YARN ship folder

Hi Andrea, You don't have to manually replicate any operations on the slaves. All files in the lib/ folder are transferred to all containers (Jobmanagers and TaskManagers). On Sat, Mar 12, 2016 at 3:25 PM, Andrea Sella wrote: > Hi Ufuk, > > I'm trying to execute

Using a POJO class wrapping an ArrayList

Hi all, Now I am building a POJO class for key selectors. Here is the class: public class Ids implements Comparable, Serializable{ private static final long serialVersionUID = 1L; private ArrayList ids = new ArrayList(); Ids() {}

Re: Log4j configuration on YARN

Hi Nick, the name of the "log4j-yarn-session.properties" file might be a bit misleading. The file is just used for the YARN session client, running locally. The Job- and TaskManager are going to use the log4j.properties on the cluster. On Fri, Mar 11, 2016 at 7:20 PM, Ufuk Celebi

Re: Behavior of SlidingProessingTimeWindow with CountTrigger

Hi, I created a visualization to help explain the situation: http://s21.postimg.org/dofhcw52f/window_example.png The SlidingProcessingTimeWindows assigner assigns elements to windows based on the current processing time. The CountTrigger only fires if a window contains 5 elements (or more). In

Re: kafka.javaapi.consumer.SimpleConsumer class not found

Yep the same issue as before(class not found) with flink 0.10.2 with scala version 2.11. I was not able to use scala 2.10 since connector for flink_connector_kafka for 0.10.2 is not available. balaji On Mon, Mar 14, 2016 at 4:20 PM, Balaji Rajagopalan < balaji.rajagopa...@olacabs.com> wrote: >

RE:Flink job on secure Yarn fails after many hours

Hello everyone, We are facing the same probleme now in our Flink applications, launch using YARN. Just want to know if there is any update about this exception ? Thanks Thomas De : ni...@basj.es [ni...@basj.es] de la part de Niels Basjes

Re: Behavior of SlidingProessingTimeWindow with CountTrigger

Hi Aljoscha, Thank you for the explanation and the link on IBM infosphere. That explains whey I am seeing (a,3) and (b,3) in my example. Yes, the name Evictor is confusing. Thanks and Regards, Vishnu Viswanath, www.vishnuviswanath.com On Mon, Mar 14, 2016 at 11:24 AM, Aljoscha Krettek

Re: Memory ran out PageRank

Hi, I understand the confusion. So far, I did not run into the problem, but I think this needs to be adressed as all our graph processing abstractions are implemented on top of the delta iteration. According to the previous mailing list discussion, the problem is with the solution set and

Re: Memory ran out PageRank

Probably the limitation is that the number of keys is different in the real and the synthetic data set respectively. Can you confirm this? The solution set for delta iterations is currently implemented as an in-memory hash table that works on managed memory segments, but is not spillable. – Ufuk

Re: Memory ran out PageRank

Hi Ovidiu, this option won't fix the problem if your system doesn't have enough memory :) It only defines whether the solution set is kept in managed memory or not. For more iteration configuration options, take a look at the Gelly documentation [1]. -Vasia. [1]:

Re: Memory ran out PageRank

This problem is surprising as I was able to run PR and CC on a larger graph (2bil edges) but with this synthetic graph (1bil edges groups of 10) I ran out of memory; regarding configuration (memory and parallelism, other internals) I was using the same. There is some limitation somewhere I

Re: Memory ran out PageRank

Yes, largely different. I was expecting for the solution set to be spillable. This is somehow very hard limitation, the layout of the data makes the difference. By contract, I am able to run successfully CC on the synthetic data but RDDs are persisted in memory or on disk. Best, Ovidiu > On

Re: Memory ran out PageRank

Correction: successfully CC I am running is on top of your friend, Spark :) Best, Ovidiu > On 14 Mar 2016, at 20:38, Ovidiu-Cristian MARCU > wrote: > > Yes, largely different. I was expecting for the solution set to be spillable. > This is somehow very hard

Re: asm IllegalArgumentException with 1.0.0

We're having the same issue (we also have a dependency on flink-connector-elasticsearch). It's only happening to us in IntelliJ though. Is this the case for you as well? On Thu, Mar 10, 2016 at 3:20 PM, Zach Cox wrote: > After some poking around I noticed > that

Re: asm IllegalArgumentException with 1.0.0

Yes pretty much - we use sbt to run the job in a local environment, not Intellij, but should be the same thing. We were also seeing that exception running unit tests locally. We did not see the exception when assembling a fat jar and submitting to a remote Flink cluster. It seems like the

XGBoost4J: Portable Distributed XGboost in Flink

Hi Flink Community: I am sending this email to let you know we just release XGBoost4J which also runs on Flink. In short, XGBoost is a machine learning package that is used by more than half of the machine challenge winning solutions and is already widely used in industry. The distributed

Integration Alluxio and Flink

Hi to all, I'm trying to integrate Alluxio and Apache Flink, I followed Running Flink on Alluxio to setup Flink. I tested in local mode executing: bin/flink run ./examples/batch/WordCount.jar --input

Re: Application logging on YARN

Ok, my bad, I was simply looking in the wrong place. I though the logs were sent to YARN but they were actually stored in the Flink logs folder. Problem solved, sorry for the mix up. On Sun, Mar 13, 2016 at 8:48 PM, Stefano Baghino < stefano.bagh...@radicalbit.io> wrote: > There's another open

Re: TimeWindow not getting last elements any longer with flink 1.0 vs 0.10.1

Hi Arnaud, with version 1.0 the behaviour for window triggering in case of a finite stream was slightly changed. If you use event time, then all unfinished windows are triggered in case that your stream ends. This can be motivated by the fact that the end of a stream is equivalent to no elements

Re: Flink and YARN ship folder

Hi Robert, Ok, thank you. 2016-03-14 11:13 GMT+01:00 Robert Metzger : > Hi Andrea, > > You don't have to manually replicate any operations on the slaves. All > files in the lib/ folder are transferred to all containers (Jobmanagers and > TaskManagers). > > > On Sat, Mar 12,