Re: Internal buffers supervision and yarn vCPUs

Concerning the first question: What you are looking for is backpressure monitoring. If a task cannot push its data to the next task, it is backpressured. This pull request adds a first version of backpressure monitoring: https://github.com/apache/flink/pull/1578 We will try and get it merged

Re: Flink cluster and Java 8

Anyone looking into this? Java 7 reached its end of life at april 2015 with its last public update (numer 80) and the ability to run Java 8 jobs would be more and more important in the future. IMHO, the default target of the maven compiler plugin should be set to 1.8 in the 1.0 release. In most of

Re: Flink cluster and Java 8

I've tested several configurations (also changing my compilation to 1.7 but then sesame 4 was causing the error [1]): 1. Flink compiled with java 1.7 (default), runned within Eclipse with Java 8: OK 2. Flink compiled with java 1.7 (default), runned the cluster with java 8: not able to

RE: Internal buffers supervision and yarn vCPUs

Ok thanks ! All that’s left is to wait then. B.R. From: ewenstep...@gmail.com [mailto:ewenstep...@gmail.com] On Behalf Of Stephan Ewen Sent: jeudi 4 février 2016 11:19 To: user@flink.apache.org Subject: Re: Internal buffers supervision and yarn vCPUs Concerning the first question: What you

Re: Internal buffers supervision and yarn vCPUs

> On 04 Feb 2016, at 12:02, Gwenhael Pasquiers > wrote: > > Ok thanks ! > > All that’s left is to wait then. If you have spare time and are working with the current snapshot version, it would be great to get some feedback on the pull request. :-) – Ufuk

Re: Internal buffers supervision and yarn vCPUs

Hi Gwen, let me answer the second question: There is a JIRA to reintroduce the configuration parameter: https://issues.apache.org/jira/browse/FLINK-2213. I will try to get a fix for this into the 1.0 release. I think I removed back then because users were unable to define the number of vcores

Re: Understanding code of CountTrigger

Very good, you are absolutely right. :D > On 04 Feb 2016, at 05:07, Nirmalya Sengupta > wrote: > > on

Flink writeAsCsv

Hi all, I am new to flink. I wrote a simple program and I want it to output as csv file. timeWindowAll(Time.of(3, TimeUnit.MINUTES)) .apply(newFunction1()) .writeAsCsv("file:///user/someuser/Documents/somefile.csv"); When I change the sink to . print(), it works and outputs some results. I

RE: Flink writeAsCsv

Hi Radu, It is indeed interesting to know how each window could be registered separately - I am not sure it any of the existing mechanisms in Flink support this. I think you need to create your own output sink. It is a bit tricky to pass the window sequence number (actually I do not think

Internal buffers supervision and yarn vCPUs

Hi, I’ve got two more questions on different topic… First one : Is there a way to monitor the buffers status. In order to find bottleneck in our application we though it could be usefull to be able to have a look at the different exchange buffers’ status. To know if they are full (or as an

Re: Stream conversion

Hi! If I understand you correctly, what you are looking for is a kind of periodic batch job, where the input data for each batch is a large window. We have actually thought about this kind of application before. It is not on the short term road map that we shared a few weeks ago, but I think it

Re: Flink writeAsCsv

Hey Radu, As you are using the streaming api I assume that you call env.execute() in both cases. Is that the case? Do you see any errors appearing? My first call would be if your data type is not a tuple type then writeAsCsv does not work by default. Best, Marton On Thu, Feb 4, 2016 at 11:36

Merging of streams

I found that merge method for datastream does not exist in latest version . What is the equivalent for it ? Shall I use union or join ? Thanks Ashutosh

Re: Merging of streams

Hi, sorry for the inconvenience. The new name of merge is now union. Regards, Aljoscha > On 04 Feb 2016, at 13:01, Ashutosh Kumar wrote: > > I found that merge method for datastream does not exist in latest version . > What is the equivalent for it ? Shall I use

Re: Flink writeAsCsv

Hi Marton, Thanks to your comment I managed to get it worked. At least it outputs the results. However, what I need is to output each window result seperately. Now, it outputs the results of parallel working windows (I think) and appends the new results to them. For example, If I have parallelism

GC on TaskManagers stats

Hello, I have few questions regarding garbage collector’s stats on Taskmanagers and any help or further documentation would be great. I have collected “1 second polling requesting" stats on 7 Taskmanagers, through the relative request (/taskmanagers//) of the Monitoring REST API while a job,

Re: Stream conversion

For example, I will do aggregate operations with other windows (n-window aggregations) that are already outputted. I tried your suggestion and used filesystem sink, outputted to HDFS. I got k files in HDFS directory where k is the number of parallelism (I used single machine). These files get

Re: Distribution of sinks among the nodes

To your other question, there are two things in Flink: (1) Chaining. Tasks are folded together into one task, run by one thread. (2) Resource groups: Tasks stay separate, have separate threads, but share a slot (which means share memory resources). See the link in my previous mail for an

Re: Stream conversion

I'm wondering which kind of transformations you want to apply to the window you cannot apply with the DataStream API? Would it be sufficient for you to have the windows as files in HDFS and then run batch jobs against the windows on disk? If so, you could use our filesystem sink, which creates

DistributedMatrix in Flink

Hi all, as mentioned before I am trying to import the RowMatrix from Spark to Flink… In the code I already ran into a dead end… In the function multiplyGramianMatrixBy() (see end of mail) there is the line: rows.context.broadcast(v) (rows is a DataSet[Vector] What exactly is this line doing?

RE: Distribution of sinks among the nodes

Sorry I was confused about the number of slots, it’s good now. However, is disableChaing or disableOperatorChaining working properly ? I called those methods everywhere I could but it still seems that some of my operators are being chained together I can’t go over 16 used slot where I should be

Re: Checkpoints and event ordering

Hi Shikhar, the currently open windows are also part of the operator state. Whenever a window operator receives a barrier it will checkpoint the state of the user function and additionally all uncompleted windows. This also means that the window operator does not buffer the barriers. Once it has

Re: Distribution of sinks among the nodes

Hi Gwen! You actually need not 24 slots, but only as many as the highest parallelism is (16). Slots do not hold individual tasks, but "pipelines". Here is an illustration how that works.

RE: Distribution of sinks among the nodes

Okay ; Then I guess that the best we can do is to disable chaining (we really want one thread per operator since they are doing long operations) and have the same parallelism for sinks as mapping : that way each map will have it’s own sink and there will be no exchanges between flink

Re: Flink cluster and Java 8

Flink compiles correctly using java 8 as long as you leave java 1.7 source and target in the maven java compiler. If you change them to 1.8 flink-core doesn't compile anymore. On Thu, Feb 4, 2016 at 4:23 PM, Maximilian Michels wrote: > Hi Flavio, > > To address your points: > >

Re: release of task slot

Hey, I am actually facing a similar issue lately, where the job manager release the task slots as it cannot contact the taskmanager. Meanwhile the taskmanager is also trying to connect to the Jobmanager and fails multiple times. This happens on multiple taskmanagers seemingly randomly. So the TM

Re: Where does Flink store intermediate state and triggering/scheduling a delayed event?

Fabian, Thank a lot for your response. Really appreciated. I've some additional questions (please see inline) On Wed, Feb 3, 2016 at 2:42 PM, Fabian Hueske wrote: > Hi, > > 1) At the moment, state is kept on the JVM heap in a regular HashMap. > Is this state replicated

Re: Stream conversion

Hi Sane, Currently, DataSet and DataStream API a strictly separated. Thus, this is not possible at the moment. What kind of operation do you want to perform on the data of a window? Why do you want to convert the data into a data set? -Matthias On 02/04/2016 10:11 AM, Sane Lee wrote: > Dear

Re: Possibility to get the line numbers?

I had that problem/question some time ago, too. The quick fix is to just put the line number in the line itself. Go for it. However, we worked out a solution for another distributed processing system, that did the following: Read each partition, count the lines, broadcast a map

Re: Stream conversion

Hi Matthias, This need not to be necessarily in api functions. I just want to get a roadmap to add this functionality. Should I save each window's data into disk and create a new dataset environment in parallel? Or change trigger functionality maybe? I have large windows. As I asked in previous

Re: Stream conversion

I have also, similar scenario. Any suggestion would be appreciated. On Thu, Feb 4, 2016 at 10:29 AM Jeyhun Karimov wrote: > Hi Matthias, > > This need not to be necessarily in api functions. I just want to get a > roadmap to add this functionality. Should I save each

Re: Flink cluster and Java 8

yes I did On Thu, Feb 4, 2016 at 5:44 PM, Maximilian Michels wrote: > I see. Did you perform a full "mvn clean package -DskipTests" after > you changed the source level to 1.8? > > On Thu, Feb 4, 2016 at 4:33 PM, Flavio Pompermaier > wrote: > > Flink

Re: Flink writeAsCsv

You can get the end time of a window from the TimeWindow object which is passed to the AllWindowFunction. This is basically a window ID / index. I would go for a custom output sink which writes records to files based on their timestamp. IMO, this would be cleaner & easier than implementing the

Re: Flink cluster and Java 8

Hi Flavio, To address your points: 1) It runs. That's fine. 2) It doesn't work to run a Java 8 compiled Flink job with Java 7 Flink cluster if you use Java 8 non-backwards-compatible features in your job. 3) I compile Flink daily with Java 8. Also, we have Travis CI tests which uses OpenJDK and

Re: FlinkML 0.10.1 - Using SparseVectors with MLR does not work

Hi Sourigna, it turned out to be a bug in the GradientDescent implementation which cannot handle sparse gradients. That is not so problematic by itself, because the sum of gradient vectors is usually dense even if the individual gradient vectors are sparse. We simply forgot to initialize the

Re: release of task slot

Hi Radu, what does the log of the TaskManager 10.204.62.80:57910 say? Cheers, Till ​ On Wed, Feb 3, 2016 at 6:00 PM, Radu Tudoran wrote: > Hello, > > > > > > I am facing an error which for which I cannot figure the cause. Any idea > what could cause such an error? > >

Re: release of task slot

Yes exactly , it says it is quarantined. Gyula Gyula On Thu, Feb 4, 2016 at 4:09 PM Stephan Ewen wrote: > @Gyula Do you see log messages about quarantined actor systems? > > There may be an issue with Akka Death watches that once the connection is > lost, it cannot be

Re: DistributedMatrix in Flink

Hi Lydia, Spark and Flink are not identical. Thus, you’ll concepts in both system which won’t have a corresponding counter part in the other system. For example, rows.context.broadcast(v) broadcasts the value v so that you can use it on all Executors. Flink follows a slightly different concept

Re: Distribution of sinks among the nodes

I added a new Ticket: https://issues.apache.org/jira/browse/FLINK-3336 This will implement the data shipping pattern that you mentioned in your initial mail. I also have the Pull request almost ready. > On 04 Feb 2016, at 16:25, Gwenhael Pasquiers > wrote: >

RE: release of task slot

Hi, Well…yesterday when I looked into it there was no additional info than the one I have send. Today I reproduced the problem and I could see in the log file. akka.actor.ActorInitializationException: exception during creation at

Re: release of task slot

We should probably add to the TaskManager a "restart on quarantined" strategy anyways. We can detect it as follows: http://stackoverflow.com/questions/32471088/akka-cluster-detecting-quarantined-state On Thu, Feb 4, 2016 at 5:18 PM, Stephan Ewen wrote: > Okay, here are the

Re: release of task slot

@Gyula Do you see log messages about quarantined actor systems? There may be an issue with Akka Death watches that once the connection is lost, it cannot be re-established unless the TaskManager is restarted

Re: release of task slot

Okay, here are the docs for the Akka version we are using: http://doc.akka.io/docs/akka/2.3.14/scala/remoting.html#Lifecycle_and_Failure_Recovery_Model It says that after a remote deathwatch trigger, the actor system must be restarted before it can connect again. We probably need to do the

Re: Flink cluster and Java 8

I see. Did you perform a full "mvn clean package -DskipTests" after you changed the source level to 1.8? On Thu, Feb 4, 2016 at 4:33 PM, Flavio Pompermaier wrote: > Flink compiles correctly using java 8 as long as you leave java 1.7 source > and target in the maven java

Re: Flink_Kafka

Hi, I don't understand what you are trying to achieve. If you want to read the topic from the beginning, use a different group.id. Flink should consume data from the topic when you produce something into it. As you can see from the log statement, its at the "INFO" log level, hence its not an

RE: Distribution of sinks among the nodes

Don’t we need to set the number of slots to 24 (4 sources + 16 mappers + 4 sinks) ? Or is there a way not to set the number of slots per TaskManager instead of globally so that they are at least equally dispatched among the nodes ? As for the sink deployment : that’s not good news ; I mean we