Re: OutOfMemoryError in netty local transport

Hi Robert, Just a quick update: The issue has been resolved in the latest Maven 0.10-SNAPSHOT dependency. Cheers, Max On Wed, Sep 30, 2015 at 3:19 PM, Robert Schmidtke wrote: > Hi Max, > > thanks for your quick reply. I found the relevant code and commented it out > for

Re: DataSet transformation

Hi, for that you have to collect the dataset to your local machine and then transform the collection into the array. Note that this only advised for small data sets. Robert On Thu, Oct 1, 2015 at 2:13 PM, Lydia Ickler wrote: > Hi all, > > so I have a case class

DataSet transformation

Hi all, so I have a case class Spectrum(mz: Float, intensity: Float) and a DataSet[Spectrum] to read my data in. Now I want to know if there is a smart way to transform my DataSet into a two dimensional Array ? Thanks in advance, Lydia

kryo exception due to race condition

Hi guys, I hit a Kryo exception while running a process 'crossing' POJOs datasets. I am using the 0.10-milestone-1. Checking the serializer: org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:210) I have noticed that the Kryo instance is reused along

Re: All but one TMs connect when JM has more than 16G of memory

So for anyone who is interested, here are some code references for getting started with Flink on Slurm. I added basic start and stop scripts for Flink on Slurm in my fork: https://github.com/robert-schmidtke/flink/tree/flink-slurm/flink-dist/src/main/flink-bin/bin And I also created an example

Re: kryo exception due to race condition

This looks to me like a bug where type registrations are not properly forwarded to all Serializers. Can you open a JIRA ticket for this? On Thu, Oct 1, 2015 at 6:46 PM, Stefano Bortoli wrote: > Hi guys, > > I hit a Kryo exception while running a process 'crossing' POJOs

Re: All but one TMs connect when JM has more than 16G of memory

I see, thanks for the info. I only have access to my cluster via SLURM and we don't have ssh between our nodes which is why I haven't really considered the Standalone mode. A colleague has set up YARN on SLURM and it was just the easiest to use. I briefly looked into the Flink Standalone mode but

Re: All but one TMs connect when JM has more than 16G of memory

Feel free to contribute a documentation to Flink on how to run Flink on SLURM. On Thu, Oct 1, 2015 at 11:45 AM, Robert Schmidtke wrote: > I see, thanks for the info. I only have access to my cluster via SLURM and > we don't have ssh between our nodes which is why I

Re: OutOfMemoryError in netty local transport

By the way, you might have to use the "-U" flag to force Maven to update its dependencies: mvn -U clean install -DskipTests On Thu, Oct 1, 2015 at 10:19 AM, Robert Schmidtke wrote: > Sweet! I'll pull it straight away. Thanks! > > On Thu, Oct 1, 2015 at 10:18 AM,

Re: All but one TMs connect when JM has more than 16G of memory

Hi Robert, I had a job failure yesterday with what I believe is the setup I have described above. However when trying to reproduce now, the behavior is the same: Flink waiting for resources to become available. So no hard error. Ok, the looping makes sense then. I haven't thought about shared

Re: OutOfMemoryError in netty local transport

Great to hear :) On Thu, Oct 1, 2015 at 11:21 AM, Robert Schmidtke wrote: > I pulled the current master branch and rebuilt Flink completely anyway. > Works like a charm. > > On Thu, Oct 1, 2015 at 11:11 AM, Maximilian Michels wrote: >> >> By the way, you

Re: All but one TMs connect when JM has more than 16G of memory

Hi, there is currently no option for forcing certain containers onto specific machines. For running the JM (or any other YARN container) on the AM host, you first need to have a NodeManager running on the host with the RM. Maybe YARN is smart enough to schedule the small JM container onto that