Re: How to perform this join operation?

Till, An issue with your suggestion is that the job state may grow unbounded. You are managing expiration of data from the cache in the operator, but the state is partitioned by the stream key. That means if we no longer observe a key, the state associated with that key will never be removed. In

Re: Weird Kryo exception (Unable to find class: java.ttil.HashSet)

Right now I'm using Flink 1.0.2...to which version should I downgrade? The hardware seems to be ok..how could I detect a faulty hardware? These errors appeared in every run of my job after I moved the temporary directory from ssd to hdd and I extended my pipeline with a dataset that grows as the

Re: inheritance of Program interface in Program.java in org.apache.flink.api.common

then this.program of PackageProgram's object has null. [1] in run() of CliFrontend.java, invokes executeProgramBlocking(program, client, userParallelism) [2] and this method invokes client.runBlocking(program, parallelism) [3] and this method invokes runBlocking(prog.getPlanWithJars(),

Logging Exceptions

Hello! Using flink 1.0.2, I noticed that exceptions thrown during a flink program would show up on the flink dashboard in the 'Exceptions' tab. That's great! However, I don't think flink currently logs this same exception. I was hoping there would be an equivalent `log.error` call so that third

Re: Weird Kryo exception (Unable to find class: java.ttil.HashSet)

The problem seems to occur quite often. Did you update your Flink version recently? If so, could you try to downgrade and see if the problem disappears. Is it otherwise possible that it is cause by faulty hardware? 2016-05-20 18:05 GMT+02:00 Flavio Pompermaier : > This

Re: Weird Kryo exception (Unable to find class: java.ttil.HashSet)

This time (Europed instead of Europe): java.lang.Exception: The data preparation for task 'CHAIN GroupReduce (GroupReduce at createResult(PassaggioWithComprAndVend.java:132)) -> Map (Key Extractor)' , caused an error: Error obtaining the sorted input: Thread 'SortMerger spilling thread'

Re: Run jar job in local cluster

Nope - thanks for advice ;) I will try that tomorrow 2016-05-20 11:24 GMT+02:00 Ufuk Celebi : > I would suggest to set the log level to DEBUG and check the logs why > the client can not connect to your elastic search cluster. Did you try > that? > > On Mon, May 9, 2016 at 3:54

[RichFlattMapfunction] Configuration File

Hi folks I'm extending a RichFlatMapFunction in order to use states on a keyed stream. Concerning this i have two questions: 1. I have a var state_item: ValueState[Option[String]] as a local variable in this class. Initialized with state_item = getRuntimeContext.getState(new

Re: Weird Kryo exception (Unable to find class: java.ttil.HashSet)

This time another error (rerialization instead of serialization): com.esotericsoftware.kryo.KryoException: Unable to find class: it.okkam.flink.entitons.*rerialization*.pojo.EntitonQuadPojo at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)

Re: TypeExtractor:1672 - class org.joda.time.DateTime is not a valid POJO type

Ok..great to hear that! Thanks to all for the support On Fri, May 20, 2016 at 3:53 PM, Ufuk Celebi wrote: > On Fri, May 20, 2016 at 3:27 PM, Aljoscha Krettek > wrote: > > I think it might just be a warning. When using Kryo it is in the end a > >

Re: TypeExtractor:1672 - class org.joda.time.DateTime is not a valid POJO type

On Fri, May 20, 2016 at 3:27 PM, Aljoscha Krettek wrote: > I think it might just be a warning. When using Kryo it is in the end a > GenericTypeInformation but the TypeAnalyzer might still initially try to > analyze it as a POJO. Yes, that's correct. To get back at the

Re: TypeExtractor:1672 - class org.joda.time.DateTime is not a valid POJO type

Indeed you can test this problem with: import org.apache.flink.api.java.ExecutionEnvironment; import org.joda.time.DateTime; import de.javakaffee.kryoserializers.jodatime.JodaDateTimeSerializer; public class DateTimeError { public static void main(String[] args) throws Exception {

Re: flink async snapshots

Thats correct. With the fully async option the checkpoints take longer but you don't impact ongoing processing of elements. With the semi-async method snapshots take a shorter time but during the synchronous part no element processing can happen. On Fri, 20 May 2016 at 15:04 Abhishek Singh

Re: TypeExtractor:1672 - class org.joda.time.DateTime is not a valid POJO type

I think it might just be a warning. When using Kryo it is in the end a GenericTypeInformation but the TypeAnalyzer might still initially try to analyze it as a POJO. On Fri, 20 May 2016 at 12:51 Ufuk Celebi wrote: > I tried to reproduce this and if you set up the depdency > >

Re: join stream with last available element of other stream

Aljoscha answered this in the other thread you started for this ("'Last One' Window") On Fri, May 20, 2016 at 12:43 PM, Artem Bogachev wrote: > Hi, > > I’ve faced a problem trying to model our platform using Flink Streams. > > Let me describe our model: > > // Stream

Re: custom sources

On Fri, May 20, 2016 at 3:12 PM, Abhishek Singh wrote: > Thanks. I am still in theory/evaluation mode. Will try to code this up to > see if checkpoint will become an issue. I do have a high rate of ingest and > lots of in flight data. Hopefully flink back pressure

Re: custom sources

Thanks. I am still in theory/evaluation mode. Will try to code this up to see if checkpoint will become an issue. I do have a high rate of ingest and lots of in flight data. Hopefully flink back pressure keeps this nicely bounded. I doubt it will be a problem for me - because even spark is

Re: flink async snapshots

Yes. Thanks for explaining. On Friday, May 20, 2016, Ufuk Celebi wrote: > On Thu, May 19, 2016 at 8:54 PM, Abhishek R. Singh > > wrote: > > If you can take atomic in-memory copies, then it works (at the cost of > > doubling your

Re: unsubscribe

You have to send this to user-unsubscr...@flink.apache.org You've sent this this to user@f.a.o instead. On Fri, May 20, 2016 at 1:01 PM, Christophe Salperwyck wrote: >

unsubscribe

Re: custom sources

On Thu, May 19, 2016 at 7:48 PM, Abhishek R. Singh wrote: > There seems to be some relationship between watermarks, triggers and > checkpoint that is someone not being leveraged. Checkpointing is independent of this, yes. Did the state size become a problem for

Re: TypeExtractor:1672 - class org.joda.time.DateTime is not a valid POJO type

I tried to reproduce this and if you set up the depdency de.javakaffee kryo-serializers 0.28 and register the Kryo type as suggested you should not see any log messages (e.g. the type should be treated as a generic type and not a pojo type). Does the program

Re: Weird Kryo exception (Unable to find class: java.ttil.HashSet)

Hi Ufuk, my records could be quite large Pojos (I think some MB). The only thing I do to configure Kryo is: env.registerTypeWithKryoSerializer(DateTime.class, JodaDateTimeSerializer.class ); Best, Flavio On Fri, May 20, 2016 at 12:38 PM, Ufuk Celebi wrote: > @Stefano: the

join stream with last available element of other stream

Hi, I’ve faced a problem trying to model our platform using Flink Streams. Let me describe our model: // Stream of data, ex. stocks: (AAPL, 100.0), (GZMP, 100.0) etc. val realData: DataStream[(K, V)] = env.addSource(…) // Stream of forecasts (same format) based on some window aggregates val

Re: Weird Kryo exception (Unable to find class: java.ttil.HashSet)

@Stefano: the records are serialized anyway for batch jobs. The spilling deserializer is only relevant if single records are very large. How large are your records? In any case, I don't expect this to be the problem. @Flavio: The class name "typo" errors (Vdhicle instead of Vehicle and ttil

Re: inheritance of Program interface in Program.java in org.apache.flink.api.common

On Thu, May 19, 2016 at 4:46 PM, 윤형덕 wrote: > > how can this.mainClass that doesn't override getPlan method that is abstract method of Program interface(program.class) and has only static main method be instantiate as Program? This is only called if the class is actually a

Re: hot deployment of stream processing(event at a time) jobs

I think you are looking for the savepoints feature: https://ci.apache.org/projects/flink/flink-docs-release-1.0/apis/streaming/savepoints.html The general idea is to trigger a savepoint, start the second job from this savepoint (reading from the same topic), and then eventually cancel the first

Re: flink-kafka-connector offset management

Hey Arun! How did you configure your Kafka source? If the offset has been committed and you configured the source to read from the latest offset, the message should not be re-processed. – Ufuk On Fri, May 13, 2016 at 2:19 PM, Arun Balan wrote: > Hi, I am trying to use the

Re: Java heap space error

Indeed I can confirm that I resolved this problem reducing the number of slots per Task Manager (and thus incrementing the available memory of each task)! However from time to time I have serialization issue that I don't know where they come from..it looks like the PjoSerialization has some issue

Re: Java heap space error

The job is running out of heap memory, probably because a user function needs a lot of it (the parquet thrift sink?). You can try to work around it by reducing the amount of managed memory in order to leave more heap space available. On Thu, May 12, 2016 at 6:55 PM, Flavio Pompermaier

Re: Run jar job in local cluster

I would suggest to set the log level to DEBUG and check the logs why the client can not connect to your elastic search cluster. Did you try that? On Mon, May 9, 2016 at 3:54 PM, rafal green wrote: > Dear Sir or Madam, > > Can you tell me why I have a problem with

Re: Weird Kryo exception (Unable to find class: java.ttil.HashSet)

Today I've got this other strange error.. Obviously I don't have a VdhicleEvent class, but a VehicleEvent class :( java.lang.RuntimeException: Cannot instantiate class. at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:407) at

Re: flink async snapshots

On Thu, May 19, 2016 at 8:54 PM, Abhishek R. Singh wrote: > If you can take atomic in-memory copies, then it works (at the cost of > doubling your instantaneous memory). For larger state (say rocks DB), won’t > you have to stop the world (atomic snapshot) and make