RE: Problems to use toAppendStream

Hi Actually I have also line “import org.apache.flink.streaming.api.scala” on my code, but this line seems to be highlighted weaker in window of IDEA IntelliJ editor. What does this mean ? But the same errors will still be generated. Esa From: Fabian Hueske [mailto:fhue...@gmail.com] Sent:

Re: Share state across operators

Hey Timo! I am using Java for my implementation and I have found this article [1] in stackoverflow for simulating the Either in Java. Now, for my case, I have a coordinator instance (parallelism = 1) that needs to both distribute incoming tuples in a specific way, but also needs to

Re: Use keyBy to deterministically hash each record to a processor/task/slot

Hello! I have used up till now your method to generate keys for the .keyBy() function, in order to specifically know at which processor id each tuple will end up in the end (w.r.t the key % #procs operation). Though I had to shift to Java cause the documentation is better. And I implemented your

Re: Deep Copy in FLINK, Kryo Copy is used in the different operator

@Gábor Gévay, Thanks Gábor I just use flink in produce environment, but the performance is not good, especially in aggregation. At the beginning I used Java serialization, but it does not work well. Maybe I do not understood flink very well then. I will try change the serialization

Re: Deep Copy in FLINK, Kryo Copy is used in the different operator

@Aljoscha Krettek, Thanks Aljoscha, I will try this way to test the performance. Last 7 days is chinese spring fastival, sorry for response you so late. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Machine Learning: Flink and MOA

Hi guys, I know there is FlinkML to do some machine learning with Flink but it works on DataSet and not on DataStream, there is also SAMOA which can run on Flink but I find it a bit too complicated. I wanted to see if it would be easy to plug directly MOA on Flink and tried to present it in the

Re: SQL materialized upsert tables

Hi Elias, it would be great if you could let us know if the approach works. Btw. I should point out that the join in a query like: SELECT s.tstamp, s.item, s.score, t.source FROM( SELECT item, source, MAX(tstamp), LAST_VAL(score, tstamp) FROM Telemetry GROUP BY item, source ) INNER JOIN

Re: SQL materialized upsert tables

On Wed, Feb 21, 2018 at 3:24 AM, Fabian Hueske wrote: > Hi Elias, > > Flink does not have built-in support for upsert stream -> table > conversions, yet. However, the community is working on that (see FLINK-8545 > [1]). > With a workaround, you can also solve the issue with

Problems to use toAppendStream

|Hi | |I have tried to solve below Errors for long time, but no succeed yet. Could you give some hint how to solve it ? |Errors in compiling: -- ||Error:(56,46)could not find implicitvalue forevidence parameter of

TaskManager crashes with PageRank algorithm in Gelly

Folks, We are running a simple PageRank algorithm in Gelly with about 1M edges and we are seeing that one the TaskManager just crashes. We suspect it is some configuration issue because each TaskManager has a total of 136GB memory and we have 8 of these. So, the total memory is more than enough.

FINAL REMINDER: CFP for Apache EU Roadshow Closes 25th February

Hello Apache Supporters and Enthusiasts This is your FINAL reminder that the Call for Papers (CFP) for the Apache EU Roadshow is closing soon. Our Apache EU Roadshow will focus on Cloud, IoT, Apache Tomcat, Apache Http and will run from 13-14 June 2018 in Berlin. Note that the CFP deadline

Re: Important (proposed) CEP changes for Flink 1.5.

Hi Vishal, Yes these changes will only affect CEP. Cheers, Kostas > On Feb 21, 2018, at 5:42 PM, Vishal Santoshi > wrote: > > That should be fine by us. We do not plan to do a SP and restore on our prod > CEP pipe, when we move to 1.5 from 1.4. I do hope that all

Re: Important (proposed) CEP changes for Flink 1.5.

That should be fine by us. We do not plan to do a SP and restore on our prod CEP pipe, when we move to 1.5 from 1.4. I do hope that all other pipes will remain back ward compatible ( 1.4 to 1.5 ). On Wed, Feb 21, 2018 at 5:06 AM, Kostas Kloudas wrote: > Hi all, > > Currently

Kryo Serializator to override AVRO default

Hi, I have a dataflow with a Kafka source that uses the avro generated class to deserialize: AvroDeserializer avroSchema = new AvroDeserializer<>( MyAvroType.class); FlinkKafkaConsumer011 kafkaReader = new FlinkKafkaConsumer011( "kafkaTopic", avroSchema, properties); From this point

FLIP-6 + Mesos - Questions

According to the docs: The Mesos scheduler currently resides with the JobManager but will be started independently of the JobManager in future versions (see FLIP-6 ). The proposed changes will also add a Dipsatcher

Re: Regarding BucketingSink

Thank you Fabian, What is more important ( and I think you might have addressed it in your post so sorry for being a little obtuse ) is that deleting them does not violate "at-least-once" delivery. And if that is a definite than we are fine with it, though we will test it further. Thanks

Re: Stopping a kafka consumer gracefully (no losing of inflight events, StoppableFunction)

Ok. Thanks a lot of the clarification. That was my initial understanding but then was confused by the "losing in-flight events" wording. On Wed, Feb 21, 2018 at 10:26 AM, Till Rohrmann wrote: > Hi Christophe, > > yes I think you misunderstood the thread. Cancel with

Re: Need to understand the execution model of the Flink

Cardinality and size estimation are fundamental requirements for cost-based query optimization. I hope we will work on this at some point but right now it is not on the roadmap. In case of very complex plans, it might make sense to write an intermediate result to persistent storage and start

Re: Regarding BucketingSink

Hi Vishal, hi Mu, After the savepoint state has been written, the sink might start new .in-progress files. These files are not part of the savepoint but renamed to .pending in close(). On restore all pending files that are part of the savepoint are moved into final state (and possibly truncated).

Strange Kafka consumer behaviour

Hi, I have observed a weird behaviour when changing kafka topics when restoring from a checkpoint. It seems that the job started consuming both the topics from the state, and the new topic that I assigned. This happened while changing from kafka 08 to kafka 10. Is this expected? Thanks, Gyula

Re: Stopping a kafka consumer gracefully (no losing of inflight events, StoppableFunction)

@Bart, I think there is no Flip yet for the proper stop with savepoint implementation. My gut feeling is that the community will soon address this problem since it's a heavily requested feature. Cheers, Till On Wed, Feb 21, 2018 at 10:26 AM, Till Rohrmann wrote: > Hi

Re: Stopping a kafka consumer gracefully (no losing of inflight events, StoppableFunction)

Hi Christophe, yes I think you misunderstood the thread. Cancel with savepoint will never cause any data loss. The only problem which might arise if you have an operator which writes data to an external system immediately, then you might see some data in the external system which originates from