Re: Regarding Keyed State of Connected Stream

Hi Garvit, I think you don't need to lock it, they are executed in the same thread sync. Best, Sihua 在2018年05月25日 10:26，Garvit Sharma 写道： Hi, Let's consider, I have two keyed streams one for rules and another for data and I have created a connected stream. I am maintaining a managed

Re:Kryo Exception

Hi, FYI, this is the JIRA ticket for the issue:  https://issues.apache.org/jira/browse/FLINK-8836 Yes, this seems to be only included in 1.5.0 (to be released), and 1.4.3 (there has been no discussion on releasing that yet). It could also be possible that the reported issue was caused by 

Re:Kryo Exception

Hi, this looks like the bug "when duplicating a KryoSerializer does not duplicate registered default serializers", and this has been fixed on the branch master, 1.5.0, and 1.4.x. But, unfortunately not included in 1.4.2(because this bug was discovered after 1.4.2 release). @Stefan plz correct

Regarding Keyed State of Connected Stream

Hi, Let's consider, I have two keyed streams one for rules and another for data and I have created a connected stream. I am maintaining a managed keyed state (ValueState), rules stream will keep updating the state and data stream will keep reading from it. Do I need to take a lock explicitly

Re: Multiple stream operator watermark handling

On Thu, May 24, 2018 at 9:20 AM, Elias Levy wrote: > On Thu, May 24, 2018 at 7:26 AM, Piotr Nowojski > wrote: > >> From top of my head I can imagine two solutions: >> >> 1. Override the default behaviour of the operator via for example >>

Kryo Exception

Hello, We're using Flink version 1.4.2. Our Flink job runs pretty well most of the time. But sometimes we see exceptions in the Kryo serializer. The timing on when the exceptions would occur seems pretty random. Sometimes we don't see any exceptions for 5 days. Sometimes we get exceptions

Re: Multiple stream operator watermark handling

On Thu, May 24, 2018 at 7:26 AM, Piotr Nowojski wrote: > From top of my head I can imagine two solutions: > > 1. Override the default behaviour of the operator via for example > org.apache.flink.streaming.api.datastream.ConnectedStreams#transform > That seems the safer,

Re: logging question

I finally sorted my problem out. Using the CLI instead of the Web UI for debugging with simple System.out.println() statements. I noticed that a local installation launched with start-cluster.sh worked A-OK. In order to reproduce a simple learning environment with docker images, I ended up

Re: When is 1.5 coming out

Hi, There will be a new release candidate out today, but again, exact release depends if we find some release blocking bugs or not. In the mean time you could try out and test the RC :) Piotrek > On 24 May 2018, at 16:33, Vishal Santoshi wrote: > > Thanks. Do we

Re: When is 1.5 coming out

Thanks. Do we have a timeline for when we would be out with 1.5. It seems got delayed again ? On Thu, May 24, 2018 at 8:07 AM, Piotr Nowojski wrote: > It seems like Flink-9349 will make it to the 1.5, because RC5 was > cancelled. > > Piotrek > > > On 24 May 2018, at

Re: Multiple stream operator watermark handling

Hi, From top of my head I can imagine two solutions: 1. Override the default behaviour of the operator via for example org.apache.flink.streaming.api.datastream.ConnectedStreams#transform 2. Can you set control stream’s watermark to Watermark#MAX_WATERMARK or maybe Watermark#MAX_WATERMARK - 1

Re: How to restore state from savepoint with flink SQL

Hi Yan, You are right. That's an unnecessary limitation at the moment and should be fixed. The UIDs should only depend on the query itself and not on any preceding or subsequent operators. The JIRA ticket to track the issue is FLINK-6966. Best, Fabian 2018-05-24 0:34 GMT+02:00 Yan Zhou [FDS

Re: Akka Http used in custom RichSourceFunction

Hi, Please take a look on https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/debugging_classloading.html . Especially check if you are using child-first class loading

Multiple stream operator watermark handling

Is there mechanism for a multiple stream operator to ignore watermarks from one of the streams? The use case is a multiple stream operator that consumes a primary stream and a secondary control stream. The control stream may only receive messages in rare occasion, and possibly never. The

Re: FlinkKinesisProducer weird behaviour

Hi, I’m glad that you have figured it out. Unfortunately it’s almost impossible to mention in our documentation all of the quirks of connectors that we are using, since it would more or less finally come down to fully coping their documentation :( However I created a small PR that mentions

Akka Http used in custom RichSourceFunction

Hi All, I wrote a custom source function (RichSourceFunction) which connects to a web socket using the Akka Http Library. The job using this source runs fine on a local environment until upon shutdown I see the following error in the log: "Exception in thread "main" java.lang.NoSuchMethodError:

Re: FlinkKinesisProducer weird behaviour

Hi, Thanks Piotr for your response. I've further investigated the issue and found the root cause. There are 2 possible ways to produce/consume records to/from Kinesis: 1. Using the Kinesis Data Streams service API directly 2. Using the KCL & KPL. The FlinkKinesisProducer uses the AWS

Re: When is 1.5 coming out

It seems like Flink-9349 will make it to the 1.5, because RC5 was cancelled. Piotrek > On 24 May 2018, at 14:03, Vishal Santoshi wrote: > > Flink-9349 can we out this to 1.5 release. > > On Thu, May 24, 2018, 7:59 AM Vishal Santoshi

Re: When is 1.5 coming out

Flink-9349 can we out this to 1.5 release. On Thu, May 24, 2018, 7:59 AM Vishal Santoshi wrote: > Thanks, I'll check it out. > > On Wed, May 23, 2018, 9:08 AM Fabian Hueske wrote: > >> Hi Vishal, >> >> Release candidate 5 (RC5) has been published

Re: When is 1.5 coming out

Thanks, I'll check it out. On Wed, May 23, 2018, 9:08 AM Fabian Hueske wrote: > Hi Vishal, > > Release candidate 5 (RC5) has been published and the voting period ends > later today. > Unless we find a blocking issue, we can push the release out today. > > FYI, if you are

Re: Batch job stuck in Canceled state in Flink 1.5

Thanks! Till. I'll give a try on your suggestions and update the thread. On Wed, May 23, 2018 at 4:43 AM, Till Rohrmann wrote: > Hi Amit, > > it looks as if the current cancellation cause is not the same as the > initially reported cancellation cause. In the current case,

Re: sharebuffer prune code

Can you explain it more explictly? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: increasing parallelism increases the end2end latency in flink sql

Hi Yan, SQL should not be the cause here. It is true that Flink removes the timestamp from a record when entering the SQL API but this timestamp is set again before time-based operations such as OVER windows. Watermarks are not touched. I think your issue is related to [2]. One explanation

Re: Starting beam pipeline from savepoint

Hi, I am not sure, but it probably brings down to the question, whether you can restore from a save point when using RemoteEnvironment - and answer for this question is unfortunately no. Maybe Aljoscha will know something more. Piotrek > On 23 May 2018, at 19:25, borisbolvig

Re: FlinkKinesisProducer weird behaviour

Hi, Have you tried to write the same records, with exactly the same configuration to the Kinesis, but outside of Flink (with some standalone Java application)? Piotrek > On 24 May 2018, at 09:40, Rafi Aroch wrote: > > Hi, > > We're using Kinesis as our input & output

FlinkKinesisProducer weird behaviour

Hi, We're using Kinesis as our input & output of a job and experiencing parsing exception while reading from the output stream. All streams contain 1 shard only. While investigating the issue I noticed a weird behaviour where records get a PartitionKey I did not assign and the record Data is

Re: Efficient Stateful Processing with Time-Series Data and Enrichments

Hi Sihua, I will test keying by device ID. I was trying to implement this suggestion: https://stackoverflow.com/a/49395606, but I guess that may be unnecessary in my case. Thanks, Mike On Wed, May 23, 2018 at 11:30 PM, sihua zhou wrote: > Hi Mike, > if I'm not

Re:Efficient Stateful Processing with Time-Series Data and Enrichments

Hi Mike, if I'm not misunderstand, you are doing aggregation for every device on the stream. You mentioned that, you want to use the MapState to store the state for each device ID? this is a bit confusing to me, I think what you need maybe a ValueState. In flink, every keyed state(Value,