Re: DataStream with one DataSource and two different Sinks with diff. schema.

Hi Marke, You can use split() and select() as is shown here[1]. Best, Hequn [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/operators/#datastream-transformations On Sat, Nov 10, 2018 at 12:23 AM Marke Builder wrote: > Hi, > > what is the recommended way to implement the

Re: FLINK Kinesis Connector Error - ProvisionedThroughputExceededException

If it's running in parallel aren't you just adding readers which maxes out your provisioned throughput? probably doesn't belong in here but rather a Kinesis thing, but i suggest increasing your number of shards? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: java.io.IOException: NSS is already initialized

Hey Hao Sun, - Is this an intermittent failure or permanent? The logs indicate that some checkpoints completed before the error occurs (e.g. checkpoint numbers are greater than 1). - Which Java versions are you using? And which Java image? I've Googled similar issues that seem to be related to

Re: Manually clean SQL keyed state

Thanks Fabian! -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: AvroInputFormat Serialisation Issue

Hi, Changing the classloader config to parent-first solved the issue. Regards, Vinay Patil On Wed, Nov 7, 2018 at 7:25 AM Vinay Patil wrote: > Hi, > > Can someone please help here. > > On Nov 6, 2018 10:46 PM, "Vinay Patil [via Apache Flink User Mailing List > archive.]" wrote: > >> Hi, >>

Re: Flink Web UI does not show specific exception messages when job submission fails

+1! On Fri, Nov 9, 2018 at 1:34 PM Gary Yao wrote: > Hi, > > We only propagate the exception message but not the complete stacktrace > [1]. > Can you create a ticket for that? > > Best, > Gary > > [1] >

Re: Flink Web UI does not show specific exception messages when job submission fails

Hi, We only propagate the exception message but not the complete stacktrace [1]. Can you create a ticket for that? Best, Gary [1]

Re: Never gets into ProcessWindowFunction.process()

Hi Gary, Bang on the money. I did not have an assigned Watermark and once I put that in, the code entered the process() method. Thx a ton for your help.Life-saver DataStream kinesisStream = env .addSource(kinesisConsumer) .assignTimestampsAndWatermarks(new

Re: Per job cluster doesn't shut down after the job is canceled

Hi Paul, Can you share the complete logs, or at least the logs after invoking the cancel command? If you want to debug it yourself, check if MiniDispatcher#jobReachedGloballyTerminalState [1] is invoked, and see how the jobTerminationFuture is used. Best, Gary [1]

Re: Never gets into ProcessWindowFunction.process()

Hi, You are using event time but are you assigning watermarks [1]? I do not see it in the code. Best, Gary [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kinesis.html#event-time-for-consumed-records On Fri, Nov 9, 2018 at 6:58 PM Vijay Balakrishnan wrote: > Hi, >

Re: Never gets into ProcessWindowFunction.process()

Hi, Any help is appreciated.Dug into this. *I can see the deserialized output log from FlinkKinesisConsumer deserialization but it keeps looping to pull from Kinesis Stream but never gets into the Windowing operation for process() or apply().* FlinkKinesisConsumer seems to be stuck in a loop

Re: Never gets into ProcessWindowFunction.process()

Hi Gary, Just posted the code.Pls let me know if that clarifies the problem. Have been digging into how the FlinkKinesisConsumer deserialized output gets passed into the process() or apply() method to no avail. The coding pattern I used matches all the fink-examples I have seen for Flink 1.6.1

Re: Never gets into ProcessWindowFunction.process()

Hi, If the job is actually running and consuming from Kinesis, the log you posted is unrelated to your problem. To understand why the process function is not invoked, we would need to see more of your code, or you would need to provide an executable example. The log only shows that all offered

DataStream with one DataSource and two different Sinks with diff. schema.

Hi, what is the recommended way to implement the following use-case for DataStream: One data sink, same map() functions for parsing and normalization and different map() function for format and two different sinks for the output? The (same)data must be stored in both sinks. And I prefere one job

Re: Where is the "Latest Savepoint" information saved?

Can we add an option to allow job cluster mode to start from the latest save point? Otherwise I have to somehow get the info from ZK, before job cluster's container started by K8s. On Fri, Nov 9, 2018, 01:29 Paul Lam wrote: > Hi Hao, > > The savepoint path is stored in ZK, but it’s in binary

Flink Question

I am having problems with the Flink Kinesis adapter. I have some native KCL code that works fine. I want to replace the .addSource with the CSV String data that is coming in from my KCL code. How can I do that? // Consume the data streams from AWS Kinesis stream DataStream dataStream =

FLINK Kinesis Connector Error - ProvisionedThroughputExceededException

I am getting this error from the Flink Kinesis Connector. I have a native KCL app running in parallel with no problems. Any help would be appreciated Thanks so much!! Steve flink-sbistl919-taskexecutor-0-CACSVML-15736.log:2018-11-09 07:46:11,579 WARN

Re: flink-1.6.1 :: job deployment :: detached mode

To verify it just add something after the env.execute() in the wordCount, like a log or a system out or try to run the program twice and submit it using the rest API or Web UI. You'll see that the program will run until the first call to execute, then nothing else will happen

Re: flink-1.6.1 :: job deployment :: detached mode

Hi Mike, the job seems to run. It might indeed only be a problem with shutting down the ZooKeeper utils on the client side after the job has been submitted. I will try to reproduce it locally. Keep us posted on the state of CURATOR-466 if something should change. Cheers, Till On Thu, Nov 8,

Re: Task Manager allocation issue when upgrading 1.6.0 to 1.6.2

Hi Cliff, this sounds not right. Could you share the logs of the Yarn cluster entrypoint with the community for further debugging? Ideally on DEBUG level. The Yarn logs would also be helpful to fully understand the problem. Thanks a lot! Cheers, Till On Thu, Nov 8, 2018 at 9:59 PM Cliff Resnick

ProvisionedThroughputExceededException

I am trying to use the Kinesis Connector and getting the following error message. I am not getting any data at all because of this. I have a native KCL client running in parallel and it is receiving data fine. Any thoughts? flink-sbistl919-taskexecutor-0-CACSVML-15736.log:2018-11-09

Re: Queryable state when key is UUID - getting Kyro Exception

Could you send us a small example program which we can use to reproduce the problem? Cheers, Till On Fri, Nov 9, 2018 at 6:57 AM Jayant Ameta wrote: > Yeah, it IS using Kryo serializer. > > Jayant Ameta > > > On Wed, Nov 7, 2018 at 9:57 PM Till Rohrmann wrote: > >> Hi Jayant, could you check

Re: Error with custom `SourceFunction` wrapping `InputFormatSourceFunction` (Flink 1.6)

Hi, I think for this case a model that is similar to how the Streaming File Source works should be good. You can have a look at ContinuousFileMonitoringFunction and ContinuousFileReaderOperator. The idea is that the first emits splits that should be processed and the second is responsible for

Re: Stopping a streaming app from its own code : behaviour change from 1.3 to 1.6

Hi Arnaud, Thanks for reporting the issue! Best, Fabian Am Do., 8. Nov. 2018 um 20:50 Uhr schrieb LINZ, Arnaud < al...@bouyguestelecom.fr>: > 1.FLINK-10832 > > Created (with heavy difficulties as typing java code in a jira description >

Re: Always trigger calculation of a tumble window in Flink SQL

Hi, SQL does not support any custom triggers or timers. In general, computations are performed when they are complete with respect to the watermarks (applies for GROUP BY windows, OVER windows, windowed and time-versioned joins, etc. Best, Fabian Am Fr., 9. Nov. 2018 um 05:08 Uhr schrieb

Re: Manually clean SQL keyed state

Hi Shahar, That's not possible at the moment. The SQL API does not provide any knobs to control state size besides the idle state retention. The reason is that it aims to be as accurate as possible. In the future it might be possible to provide more information to the system (like constraints in

Re: Where is the "Latest Savepoint" information saved?

Hi Hao, The savepoint path is stored in ZK, but it’s in binary format, so in order to retrieve the path you have to deserialize it back to some Flink internal object. A better approach would be using REST api to get the path. You could find it here[1]. [1]

Re: FlinkCEP, circular references and checkpointing failures

Thank you, Stefan. Any ideas on when can we expect 1.6.3 release? On Thu, Nov 8, 2018 at 4:28 PM Stefan Richter wrote: > Sure, it is already merged as FLINK-10816. > > Best, > Stefan > > On 8. Nov 2018, at 11:53, Shailesh Jain > wrote: > > Thanks a lot for looking into this issue Stefan. > >