Flink with Mesos: Fetcher error

I am trying to configure Flink to work on top of Mesos. I am using Flink release-1.3. I am using DCOS 1.9's underlying mesos which is version 1.2. I am able to start Flink without any issues when the taskmanager starts on the same host as that of appmaster. But when the taskmanager is launched on

Re: SingleOutputStreamOperator addsink Error

bq. new SinkFunction(){ Note the case in JsonObject. It should be JSONObject FYI On Thu, Jun 8, 2017 at 1:27 PM, G.S.Vijay Raajaa wrote: > Hi, > > I am trying to pass the SingleOutputStreamOperator to a custom sink. I am > getting an error while implementing the same.

Re: Flink on kubernetes -> shell deployment

Hi Nico, thanks for your help. $ kubectl exex -it /bin/bash that was what I was looking for. This command provides a shell directly into my job-manager instance. Best, Marc > Am 08.06.2017 um 12:05 schrieb Nico Kruber : > > If you have access to the web dashboard,

Re: In-transit Data Encryption in EMR

Hi Guys, I am able to setup SSL correctly, however the following command does not work correctly and results in the error I had mailed earlier flink run -m yarn-cluster -yt deploy-keys/ TestJob.jar Few Doubts: 1. Can anyone please explain me how do you test if SSL is working correctly ?

Re: Fink: KafkaProducer Data Loss

I built Flink v1.3.0 with cloudera hadoop and am able to see the data loss. Here are the details: *tmOneCloudera583.log* Received session window task: *2017-06-08 15:10:46,131 INFO org.apache.flink.runtime.taskmanager.Task -

Re: Painful KryoException: java.lang.IndexOutOfBoundsException on Flink Batch Api scala

Hi guys, thank you for your interest. Yes @Flavio, I tried both 1.2.0 and 1.3.0 versions. Following Gordon suggestion I tried to put setReference to false but sadly it didn't help. What I did then was to declare a custom serializer as the following: class BlockSerializer extends

Re: Error running Flink job in Yarn-cluster mode

I'm no expert here, but are 4 yarn containers/task managers (-yn 4) not too many for 3 data nodes (=3 dn?)? also, isn't the YARN UI reflecting its own jobs, i.e. running flink, as opposed to running the actual flink job? or did you mean that the flink web ui (through yarn) showed the submitted

Re: [POLL] Who still uses Java 7 with Flink ?

Hi all, as promised in March, I want to revive this discussion! Our users are begging for Scala 2.12 support [1], migration to Akka 2.4 would solve a bunch of shading / dependency issues (Akka 2.4 will remove Akka's protobuf dependency [2][3]) and generally Java 8's new language features all

Error running Flink job in Yarn-cluster mode

I am trying to run a flink job on our cluster with 3 dn and 1 nn. I am usng the following command line argument to run this job, but I get an exception saying "Could not connect to the leading JobManager. Please check that the JobManager is running" ... what could I be doing wrong? Surprisingly,

Re: Streaming job monitoring

Hi Chesnay, this is basically my job: TextInputFormat input = new TextInputFormat(new Path(jsonDir, fileName)); DataStream json = env.createInput(input, BasicTypeInfo.STRING_TYPE_INFO); json.addSink(new ElasticsearchSink<>(userConf, transportAddr, sink)); JobExecutionResult jobInfo =

Re: Queries regarding FlinkCEP

Hi, Can anyone check, whether they can reproduce this issue on their end? There's no log yet as t what is happening. Any idea to debug this issue is well appreciated. Regards, Biplob -- View this message in context:

Re: Deterministic Update

yes - you need to implement the CheckpointedFunction interface. (as an example: our BucketingSink uses this) Nico On Thursday, 8 June 2017 06:44:10 CEST rhashmi wrote: > Is there any possibility to trigger sink operator on completion of > checkpoint? > > > > -- > View this message in context:

Re: Streaming job monitoring

Hello Flavio, I'm not sure what source you are using, but it looks like the ContinouosFileMonitoringSource which works with 2 operators. The first operator (what is displayed as the actual Source) emits input splits (chunks of files that should be read) and passes these to the second operator

Re: Flink on kubernetes -> shell deployment

If you have access to the web dashboard, you probably have access to the Jobmanager in general and can submit jobs from your command line by passing flink run --jobmanager ... I've looped in Patrick in case I am missing something kubernetes-specific here. Nico On Wednesday, 7 June 2017

Re: Use Single Sink For All windows

How about using asynchronous I/O operations? https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/ asyncio.html Nico On Tuesday, 6 June 2017 16:22:31 CEST rhashmi wrote: > because of parallelism i am seeing db contention. Wondering if i can merge > sink of multiple windows

Streaming job monitoring

Hi to all, we've successfully ran our first straming job on a Flink cluster (with some problems with the shading of guava..) and it really outperforms Logstash, from the point of view of indexing speed and easiness of use. However there's only one problem: when the job is running, in the Job

Re: Painful KryoException: java.lang.IndexOutOfBoundsException on Flink Batch Api scala

@Flavio, doesn’t this look like the exception you often encountered a while back? If I remember correctly that was fixed by Kurt, right? Best, Aljoscha > On 7. Jun 2017, at 18:11, Tzu-Li (Gordon) Tai wrote: > > Hi Andrea, > > I did some quick issue searching, and it