Re: Error during Kafka connection

Hi, You mentioned that your kafka broker is behind a proxy. This could be a problem, because when the client try to get the cluster's topology, it will get the brokers ' private addresses , which is not reachable. Regards, Kien On Aug 11, 2017, 18:18, at 18:18, "Tzu-Li (Gordon) Tai"

Re: No file system found with scheme s3

Shouldn't the config key be : org.apache.hadoop.fs.s3.S3FileSystem Cheers On Fri, Aug 11, 2017 at 5:38 PM, ant burton wrote: > Hello, > > After following the instructions to set the S3 filesystem in the > documentation (https://ci.apache.org/projects/flink/flink-docs- >

No file system found with scheme s3

Hello, After following the instructions to set the S3 filesystem in the documentation (https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/aws.html#set-s3-filesystem ) I encountered the

Re: Standalone cluster - taskmanager settings ignored

Hi Greg, I guess I restarted the cluster too fast. Combined with a high cpu inside the cluster. I tested it again few minutes ago and there was no issue! With „$ jps“ I checked if there any Java process -> there wasn’t But if the master don’t know slave5, how can slave5 reconnect to the

Re: [EXTERNAL] Re: difference between checkpoints & savepoints

Thanks for the discussion. That answered many questions I have. Also, in the same line, can someone detail the difference between State Backend & External checkpoint? Also, programmatic API, thru which methods we can configure those. Regards, Raja. From: Stefan Richter

FsStateBackend with incremental backup enable does not work with Keyed CEP

Hi, I am running fling 1.3.1 on EMR. But I am getting this exception after running for a while. java.lang.RuntimeException: Exception occurred while processing valve output watermark: at

Re: Standalone cluster - taskmanager settings ignored

Hi Marc, By chance did you edit the slaves file before shutting down the cluster? If so, then the removed worker would not be stopped and would reconnect to the restarted JobManager. Greg > On Aug 11, 2017, at 11:25 AM, Kaepke, Marc wrote: > > Hi, > > I have a

Re: Standalone cluster - taskmanager settings ignored

I start my cluster with: bigdata@master:/usr/lib/flink-1.3.2$ ./bin/start-cluster.sh Starting cluster. Starting jobmanager daemon on host master. Starting taskmanager daemon on host master. Starting taskmanager daemon on host slave1. Starting taskmanager daemon on host slave3. And if I stop it:

Standalone cluster - taskmanager settings ignored

Hi, I have a cluster of 4 dedicated machines (no VMs). My previous config was: 1 master and 3 slaves. Each machine provides a task- or jobmanager. Now I want to reduce my cluster and have 1 master and 3 slaves, but one machine provides a jobmanager and one task manager in parallel. I changed

Re: stream partitioning to avoid network overhead

Hi Karthik, maybe I'm misunderstanding, but there are a few things in your description that seem strange to me: - Your "slow" operator seems to be slow not because it's compute-heavy, but because it's waiting for a response. Is AsyncIO (

Re: Overwrite environment variables in single-job deployment on YARN

Hi, Have you tried ./bin/flink run -m yarn-cluster -yD yarn.taskmanager.env.JAVA_HOME=“/opt/jre1.8.0” -yD can be used to dynamically specify settings. Best, Aljoscha > On 11. Aug 2017, at 13:20, Mariusz Wojakowski wrote: > > Hi, > > I want to run Flink job on YARN

Re: Using latency markers

Yes, they are enabled by default I think. Gyula On Fri, Aug 11, 2017, 14:14 Aljoscha Krettek wrote: > It seems you have to enable latency tracking > via ExecutionConfig.setLatencyTrackingInterval(...). This will make the > sources emit latency tokens, which then in turn

Re: Using latency markers

It seems you have to enable latency tracking via ExecutionConfig.setLatencyTrackingInterval(...). This will make the sources emit latency tokens, which then in turn update the latency metric: https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/metrics.html#latency-tracking

Re: Evolving serializers and impact on flink managed states

Hi Stefan, Thanks a lot for such a helpful response. That really made thing a lot clearer for me. Although at this point I have one more and probably last question. According to the Flink documentation, [Attention] Currently, as of Flink 1.3, if the result of the compatibility check

Re: Error during Kafka connection

I just tried to use telnet to public ip:port from outside and it works. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Error-during-Kafka-connection-tp14822p14829.html Sent from the Apache Flink User Mailing List archive. mailing list

Re: Error during Kafka connection

No, there should be no difference between setting it up on Ubuntu or OS X. I can’t really tell any anything suspicious from the information provided so far, unfortunately. Perhaps you can try first checking that the Kafka topic is consumable from where you’re running Flink, e.g. using the

Re: Aggregation based on Timestamp

Hi, Yes, this is definitely doable in Flink, and should be very straightforward. Basically, what you would do is define a FlinkKafkaConsumer source for your Kafka topic [1], following that a keyBy operation on the hostname [2], and then a 1-minute time window aggregation [3]. At the end of

Re: Error during Kafka connection

the kafka version I use is the latest (0.11.0.0). But to be honestly, also locally I use 0.11.0.0 and in that case it works correctly. Anyway the last kafka connector on flink is designed for kafka 0.10.x.x I use OS X locally and Ubuntu on the cluster. It has importance? -- View this message

Re: Error during Kafka connection

Hi, AFAIK, Kafka group coordinators are supposed to always be marked dead, because we use static assignment internally and therefore Kafka's group coordination functionality is disabled. Though it may be obvious, but to get that out of the way first: are you sure that the Kafka installation

Error during Kafka connection

Hi, In the last week I have correctly deployed a flink program which get data from a kafka broker on my local machine. Now I'm trying to produce the same thing but moving the kafka broker on a cluster. I didn't change any line of code, I report it here: DataStream>