Re: Flink Mesos Outstanding Offers - trouble launching task managers

Hello, did you resolve this issue? Thanks, Eron Wright On Wed, Jul 12, 2017 at 11:09 AM, Prashant Nayak < prash...@intellifylearning.com> wrote: > > Hi > > We’re running Flink 1.3.1 on Mesos. > > From time-to-time, the Flink app master seems to have trouble with Mesos > offers… At such time, it

Elasticsearch Sink - Error

Hi, I am using elasticsearch 5.4.3 version in my flink project(flink version 1.3.1) Details 1. Using Maven build tool. 2. Running from intellij IDE. 3. Elasticsearch is running on the local machine. Have added the following maven dependency org.apache.flink

Re: Using Azure Blob Storage with Flink

There is HADOOP-14753 which is still Open. FYI On Tue, Aug 29, 2017 at 3:41 PM, Joshua Griffith wrote: > Yes, hadoop-azure and azure-storage are both on the classpath. > hadoop-azure is declared as a dependency in my build.sbt file and I’m using > assembly to copy all

Re: Using Azure Blob Storage with Flink

Yes, hadoop-azure and azure-storage are both on the classpath. hadoop-azure is declared as a dependency in my build.sbt file and I’m using assembly to copy all of the dependencies into a single jar which is submitted to Flink. I suspect the wasb format needs to be explicitly registered with

Re: Using Azure Blob Storage with Flink

Was hadoop-azure jar on the classpath ? Please also see the following from https://hadoop.apache.org/docs/current/hadoop-azure/index.html : The built jar file, named hadoop-azure.jar, also declares transitive dependencies on the additional artifacts it requires, notably the Azure Storage SDK for

Using Azure Blob Storage with Flink

I’m attempting to write to Azure Blob Storage using Flink's FileOutputFormat. I’ve included hadoop-azure within the jar I submit to Flink and configured the paths to be prefixed with

Twitter example

I am trying to run the basic twitter example,  it does deploy correctly and show as RUNNING both in the command line and in the UI.  However I see no output.  I am able to verify that my credentials are correct with another small client I have built that does connect to twitter and prints out

Re: Consuming a Kafka topic with multiple partitions from Flink

Hi Tony, Thanks for your thoughts. I found the issue in my Flink processing chain. I had Kafka partition ids 0, 1, 2 and therefore it was a different issue. I had a keyBy operation before my process operation (which contains my main stream processing logic) and there was only one key being

Re: datastream.print() doesn't works

Using mvn clean now I obtain: Error: Could not find or load main class org.apache.flink.mainProgram.StreamingJob -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/datastream-print-doesn-t-works-tp15223p15239.html Sent from the Apache Flink

Re: datastream.print() doesn't works

I'm afraid I don't know anything about eclipse. Running mvn clean package will first delete all files in the /target directory, and then recompile your code to that very location. It shouldn't affect an IDE in a way that isn't resolvable by rebuilding the project in it. You could also try

Re: datastream.print() doesn't works

Chesnay Schepler wrote > The easiest explanation is that there is nothing to print. > > Since print statements within the select function don't appear in the > logs I assume > that the result of HTM.learn is empty. > > Please check via the webUI or metrics whether any of these operations >

Re: metrics for Flink sinks

Hello, 1. Because no one found time to fix it. In contrast to the remaining byte/record metrics, input metrics for sources / output metrics for sinks have to be implemented for every single implementation with their respective semantics. In contrast, the output metrics are gathered in the

API to launch flink session from Java?

Is there a way to start a yarn session from java? We want to use Kerberos impersonation and call doAs before running the session so it runs using the "correct" credential rather than a fixed one.

Re: Flink session on Yarn - ClassNotFoundException

Hello, The ClassNotFoundException indicates that you are using a Flink version that wasn't compiled against hadoop 2.7. Replacing one part of the hadoop dependency will most likely not cut it (or fail mysteriously down the line), so i would suggest to check the downloads

Re: Default chaining & uid

Hello, That depends a bit on the used version. For 1.3 and above it does not affect chaining; the maps will be chained and setting the UIDs will work as if the maps weren't chained. For 1.2, setting the UID on a chained operator is forbidden and will fail with an exception. On 28.08.2017

Re: Kafka consumer are too fast for some partitions in "flatMap" like jobs

How many partitions does the output topic have? If it has the same number of partitions as the input topic (30), have you considered simply using a custom partitioner for the Kafka sink that uses the input partition number as the output partition number? If the input messages are ordered per

Re: Set Savepoints configuration after cluster bootstrap

Hello, it is not possible to permanently set the savepoint directory after the cluster has started, but the configured value can be overridden when taking a savepoint as described here .

Set Savepoints configuration after cluster bootstrap

Hi, I am using Flink v1.3.1. My question is about how to set the configuration for the savepoints feature. As long as I know the configuration entry `state.savepoints.dir` must be set in the file flink/conf/flink-conf.yaml But I would like to know if it is possible to programmatically

Re: yarn and checkpointing

Checkpoints are only used for recovery during the job execution. If the entire cluster is shutdown and restarted you will need to take a savepoint and restore from that. On 29.08.2017 16:46, Gwenhael Pasquiers wrote: Hi, Is it possible to use checkpointing to restore the state of an app

Re: datastream.print() doesn't works

Don't forget to call env.execute() at the end and make sure you have configured your logger correctly. Regards, Timo Am 29.08.17 um 14:59 schrieb Chesnay Schepler: The easiest explanation is that there is nothing to print. Since print statements within the select function don't appear in the

Re: Atomic savepint and cancel

Hello, savepoint is in general not an atomic operation, it only guarantees that no other checkpoint will be completed between the savepoint and the job cancellation. You can only guarantee that no messages are sent out if you used a sink that supports exactly-once, which as far as i know,

yarn and checkpointing

Hi, Is it possible to use checkpointing to restore the state of an app after a restart on yarn ? From what I've seen it looks like that checkpointing only works within a flink cluster life-time. However the yarn mode has one cluster per app, and (unless the app crashes and is automatically

Re: datastream.print() doesn't works

The easiest explanation is that there is nothing to print. Since print statements within the select function don't appear in the logs I assume that the result of HTM.learn is empty. Please check via the webUI or metrics whether any of these operations actually return records. On 29.08.2017

Atomic savepint and cancel

Hi, I'm a bit new to Flink and I'm trying to figure out what's the best way to make an upgrade for my current running topology without having duplicate messages being sent by the the sink. (One time prior the upgrade and one time after). I thought that the "atomic" part of the savepoint & cancel

datastream.print() doesn't works

Hi, I have a simple datastream of a Tuple2. Unfortunately when I call the print() method. No one output is showed although no errors or exceptions are raised. I want to highlight that I have also other data streams which are correctly printed. This is the stream:

Flink session on Yarn - ClassNotFoundException

Hi, I’m trying to start a flink (1.3.2) session as explained in the docs (https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/yarn_setup.html#start-a-session ), but I keep getting

Re: Issues in recovering state from last crash using custom sink

Hi, If you are not manually doing a savepoint and then restoring from that savepoint you will not restore state. Simply stopping a job and then restarting will not restore state. The regular checkpoints are only used for recovery if a job fails, not for a user-induced shutdown. Best, Aljoscha

Kafka consumer are too fast for some partitions in "flatMap" like jobs

Hello, There is one Flink job which consumes from Kafka topic (TOPIC_IN), simply flatMap / map data and push to another Kafka topic (TOPIC_OUT). TOPIC_IN has around 30 partitions, data is more or less sequential per partition and the job has parallelism 30. So in theory there should be 1:1

RE: Great number of jobs and numberOfBuffers

Hello, Sorry to ask you again, but no idea on this ? -Original Message- From: Gwenhael Pasquiers [mailto:gwenhael.pasqui...@ericsson.com] Sent: lundi 21 août 2017 12:04 To: Nico Kruber Cc: Ufuk Celebi ; user@flink.apache.org Subject: RE: Great

Re: Consuming a Kafka topic with multiple partitions from Flink

Hi Isuru, The way to assign partitions by FlinkKafkaConsumer can refer to this java document. ( https://ci.apache.org/projects/flink/flink-docs-release-1.3/api/java/org/apache/flink/streaming/connectors/kafka/internals/KafkaTopicPartitionAssigner.html ) That means your partitions should have

"Unable to find registrar for hdfs" on Flink cluster

Hi All, build jar file from the beam quickstart. while run the jar on Flinkcluster got below error.? anybody got this error? Could you please help how to resolve this? root1@master:~/NAI/Tools/flink-1.3.0$ *bin/flink run -c org.apache.beam.examples.WordCount