Re: After job cancel, leftover ZK state prevents job manager startup

We have a single Jobmanager in an HA setup. From looking at logs and metrics, it appears that before the issue occurred there was a long (15s) GC pause on the jobmanager, which then caused a leadership election. Because there is only one jobmanager, the same one became leader again after it

RocksDB "shared" directory on HDFS getting bigger

Hi, I'm using RocksDB state backend w/ incremental checkpoint. hdfs:shared directory is getting bigger as time goes on. Does it mean that I forget to clean up state somewhere? - Dongwon

Re: Is there an example of flink cluster "as a job" deployment on k8s ?

thanks On Thu, Dec 6, 2018 at 5:39 AM Dawid Wysakowicz wrote: > Hi Vishal, > > You might want to have a look at the flink-container/kubernetes module: > https://github.com/apache/flink/tree/master/flink-container/kubernetes > > Best, > > Dawid > > On 05/12/2018 22:50, Vishal Santoshi wrote: > >

Re: Flink with Openstack Swift example

Hi Andrey, Thanks for the help. Let me try it out. Regards, Tejash On Wed, Dec 12, 2018 at 10:32 PM Andrey Zagrebin wrote: > Hi Tejash, > > Flink has an implementation for swift file system (see > SwiftFileSystemFactory). > It should be enough just prepend your checkpoint/savepoint paths with

Re: Cannot configure akka.ask.timeout

Qi, Job submission timeout is caused by listing too many files in S3 during env.readFile call to create input DataSet. Is there a way NOT to list S3 files during a job submission? It seems like it should help to mitigate that timeout problem. What hardcoded value you were referring to? Best,

Re: Flink with Openstack Swift example

Hi Tejash, Flink has an implementation for swift file system (see SwiftFileSystemFactory). It should be enough just prepend your checkpoint/savepoint paths with “swift://“ (e.g. instead of "hdfs://“). See configuration 'how to’ in Flink docs [1], [2]. Best, Andrey [1]

Re: Read all json files from a hdfs partition folder

Actually, does it not work if you just provide directory in env.readTextFile as in your code example or what is the problem? > On 12 Dec 2018, at 17:24, Andrey Zagrebin wrote: > > Hi, > > If the question is how to read all files from hdfs directory, > in general, each file is potentially a

Re: Connection leak with flink elastic Sink

Hi Bhaskar, I think Gordon might help you, I am pulling him into the discussion. Best, Andrey > On 12 Dec 2018, at 13:31, Vijay Bhaskar wrote: > > Hi > We are using flink elastic sink which streams at the rate of 1000 events/sec, > as described in >

Re: Read all json files from a hdfs partition folder

Hi, If the question is how to read all files from hdfs directory, in general, each file is potentially a different DataSet (not DataStream). It needs to be decided how to combine/join them in Flink pipeline. If the files are small enough, you could list them as string paths and use

Re: Apache Flink CEP how to detect if event did not occur within x seconds?

Hi Florin, I think Dawid might help you. I am pulling him into the discussion. Best, Andrey > On 12 Dec 2018, at 16:24, Spico Florin wrote: > > Hello! > I'm using the same questions as in this stackoverflow post >

Re: Question about key group / key state & parallelism

Hi Bastien, You are right, it will wait for message A to be processed. To be more generic, it is a question of how to solve the data skew problem in shuffle case. This question is common and there are already many ways to solve it according to the different scenario. I think we can solve your

Apache Flink CEP how to detect if event did not occur within x seconds?

Hello! I'm using the same questions as in this stackoverflow post https://stackoverflow.com/questions/50356981/apache-flink-cep-how-to-detect-if-event-did-not-occur-within-x-seconds?rq=1, due to fact that I need the same functionality. My use case is to detect when an device is disconnected from

Re: Question about key group / key state & parallelism

If you don’t care about having the same key processed by the same operator (so no keyed state, no windowed processing of the same keys, etc) then you can just use DataStream.rebalance() . — Ken

Re: Question about key group / key state & parallelism

Hi Hequn, thanks for your response ! Ok, that's what I was thinking about the key & operator instance If the affectation of key group to an instance is deterministic (and the hash of the key to belong to a key group) I have the following problem Let's say I have 4 key (A,B,C,D) & 2 parallel

Read all json files from a hdfs partition folder

Hi, I wanted to read all json files from hdfs with partition folder. public static void main(String[] args) { StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); //path //hdfs://localhost:8020/data/ingestion/ingestion.raw.product/2018/12/05/23 DataStream

Flink with Openstack Swift example

Hi Users, Can somebody help me by pointing to any document listing out the steps if I want to configure checkpoint and savepoint in OpenStack Swift ? Thanks in advance, Tejash

Re: Cannot configure akka.ask.timeout

Hi Qi, Thanks for looking into this. Here is ticket https://issues.apache.org/jira/browse/FLINK-11143 Best, -Alex On Tue, Dec 11, 2018 at 8:47 PM qi luo wrote: > Hi Alex and Lukas, > > This error is controlled by another RPC timeout (which is hard coded and > not affected by

Re: Flink 1.7 jobmanager tries to lookup taskmanager by its hostname in k8s environment

This is a known issue, see https://issues.apache.org/jira/browse/FLINK-11127. I'm not aware of a workaround. On 12.12.2018 14:07, Sergei Poganshev wrote: When I to deploy Flink 1.7 job to Kubernetes, the job itself runs, but upon visiting Flink UI I can see no metrics and there are WARN

Flink 1.7 jobmanager tries to lookup taskmanager by its hostname in k8s environment

When I to deploy Flink 1.7 job to Kubernetes, the job itself runs, but upon visiting Flink UI I can see no metrics and there are WARN messages in jobmanager's log: [flink-metrics-14] WARN akka.remote.ReliableDeliverySupervisor flink-metrics-akka.remote.default-remote-dispatcher-3 - Association

Re: Question about key group / key state & parallelism

Hi Bastien, Each key “belongs” to exactly one parallel instance of a keyed operator, and each parallel instance contains one or more Key Groups. Keys will be hashed into the corresponding key group deterministically. It is hashed by the value instead of the number of the total records. Different

Connection leak with flink elastic Sink

Hi We are using flink elastic sink which streams at the rate of 1000 events/sec, as described in https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/elasticsearch.html . We are observing connection leak of elastic connections. After few minutes all the open connections are

Question about key group / key state & parallelism

Hello everyone, I have a question regarding the key state & parallelism of a process operation Doc says : "You can think of Keyed State as Operator State that has been partitioned, or sharded, with exactly one state-partition per key. Each keyed-state is logically bound to a unique composite of