[DISCUSS] FLIP-6 Problems

Hi: We've deployed flink 1.5.0 and tested the new cluster manager, it's really great for flink to be elastic. However we've also found some problems that blocks us from deploying it to production environment. 1. Task manager isolation. Currently flink allows different jobs to execute on same

Re: some default config questions

thanks -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: why does flink release package preferred uber jar than small jar?

thanks rongrong, but it seems unrelevant. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: JVM metrics disappearing after job crash, restart

Fabian, It does look like it may be related. I'll add a comment. After digging a bit more I found that the crash and lack of metrics were precipitated by the JobManager instance crashing and cycling, which caused the job to restart. Chesnay, I didn't see anything interesting in our logs. Our

Re: NPE in flink sql over-window

Hi Fabian, I added some trace logs in ProcTimeBoundedRangeOver and think it should be a bug. The log should explain how cleanup_time_1 bypasses the needToCleanupState check and causes NPE. A jira ticket [1] is created. Best Yan [ts:1528149296456] [label:state_ttl_update] register for cleanup

Re: NPE in flink sql over-window

Hi Fabian, Yes, the NPE cause the job failure and recovery( instead of being the result of a recovery). And yet, during the recovery, the same exceptions are thrown from same line. Best Yan From: Fabian Hueske Sent: Thursday, May 31, 2018 12:09:03 AM To:

Re: [DISCUSS] Flink 1.6 features

Hi Stephan, Will FLINK-7129 (Support dynamically changing CEP patterns) be included in 1.6? There were discussions about possibly including it in 1.6: http://mail-archives.apache.org/mod_mbox/flink-user/201803.mbox/%3cCAMq=ou7gru2o9jtowxn1lc1f7nkcxayn6a3e58kxctb4b50...@mail.gmail.com%3e Thanks,

Re: Ask for SQL using kafka in Flink

Yes, I am also looking for examples for Kafka avro table examples in java and command line. Also, Kafka avro table sink is still missing. In addition, once we have Kafka topic, the API should read the schema directly from schema file or schema registry. The way of current API supporting lacks

Cannot determine simple type name - [FLINK-7490]

I am getting an error when using the Table API with ScalarFunction. This is previously reported here (https://issues.apache.org/jira/browse/FLINK-7490) as loading a wrong class loader and is reported as fixed. But the issue persists even after using the latest code from 1.3 branch. >From some of

Re: Trigerring Savepoint for the Flink Job

Glad to know you found a solution. would you mind sharing you workaround in Flink 1.4.2. I am pretty sure other users would benefit from your effort too :-) Regarding the checkpoints / savepoints backend, we use HDFS as our stateful backend instead of RocksDB. We are working on putting that logic

Re: why does flink release package preferred uber jar than small jar?

Hi makeyang, I am not sure if related, but it might be helpful for you to checkout some previous email threads [1][2]. -- Rong [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUS-Flink-SQL-Client-dependency-management-td21270.html [2]

Re: Ask for SQL using kafka in Flink

Yes I totally agree Shuyi Chen, if anyone has an example which does the pipeline streaming processing (Kafka-Flink-SQL), it will be useful for all of us. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Ask for SQL using kafka in Flink

Thanks Timo. I will go over there. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Ask for SQL using kafka in Flink

Given the popularity of Flink SQL and Kafka as streaming source, I think we can add some examples of using Kafka[XXX]TableSource in flink-examples/flink-examples-table module. What do you guys think? Cheers Shuyi On Mon, Jun 4, 2018 at 12:57 AM, Timo Walther wrote: > Hi, > > as you can see in

Re: Trigerring Savepoint for the Flink Job

Just out of curiosity how do you save your checkpoint. Currently I'm using filesystem but I'm migrating it to RocksDB which allows for async operations to avoid latency at higher scale as we grow. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Trigerring Savepoint for the Flink Job

Hi Rongs. Thanks for your help. I was about to look into the CLI API but then I figured out how to trigger savepoint and restore a job's savepoint with the job running in detached mode. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: TaskManager use more memory than Xmx

Hi, Szczypiński Do you have noticed the size of the maxDirectMemorySize in your taskamanger jvm options， AFAIK the total memory one Taskmanager used is the (heap memory + directMemory) which may larger than the heap size specified by -Xmx? Best wishes Aitozi -- Sent from:

Re: File does not exist prevent from Job manager to start .

Hi Miki, it looks as if you did not submit a job to the cluster of which you shared the logs. At least I could not see a submit job call. Cheers, Till On Mon, Jun 4, 2018 at 12:31 PM miki haiat wrote: > HI Till, > Iv`e managed to do reproduce it. > Full log faild_jm.log >

Re: ODP: TaskManager use more memory than Xmx

Hi, I had such issue once when opening a lot of Parquet writers and some non-Heap memory was never released. So you might check that you close all resources correctly. -- Best regards, Andrei Shumanski 2018-06-04 14:24 GMT+02:00 Flavio Pompermaier : > We had similar problems that was caused

Re: ODP: TaskManager use more memory than Xmx

We had similar problems that was caused by Netty (but we were using batch APIs). Look at this discussion, maybe it can help: http://mail-archives.apache.org/mod_mbox/flink-user/201706.mbox/%3ccaeluf_aau+u9xyfpjesdrnxb84glod_8jp-xexrf-6ey4qe...@mail.gmail.com%3E Best, Flavio 2018-06-04 13:29

Re: [DISCUSS] Flink 1.6 features

Before removing the legacy code, I would still wait a bit and see what the user feedback is. The legacy mode is a good safety net against severe deployment regressions. Thus, it should be a very conscious decision to remove the code. As far as I know, there is currently nobody actively working on

Re: ODP: TaskManager use more memory than Xmx

Hi If you use Rocksdbstate backend rocksdb use memory outside the process to my understanding. We have the same problem and I guess it started when we introduced job with larger state and moved all jobs over to rocksdb Med venlig hilsen / Best regards Lasse Nedergaard > Den 4. jun. 2018 kl.

ODP: TaskManager use more memory than Xmx

Hi, I use 1.3.1 version. Od: miki haiat [mailto:miko5...@gmail.com] Wysłano: poniedziałek, 4 czerwca 2018 13:20 Do: Szczypiński, S. (Szymon) DW: user Temat: Re: TaskManager use more memory than Xmx Which flink version ? I had the same issue on 1.4.2 . On Mon, Jun 4, 2018 at 2:14 PM

Re: TaskManager use more memory than Xmx

Which flink version ? I had the same issue on 1.4.2 . On Mon, Jun 4, 2018 at 2:14 PM Szczypiński, S. (Szymon) < szymon.szczypin...@ingbank.pl> wrote: > Hi, > > I have a problem with TaskManagers in my standalone cluster. > > > > My problem is that my host have 32GB RAM memory. TaskManager

TaskManager use more memory than Xmx

Hi, I have a problem with TaskManagers in my standalone cluster. My problem is that my host have 32GB RAM memory. TaskManager have Xms and Xmx set to 20GB but when TaskManager is working it use more memory than host have and start to consume SWAP. In the end system kills java process. Mayby

why does flink release package preferred uber jar than small jar?

what's the benefit that uber jar have while small jar doesn't? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: File does not exist prevent from Job manager to start .

HI Till, Iv`e managed to do reproduce it. Full log faild_jm.log On Mon, Jun 4, 2018 at 10:33 AM Till Rohrmann wrote: > Hmmm, Flink should not delete the

Re: [DISCUSS] Flink 1.6 features

Hi Stephen, Is it planned to consider this ticket https://issues.apache.org/jira/browse/FLINK-7883 about an atomic cancel-with-savepoint ? It is my main concern about Flink and I have to maintain a fork myself as we can't afford dupplicate events due to reprocess of messages between a savepoint

Re: is there a config to ask taskmanager to keep retrying connect to jobmanager after Disassociated?

so is there a way or config to ask taskmanager to keep continue connectting to jobmanager? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: is there a config to ask taskmanager to keep retrying connect to jobmanager after Disassociated?

when I debug the jobmanager and below is the error log in task manager: 2018-06-04 17:16:33,295 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor- The heartbeat of ResourceManager with id 35df0455efc2fb6fa3f2467f7f5d2ba1 timed out. 2018-06-04 17:16:33,297 DEBUG

Re:[DISCUSS] Flink 1.6 features

Hi Stephan, could you please also consider the "Elastic Filter " feature discussioned in thread http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/PROPOSAL-Introduce-Elastic-Bloom-Filter-For-Flink-td22430.html ? Best, Sihua On 06/4/2018 17:21，Stephan Ewen wrote： Hi Flink

Re: [DISCUSS] Flink 1.6 features

Will we remove the legacy mode for 1.6? I can see value in keeping it for now so that legacy issues are still visible on master, but at the same time removing this code would reduce a lot of complexity and ambiguity in the codebase... On 04.06.2018 11:21, Stephan Ewen wrote: Hi Flink

[DISCUSS] Flink 1.6 features

Hi Flink Community! The release of Apache Flink 1.5 has happened (yay!) - so it is a good time to start talking about what to do for release 1.6. *== Suggested release timeline ==* I would propose to release around *end of July* (that is 8-9 weeks from now). The rational behind that: There was

Re: JVM metrics disappearing after job crash, restart

Can you show us the metrics-related configuration parameters in flink-conf.yaml? Please also check the logs for any warnings from the MetricGroup and MetricRegistry classes. On 04.06.2018 10:44, Fabian Hueske wrote: Hi Nik, Can you have a look at this JIRA ticket [1] and check if it is

Re: JVM metrics disappearing after job crash, restart

Hi Nik, Can you have a look at this JIRA ticket [1] and check if it is related to the problems your are facing? If so, would you mind leaving a comment there? Thank you, Fabian [1] https://issues.apache.org/jira/browse/FLINK-8946 2018-05-31 4:41 GMT+02:00 Nikolas Davis : > We keep track of

How to set UserGroupInformation?

Thanks both of you. Our hdfs version is 1.x, so I have to set hadoop.job.ugi. However, I can't find anywhere I can set the config. Do you have any ideas? In fact, except 2018年6月2日星期六，Rong Rong 写道： > Hi Xinyu, > > To add to Till's comment, setting `HADOOP_USER_NAME` in your environment > is

[ANNOUNCE] Weekly community update #23

Dear community, this is the weekly community update thread #23. Please post any news and updates you want to share with the community to this thread. # Releasing flink-shaded 4.0 The community is currently testing the RC 2 for flink-shaded 4.0 [1] which bumps the shaded Netty version to 4.1.24

Re: Ask for SQL using kafka in Flink

Hi, as you can see in code [1] Kafka09JsonTableSource takes a TableSchema. You can create table schema from type information see [2]. Regards, Timo [1]

Re: Java Code for Kafka Flink SQL

Hi Rad, at a first glance your example does not look too bad. Which exceptions do you get? Did you create your pom.xml with the provided template [1] and then added flink-table, flink-connector-kafkaXXX, flink-streaming-scala? Regards, Timo [1]

Re: File does not exist prevent from Job manager to start .

Hmmm, Flink should not delete the stored blobs on the HA storage. Could you try to reproduce the problem and then send us the logs on DEBUG level? Please also check before shutting the cluster down, that the files were there. Cheers, Till On Sun, Jun 3, 2018 at 1:10 PM miki haiat wrote: > Hi

Re: Flink 1.2.1 - Job initial start time

Thank you! On Fri, Jun 1, 2018 at 5:21 PM Bollerman, Petrus wrote: > In addition to not resetting the CREATED timestamp, there is value to keep > a history of restarts, and exceptions. We'll add that to the ticket as well. > > > > *From:* Till Rohrmann > *Sent:* Friday, June 01, 2018 10:10 AM

is there a config to ask taskmanager to keep retrying connect to jobmanager after Disassociated?

I depoloy a standard alone cluster with flink1.5 and when I try to restart the only jobmanger, below is the log print from task manager: 2018-06-04 12:06:35,882 WARN akka.remote.ReliableDeliverySupervisor - Association with remote system [akka.tcp://flink@ipaddress:6123]