Re: Questions about taskmanager.memory.off-heap and taskmanager.memory.preallocate

Hi Ethan, For now, my suggestion is that you can set "preallocate" to false. The description(the link provided by you) of "taskmanager.memory.preallocate" says: "When taskmanager.memory.off-heap is set to true, then it is advised that this configuration is also set to true." Best, Vino Ethan

FlinkSQL??????TIMESTAMPDIFF????????E.g??????

buildin Functions??ctrC?? ctrV ?? TIMESTAMPDIFF(DAY, TIMESTAMP time1,TIMESTAMP time2) SQL validate?? ?? TIMESTAMPDIFF(DAY, cast(time1 as timestamp),cast(time2 as

Scala ListBuffer cannot be used as a POJO type in Flink

Hello everyone, When I run the code below. The log print: > class scala.collection.mutable.ListBuffer does not contain a setter for field > scala$collection$mutable$ListBuffer$$start > Class class scala.collection.mutable.ListBuffer cannot be used as a POJO type > because not all fields are

jobmanager异常日志

各位： 大家好，在standalone的jobmanager节点的日志中看到的，这个153的taskmanager挂掉了，不知道是不是下面的报错导致的，有知道下面的Error是啥意思不 2019-12-15 17:15:21.999 [flink-metrics-379] ERROR akka.remote.Remoting flink-metrics-akka.remote.default-remote-dispatcher-20 - Association to [akka.tcp://flink-metr...@xx.xx.xx.153:35929] with UID [1

Re: Jobmanager not properly fenced when killed by YARN RM

Hi Paul, I have gone through the codes and found that the root cause may be `YarnResourceManager` cleaned up the application staging directory. When it unregisters from the Yarn ResourceManager failed, a new attempt will be launched and failed quickly because of localization failed. I think it

Re: Single Dashboard with Job Cluster on K8s

Hi Akash, If you create a flink session cluster fisrt, and then submit multiple flink job to the existing session cluster, you could have a Flink Dashboard that contains multiple jobs. If you create multiple per-job flink cluster, there is no a dashboard or webui to view all the flink jobs. Each

Re: Fw: Metrics based on data filtered from DataStreamSource

Hi Sidney, Firstly, the `open` method of UDF's instance is always invoked when the task thread starts to run. >From the second code snippet image that you provided, I guess you are trying to get a dynamic handler with reflection technology, is that correct? I also guess that you want to get a

Re: [ANNOUNCE] Weekly Community Update 2019/50

Hi Konstantin, Happy holidays and thanks a lot for your great job on the updates continuously. With the updates, it is easier for us to catch up with what's going on in the community, which I think is quite helpful. I'm wondering if I can do some help and cover this during your vocation. :)

[flink-SQL-API]??????kafka????json??????????????

kafka??sql?? ?? ??"key.database,value.data.user_id,value.data.org_id"?? json { "key": { "database":

Re: TypeInformation problem

Hi Nick, >From StackOverflow, I see a similar issue which answered by @Till Rohrmann . [1] FYI. Best, Vino [1]: https://stackoverflow.com/questions/38214958/flink-error-specifying-keys-via-field-positions-is-only-valid-for-tuple-data-ty Nicholas Walton 于2019年12月14日周六 上午12:01写道： > I was

Re: NPE in blink planner code gen

Hi Jingsong, Leonard, Thanks for your response, I'll created an issue ( https://issues.apache.org/jira/browse/FLINK-15266) to track this. And further discussions can be moved to JIRA. Jingsong Li 于2019年12月16日周一 上午10:17写道： > Hi Benchao, > > Thanks for your reporting. > As Leonard said, you can

Re: NPE in blink planner code gen

Hi Benchao, Thanks for your reporting. As Leonard said, you can create an issue in JIRA. We can go on discussing in JIRA. The answer is #1, Blink's behavior ensures that the job runs as much as possible without interrupting it, so null is returned here to make it possible to continue running. In

Re: Join a datastream with tables stored in Hive

Hi Krzysztof, thanks for the discussion, you raised lots of good questions, I will try to reply them one by one. Re option 1: > Question 1: do I need to write that Hive source or can I use something ready, like Hive catalog integration? Or maybe reuse e.g. HiveTableSource class? I'm not sure if

Re: NPE in blink planner code gen

Hi, Benchao, thank you for your report. It looks egacy planner and blink planner have different behavior. Could you create an issue in https://issues.apache.org/jira/browse/FLINK ? > On Dec 15, 2019, at 16:17, Benchao Li wrote: > > hi all, > >

Re: ArrayIndexOutOfBoundException on checkpoint creation

Hi, Just for completeness sake: I followed the advice of Gyula and now everything seems to work. I changed my HashSet to managed MapState of Flink and changed my LinkedList to a custom class of mine which is recognized by Flink as POJOType and thus don't use Kryo for serialization. This new

Fw: Metrics based on data filtered from DataStreamSource

Hey, I have a question about using metrics based on filtered data. Basically, I have handlers for many types of events I get from my data source (in my case, Kafka), and every handler has it's own filter function. That given handler also has a Counter, incrementing every time it filters out an

[ANNOUNCE] Weekly Community Update 2019/50

Dear community, happy to share this week's brief community digest with updates on Flink 1.8.3 and Flink 1.10, a discussion on how to facilitate easier Flink/Hive setups, a couple of blog posts and a bit more. *Personal Note:* Thank you for reading these updates since I started them early this

Single Dashboard with Job Cluster on K8s

Hi, If I create multiple job clusters on Kubernetes then is it possible to view all the jobs in a single Flink Dashboard or do I need to maintain multiple dashboards. Thanks, Akash Goel

Flink运行一段时间后出现异常重启

请教一下，flink采用1.8.0版本，任务运行一段时间后出现AskTimeoutException异常，之后flink状态由Running转Failed，紧接着出现FlinkKafkaException:Failed to send data to kafka失败，任务定期出现这种异常重启。请教一下，这种通常有什么问题导致的呀，应该如何排除处理？ 2019-12-15 19:25:14,954 WARN org.apache.flink.runtime.rest.handler.legacy.metrics.MetricFetcherImpl - Requesting

flink-cluster restart with no job,the job manager's log always have exceptions--job not found

hallo:i have some problem when using the flink-1.8.1 .i wish your help Exception discript:2019-12-15 14:06:08,554 ERROR org.apache.flink.runtime.rest.handler.job.JobDetailsHandler - Exception occurred in REST handler: Job bf5b710cad760430305bf2b637100028 not found 2019-12-15 14:06:11,446

NPE in blink planner code gen

hi all, We are using 1.9.0 blink planner, and find flink will throw NPE when we use the following SQL: ``` create table source { age int, id varchar }; select *case when age < 20 then cast(id as bigint) else 0 end* from source; ``` After debugging the Janino generated code, I find that