Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode

@h.yuan...@gmail.com Any thoughts on this? Best, Zhanghao Chen From: hjw <1010445...@qq.com> Sent: Tuesday, September 13, 2022 11:24 To: zhanghao.chen ; user Subject: Re: A classloading question about submitting Flink jobs on

Re: [FEEDBACK] Metadata Platforms / Catalogs / Lineage integration

An interesting topic, I noticed that the datahub community has launched the feature request discussion of Flink Integration [1]. @Martijn Visser Did the Flink community had created tickets to track this topic? >From my current understanding, Flink lacks rich

Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode

Hi, The yarn.classpath.include-user-jar parameter is shown as yarn.per-job-cluster.include-user-jarparameter in Flink 1.14. I have triedDISABLED??FIRST??LAST??ORDER .But the error still happened. Best, Hjw --Original-- From:

Re: Insert into JDBC table

"tEnv.executeSql("INSERT INTO Customers (customer_number, pid_no, name) VALUES (4000, 100, 'customer')");" should work. If not work, it seems to be a bug. >> "Flink dynamic table is just a link to real data" Yes, it's. >> Is there any way to create empty table? Or table with some values defined

Re: Access to Table environent properties/Job arguents from DynamicTableFactory

Have you ever checked the DynamicTableFactory.Context#getConfiguration? Is it something that you're looking for? Best regards, Yuxia 发件人: "Krzysztof Chmielewski" 收件人: "User" 发送时间: 星期六, 2022年 9 月 10日 上午 12:51:09 主题: Access to Table environent properties/Job arguents from

Re: A classloading question about submitting Flink jobs on Yarn-Perjob Mode

Hi, Did you set any additional classloading-related configs (esp. the yarn.classpath.include-user-jar parameter)? Best, Zhanghao Chen From: hjw <1010445...@qq.com> Sent: Tuesday, September 13, 2022 1:58 To: user Subject: A classloading question about submitting

Re: How to perform something on Checkpointing and Savepointing

Hi, I think maybe you could try to create a Function that implements WithMasterCheckpointHook. These hooks will be called by the checkpoint coordinator when triggering / restoring a checkpoint. You could see more details from [1]. [1]

Re: Can flink dynamically detect high load and increase the job parallelism automatically?

Hi Erez, Unfortunately, autoscaling for streaming jobs is only available with reactive mode, which as you've already pointed out, is an MVP feature yet and only supports Standalone mode. Some vendors (e.g. Ververica) have already shipped their own private implementations of Flink autoscaling

Classloading issues with Flink Operator / Kubernetes Native

Hey everyone, I’m migrating a Flink Kubernetes standalone job to the Flink operator (with Kubernetes native mode). I have a lot of classloading issues when trying to run with the operator in native mode. For example, I have a Postgres driver as a dependency (I can confirm the files are included

A classloading question about submitting Flink jobs on Yarn-Perjob Mode

When I submit a job to yarn on Yarn-perjob Mode.An error occurred during the client-side generation of the Jobgraph submitd to yarn. Error: java.lang.NoClassDefFoundError??org/apache/orc/PhysicalWriter Because the cluster is public, there is already related

?????? Where will the state be stored in the taskmanager when using rocksdbstatebend??

thanks for everyone. I will increase the parallelism to solve the problem.Besides,I am looking forward to support remote state. Best, Hjw ---- ??:

Re: How to perform something on Checkpointing and Savepointing

Tried these as well but they need to be called with some MapFunction or something else, we cannot just apply this Listener to the ExecutionEnvironment itself? Thanks and Regards , Surendra Lalwani On Mon, Sep 12, 2022 at 8:42 PM Ben Edwards wrote: > CheckpointedFunction / CheckpointListener.

Re: How to perform something on Checkpointing and Savepointing

CheckpointedFunction / CheckpointListener. Best, Ben On Mon, Sep 12, 2022 at 4:01 PM Surendra Lalwani via user < user@flink.apache.org> wrote: > Hi Team, > > I want to perform some customized operations on checkpointing and > savepointing. For example, I want to create an additional file every

How to perform something on Checkpointing and Savepointing

Hi Team, I want to perform some customized operations on checkpointing and savepointing. For example, I want to create an additional file every time checkpointing and savepointing gets triggered. Do we have anything as such? Any listeners or anything else? Thanks and Regards , Surendra Lalwani

Can flink dynamically detect high load and increase the job parallelism automatically?

Hi, When running a streaming job that uses a kafka source, is it possible (without reactive mode) for flink to dynamically detect high load (high consumers lag, high cpu usage...) and increase the job parallelism automatically? I am running flink streaming job on an application mode cluster

Re: Insert into JDBC table

I see I can only insert into JDBC table with select from another table, something like:   tEnv.executeSql("INSERT INTO Customers SELECT customer_number, pid_no, name FROM another_table"); But what if I want to insert row that I created within Flink? For instance I made some calculation and I

Re: Fail to build Flink 1.15.1

I think you want to use --single-branch instead of --depth 1; the latter only overrides top-level files, leaving the remaining files on whatever version you checked out previously. https://git-scm.com/docs/git-clone#Documentation/git-clone.txt---depthltdepthgt On 10/09/2022 16:08, Jun Qin