Re: Flink job unable to restore from savepoint

Hi, Prashant. Which Flink version did you use? And Did you modify your job logic or configurations ? If yes, Could you share changed things ? On Wed, Mar 27, 2024 at 3:37 PM prashant parbhane wrote: > Hello, > > We have been facing this weird issue of not being able to restore from > savepoint,

RE: need flink support framework for dependency injection

Hi Ganesh, I tend to agree with Marco. However your 'feature request' is very loose and leave much room for misunderstanding. There are at least two scenarios for DI integration: - DI for job setup: - we use spring for job setup, which - lets us use the same job structure for (at

Flink job unable to restore from savepoint

Hello, We have been facing this weird issue of not being able to restore from savepoint, when we have a significant load on flink jobs. " *org.apache.flink.util.FlinkRuntimeException: Error while deserializing the user key.* *at

[discuss] [jdbc] Support Ignore deleting is required？

When using the jdbc sink connector, there are a lot of scenarios where we ignore delete (-D) record when writing out data. Our jdbc sink connector scenario needs to add a configuration sink.ignore-delete attribute. The default value is false, and data deletion is not ignored. The user can set

Discussion thread : Proposal to add Conditions in Flink CRD's Status field

 Hello, Starting discussion thread here to discuss a proposal to add Conditions field in the CR status of Flink Deployment and FlinkSessionJob. Here is the google doc with details. Please provide your thoughts/inputs.

Re: need flink support framework for dependency injection

Hi Ganesh, I disagree. I don’t think Flink needs a dependency injection framework. I have implemented many complex jobs without one. Can you please articulate why you think it needs a dependency injection framework, along with some use cases that will show its benefit? I would rather see more

need flink support framework for dependency injection

Re: Temporal join on rolling aggregate

Hey Sebastien et al, have you tried rewriting the rolling aggregate as a window-over query? A window-over aggregation creates an append-only stream which should preserve the timestamp/watermark of the source. You can then add a deduplication

Re: Understanding RocksDBStateBackend in Flink on Yarn on AWS EMR

Usually, you should use the HDFS nameservice instead of the NameNode hostname:port to avoid NN failover. And you could find the supported nameservice in the hdfs-site.xml in the key *dfs.nameservices*. Best, Yang On Fri, Mar 22, 2024 at 8:33 PM Sachin Mittal wrote: > So, when we create an EMR

Re: There is no savepoint operation with triggerId

Hi Lars, It looks like the relevant logs when retrieving savepoint. Have you frequently retrieved savepoints through the REST interface? Lars Skjærven 于2024年3月26日周二 07:17写道： > > Hello, > My job manager is constantly complaining with the following error: > > "Exception occurred in REST handler:

There is no savepoint operation with triggerId

Hello, My job manager is constantly complaining with the following error: "Exception occurred in REST handler: There is no savepoint operation with triggerId= for job bc0cb60b710e64e23195aa1610ad790a".

Re: Unable to Generate Flame Graphs in Flink UI

Hi Nitin You don't need any additional configuration. After the sampling is completed, the flame graph will be displayed. Best, Feng On Mon, Mar 25, 2024 at 5:55 PM Nitin Saini wrote: > Hi Flink Community, > > I've set up a Flink cluster using Flink version 1.17.2 and enabled the > option

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.8.0 released

Congratulations! Thanks Max for the release and all involved for the great work! A gentle reminder to users: the maven artifact has just been released and will take some time to complete. Best, Rui On Mon, Mar 25, 2024 at 6:35 PM Maximilian Michels wrote: > The Apache Flink community is very

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.8.0 released

Congratulations! Thanks Max for the release and all involved for the great work! A gentle reminder to users: the maven artifact has just been released and will take some time to complete. Best, Rui On Mon, Mar 25, 2024 at 6:35 PM Maximilian Michels wrote: > The Apache Flink community is very

[ANNOUNCE] Apache Flink Kubernetes Operator 1.8.0 released

The Apache Flink community is very happy to announce the release of the Apache Flink Kubernetes Operator version 1.8.0. The Flink Kubernetes Operator allows users to manage their Apache Flink applications on Kubernetes through all aspects of their lifecycle. Release highlights: - Flink

[ANNOUNCE] Apache Flink Kubernetes Operator 1.8.0 released

The Apache Flink community is very happy to announce the release of the Apache Flink Kubernetes Operator version 1.8.0. The Flink Kubernetes Operator allows users to manage their Apache Flink applications on Kubernetes through all aspects of their lifecycle. Release highlights: - Flink

Custom metricReporterFactory could not be found in k8s deployment.

hi all, bg: create a custom metric reporter via kafka, it works in ideaj local environment. but failed when packaged and deployed in k8s env (ververica by alibaba) flink 1.12 config: metrics.reporter.kafka.factory.class: org.apache.flink.metrics.kafka.KafkaReporterFactory

Fwd: Global connection open and close

[previous didn't cc list, sorry for dupes] The classic connection pool pattern, where expensive connections are created relatively few times and used by lots of transient short-lived tasks, each of which borrows a connection from the pool and returns it when done, would still be usable here, but

Re: Global connection open and close

The classic connection pool pattern, where expensive connections are created relatively few times and used by lots of transient short-lived tasks, each of which borrows a connection from the pool and returns it when done, would still be usable here, but as Péter points out, you can't rely on a

Re: Understanding RocksDBStateBackend in Flink on Yarn on AWS EMR

So, when we create an EMR cluster the NN service runs on the primary node of the cluster. Now at the time of creating the cluster, how can we specify the name of this NN in format hdfs://*namenode-host*:8020/. Is there a standard name by which we can identify the NN server ? Thanks Sachin On

Re: Flink unable to read from kafka source due to starting offset strategy

Hi, After some debugging I see these in the logs: 2024-03-22 14:25:47,555 INFO org.apache.kafka.clients.NetworkClient [] - [Consumer clientId=qubit-data-consumer-0, groupId=spflink] Disconnecting from node 11 due to request timeout. 2024-03-22 14:25:47,647 INFO

Re:Re: Re: Read data from elasticsearch using Java flink

Hi, Nida. About using connector with Java by Flink DataStream api, you can mainly reference these doc[1][2]. However, Elaticsearch connector only supports sink currently. What you need it to build a custom ES connector with that patch[3] yourself. The following steps may help you. 1.

Re: Flink unable to read from kafka source due to starting offset strategy

Hi, I was experimenting with different starting offset strategies for my Flink job, especially in cases where jobs are canceled and scheduled again and I would like to start with the last committed offset and if the same is not available then start from the latest. So I decided to use this:

Re: Understanding RocksDBStateBackend in Flink on Yarn on AWS EMR

Hello Sachin, Typically, Cloud VMs are ephemeral, meaning that if the EMR cluster goes down or VMs are required to be shut down for security updates or due to faults, new VMs will be added to the cluster. As a result, any data stored in the local file system, such as file://tmp, would be lost. To

Understanding RocksDBStateBackend in Flink on Yarn on AWS EMR

Hi, We are using AWS EMR where we can submit our flink jobs to a long running flink cluster on Yarn. We wanted to configure RocksDBStateBackend as our state backend to store our checkpoints. So we have configured following properties in our flink-conf.yaml - state.backend.type: rocksdb -

Re: Re: Read data from elasticsearch using Java flink

Hi Xuyang I am new to Flink & I don't know how to implement this dependency into the code. Can you please share some examples so that I can refer those. Thanks Nida On Fri, Mar 22, 2024 at 7:02 AM Xuyang wrote: > Hi, Nida. > Can you explain more details about "unable to use it." ? Did you

Re: 退订

Hi, scott 请发送任意内容的邮件到 user-zh-unsubscr...@flink.apache.org 地址来取消订阅来自 user-zh@flink.apache.org 邮件组的邮件，你可以参考[1][2] 管理你的邮件订阅。 Best, Zhongqiang Gong [1] https://flink.apache.org/zh/community/#%e9%82%ae%e4%bb%b6%e5%88%97%e8%a1%a8 [2] https://flink.apache.org/community.html#mailing-lists 己巳 于

Re:Re: Read data from elasticsearch using Java flink

Hi, Nida. Can you explain more details about "unable to use it." ? Did you get an exception after using it? -- Best！ Xuyang 在 2024-03-21 21:14:53，"Fidea Lidea" 写道： Thank you Xuyang. I added the above flink-connector-elasticsearch dependency in my project. But I am unable to use

Re: Global connection open and close

Hi Jacob, Flink jobs, tasks typically run on multiple nodes/servers. This means that it is not possible to have a connection shared on job level. You can read about the architecture in more detail in the docs. [1] I hope this helps, Péter [1] -

Re: Read data from elasticsearch using Java flink

Thank you Xuyang. I added the above flink-connector-elasticsearch dependency in my project. But I am unable to use it. Can you please share a few sample codes which are using this dependency? Thanks Nida On Tue, Mar 12, 2024 at 5:37 PM Xuyang wrote: > Hi, Fidea. > Currently, elasticsearch is

Global connection open and close

Hello, Is there a way in Flink to instantiate or open connections (to cache/db) at global level, so that it can be reused across many process functions rather than doing it in each operator's open()?Along with opening, also wanted to know if there is a way to close them at job level stop, such

Re: can we use Scan Newly Added Tables without restarting the existing job ?

Ok, I see. Hang Ruan 于2024年3月20日周三 16:08写道： > Hi, 3pang zhu. > > This `Scan Newly added tables` feature requires restarting the job from > the savepoint. We cannot add new tables to the running job without > restarting by now. > > Best, > Hang > > 3pang zhu 于2024年3月20日周三 15:22写道： > >> this

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congrattulations! Thanks for the great work! Best, Zhongqiang Gong Leonard Xu 于2024年3月20日周三 21:36写道： > Hi devs and users, > > We are thrilled to announce that the donation of Flink CDC as a > sub-project of Apache Flink has completed. We invite you to explore the new > resources available: >

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congrattulations! Thanks for the great work! Best, Zhongqiang Gong Leonard Xu 于2024年3月20日周三 21:36写道： > Hi devs and users, > > We are thrilled to announce that the donation of Flink CDC as a > sub-project of Apache Flink has completed. We invite you to explore the new > resources available: >

Re: Is there any options to control the file names in file sink

Hi Lasse, If the datastream job is used, you can try setting `OutputFileConfig` for file sink, something like[1]: ``` OutputFileConfig config = OutputFileConfig .builder() .withPartPrefix("prefix") .withPartSuffix(".ext") .build(); FileSink> sink = FileSink .forRowFormat((new

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations! Best, Zakelly On Thu, Mar 21, 2024 at 12:05 PM weijie guo wrote: > Congratulations! Well done. > > > Best regards, > > Weijie > > > Feng Jin 于2024年3月21日周四 11:40写道： > >> Congratulations! >> >> >> Best, >> Feng >> >> >> On Thu, Mar 21, 2024 at 11:37 AM Ron liu wrote: >> >> >

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations! Best, Zakelly On Thu, Mar 21, 2024 at 12:05 PM weijie guo wrote: > Congratulations! Well done. > > > Best regards, > > Weijie > > > Feng Jin 于2024年3月21日周四 11:40写道： > >> Congratulations! >> >> >> Best, >> Feng >> >> >> On Thu, Mar 21, 2024 at 11:37 AM Ron liu wrote: >> >> >

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations! Well done. Best regards, Weijie Feng Jin 于2024年3月21日周四 11:40写道： > Congratulations! > > > Best, > Feng > > > On Thu, Mar 21, 2024 at 11:37 AM Ron liu wrote: > > > Congratulations! > > > > Best, > > Ron > > > > Jark Wu 于2024年3月21日周四 10:46写道： > > > > > Congratulations and

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations! Well done. Best regards, Weijie Feng Jin 于2024年3月21日周四 11:40写道： > Congratulations! > > > Best, > Feng > > > On Thu, Mar 21, 2024 at 11:37 AM Ron liu wrote: > > > Congratulations! > > > > Best, > > Ron > > > > Jark Wu 于2024年3月21日周四 10:46写道： > > > > > Congratulations and

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations! Best, Feng On Thu, Mar 21, 2024 at 11:37 AM Ron liu wrote: > Congratulations! > > Best, > Ron > > Jark Wu 于2024年3月21日周四 10:46写道： > > > Congratulations and welcome! > > > > Best, > > Jark > > > > On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote: > > > > >

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations! Best, Feng On Thu, Mar 21, 2024 at 11:37 AM Ron liu wrote: > Congratulations! > > Best, > Ron > > Jark Wu 于2024年3月21日周四 10:46写道： > > > Congratulations and welcome! > > > > Best, > > Jark > > > > On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote: > > > > >

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations! Best, Ron Jark Wu 于2024年3月21日周四 10:46写道： > Congratulations and welcome! > > Best, > Jark > > On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote: > > > Congratulations! > > > > Best, > > Rui > > > > On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan > wrote: > > > > >

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations! Best, Ron Jark Wu 于2024年3月21日周四 10:46写道： > Congratulations and welcome! > > Best, > Jark > > On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote: > > > Congratulations! > > > > Best, > > Rui > > > > On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan > wrote: > > > > >

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations! Best! Xushuai > 2024年3月21日 10:54，Yanquan Lv 写道： > > Congratulations and Looking forward to future versions！ > > Jark Wu 于2024年3月21日周四 10:47写道： > >> Congratulations and welcome! >> >> Best, >> Jark >> >> On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote:

Indefinite hanging when querying batch transactional data from kafka

Hello community, I reported a potential bug with the flink kafka connector on Jira a few weeks ago. But I haven't received feedback yet. Could anyone kindly review this issue? Any feedback would be much appreciated. Jira issue: https://issues.apache.org/jira/projects/FLINK/issues/FLINK-34470

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations and Looking forward to future versions！ Jark Wu 于2024年3月21日周四 10:47写道： > Congratulations and welcome! > > Best, > Jark > > On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote: > > > Congratulations! > > > > Best, > > Rui > > > > On Thu, Mar 21, 2024 at 10:25 AM

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations and welcome! Best, Jark On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote: > Congratulations! > > Best, > Rui > > On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan wrote: > > > Congrattulations! > > > > Best, > > Hang > > > > Lincoln Lee 于2024年3月21日周四 09:54写道： > > >

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations and welcome! Best, Jark On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote: > Congratulations! > > Best, > Rui > > On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan wrote: > > > Congrattulations! > > > > Best, > > Hang > > > > Lincoln Lee 于2024年3月21日周四 09:54写道： > > >

Re:Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Cheers! -- Best！ Xuyang 在 2024-03-21 10:28:45，"Rui Fan" <1996fan...@gmail.com> 写道： >Congratulations! > >Best, >Rui > >On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan wrote: > >> Congrattulations! >> >> Best, >> Hang >> >> Lincoln Lee 于2024年3月21日周四 09:54写道： >> >>> >>> Congrats, thanks

Re:Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Cheers! -- Best！ Xuyang 在 2024-03-21 10:28:45，"Rui Fan" <1996fan...@gmail.com> 写道： >Congratulations! > >Best, >Rui > >On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan wrote: > >> Congrattulations! >> >> Best, >> Hang >> >> Lincoln Lee 于2024年3月21日周四 09:54写道： >> >>> >>> Congrats, thanks

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations! Best, Rui On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan wrote: > Congrattulations! > > Best, > Hang > > Lincoln Lee 于2024年3月21日周四 09:54写道： > >> >> Congrats, thanks for the great work! >> >> >> Best, >> Lincoln Lee >> >> >> Peter Huang 于2024年3月20日周三 22:48写道： >> >>>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations! Best, Rui On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan wrote: > Congrattulations! > > Best, > Hang > > Lincoln Lee 于2024年3月21日周四 09:54写道： > >> >> Congrats, thanks for the great work! >> >> >> Best, >> Lincoln Lee >> >> >> Peter Huang 于2024年3月20日周三 22:48写道： >> >>>

Re:Is there any options to control the file names in file sink

Hi, Nedergaard. IIUC, currently, filesystem sink is not supported to custom the name of the output files, and only custom naming of directories should be supported. [1]

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congrattulations! Best, Hang Lincoln Lee 于2024年3月21日周四 09:54写道： > > Congrats, thanks for the great work! > > > Best, > Lincoln Lee > > > Peter Huang 于2024年3月20日周三 22:48写道： > >> Congratulations >> >> >> Best Regards >> Peter Huang >> >> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang wrote: >>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congrattulations! Best, Hang Lincoln Lee 于2024年3月21日周四 09:54写道： > > Congrats, thanks for the great work! > > > Best, > Lincoln Lee > > > Peter Huang 于2024年3月20日周三 22:48写道： > >> Congratulations >> >> >> Best Regards >> Peter Huang >> >> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang wrote: >>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congrats, thanks for the great work! Best, Lincoln Lee Peter Huang 于2024年3月20日周三 22:48写道： > Congratulations > > > Best Regards > Peter Huang > > On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang wrote: > >> >> Congratulations >> >> >> >> Best, >> Huajie Wang >> >> >> >> Leonard Xu 于2024年3月20日周三

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congrats, thanks for the great work! Best, Lincoln Lee Peter Huang 于2024年3月20日周三 22:48写道： > Congratulations > > > Best Regards > Peter Huang > > On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang wrote: > >> >> Congratulations >> >> >> >> Best, >> Huajie Wang >> >> >> >> Leonard Xu 于2024年3月20日周三

Re: End-to-end lag spikes when closing a large number of panes

Hey Asimansu, The inputDataStream is a KeyedStream, I forgot to mention that. Caio On Wed, Mar 20, 2024 at 6:56 PM Asimansu Bera wrote: > Hello Caio, > > Based on the pseudocode, there is no keyed function present. Hence, the > window will not be processed parallely . Please check again and

Re: End-to-end lag spikes when closing a large number of panes

Hello Caio, Based on the pseudocode, there is no keyed function present. Hence, the window will not be processed parallely . Please check again and respond back. val windowDataStream = inputDataStream .window(TumblingEventTimeWindows of 1 hour) .trigger(custom trigger)

Re: Need help in understanding PojoSerializer

Hi Sachin, Check the last few comments I wrote in this thread: https://lists.apache.org/thread/l71d1cqo9xv8rsw0gfjo19kb1pct2xj1 Regards, Alexis. On Wed, 20 Mar 2024, 18:51 Sachin Mittal, wrote: > Hi, > I saw the post but I did not understand how I would configure these fields > to use those

Re: Need help in understanding PojoSerializer

Hi, I saw the post but I did not understand how I would configure these fields to use those serializers. (I can change the set type to a list type for now). As per the docs I see that we can annotate fields with @TypeInfo But what I did not get is how using this annotation I can use

Re: Need help in understanding PojoSerializer

Flink doesn’t have built-in support for serializing Sets. See this (stale) issue about the same: https://issues.apache.org/jira/browse/FLINK-16729 You could create a custom serializer for sets, see

Need help in understanding PojoSerializer

Hi, I have a Pojo class like this public class A { public String str; public Set aSet; public Map dMap; } However when I start the flink program I get this message: org.apache.flink.api.java.typeutils.TypeExtractor[] - Field A# dMap will be processed as GenericType. Please read

End-to-end lag spikes when closing a large number of panes

We run a large-scale Flink 1.16 cluster that uses windowed aggregations and we’re seeing lag spikes on window closure. I’m curious if others have encountered similar issues before and if anyone has suggestions for how to tackle this problem (other than simply increasing parallelism). ContextLag

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations Best Regards Peter Huang On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang wrote: > > Congratulations > > > > Best, > Huajie Wang > > > > Leonard Xu 于2024年3月20日周三 21:36写道： > >> Hi devs and users, >> >> We are thrilled to announce that the donation of Flink CDC as a >> sub-project

Announcing the Community Over Code 2024 Streaming Track

Hi all, Community Over Code , the ASF conference, will be held in Denver, Colorado, October 7-10, 2024. The call for presentations is open

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations Best, Huajie Wang Leonard Xu 于2024年3月20日周三 21:36写道： > Hi devs and users, > > We are thrilled to announce that the donation of Flink CDC as a > sub-project of Apache Flink has completed. We invite you to explore the new > resources available: > > - GitHub Repository:

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Congratulations Best, Huajie Wang Leonard Xu 于2024年3月20日周三 21:36写道： > Hi devs and users, > > We are thrilled to announce that the donation of Flink CDC as a > sub-project of Apache Flink has completed. We invite you to explore the new > resources available: > > - GitHub Repository:

[ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Hi devs and users, We are thrilled to announce that the donation of Flink CDC as a sub-project of Apache Flink has completed. We invite you to explore the new resources available: - GitHub Repository: https://github.com/apache/flink-cdc - Flink CDC Documentation:

[ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

Hi devs and users, We are thrilled to announce that the donation of Flink CDC as a sub-project of Apache Flink has completed. We invite you to explore the new resources available: - GitHub Repository: https://github.com/apache/flink-cdc - Flink CDC Documentation:

Is there any options to control the file names in file sink

Hi. Anyone know if it’s possible to control the file names eg change the uuid file names and extensions to something else. I know I can control the path but I would like to be able to set the file name and the extension based on the data in the stream. So I can’t use any general pre or

Re: flink operator 高可用任务偶发性报错unable to update ConfigMapLock

这种一般是因为APIServer那边有问题导致单次的ConfigMap renew lease annotation的操作失败，Flink默认会重试的 如果你发现因为这个SocketTimeoutException原因导致了任务Failover，可以把下面两个参数调大 high-availability.kubernetes.leader-election.lease-duration: 60s high-availability.kubernetes.leader-election.renew-deadline: 60s Best, Yang On Tue, Mar 12,

Re: can we use Scan Newly Added Tables without restarting the existing job ?

Hi, 3pang zhu. This `Scan Newly added tables` feature requires restarting the job from the savepoint. We cannot add new tables to the running job without restarting by now. Best, Hang 3pang zhu 于2024年3月20日周三 15:22写道： > this link has describe the usage for [Scan Newly Added Tables] >

can we use Scan Newly Added Tables without restarting the existing job ?

this link has describe the usage for [Scan Newly Added Tables] https://nightlies.apache.org/flink/flink-cdc-docs-release-3.0/docs/connectors/legacy-flink-cdc-sources/mysql-cdc/#scan-newly-added-tables . if we can use if without restarting job. i have try this patch, use a schedule task in

Re: Elasticsearch sink suppport

Hi Tauseef, > Any version greater than 1.17 does not support elasticsearch sink connector Can you explain in more detail what does not support mean here? IIRC, flink-connector-elasticsearch is guaranteed to compile in 1.18 and 1.19 via nightly CI workflow. Best regards, Weijie Tauseef

退订

退订

Elasticsearch sink suppport

Dear Team, We are using flink 1.17.2 as of now and we use elasticsearch. Any version greater than 1.17 does not support elasticsearch sink connector or as a matter of fact any connector like mongodb, opensearch, cassandra, etc.. Is there any reason for this? What should we do to upgrade to

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congrats and thanks all for the efforts! Best Regards, Yu On Tue, 19 Mar 2024 at 11:51, gongzhongqiang wrote: > > Congrats! Thanks to everyone involved! > > Best, > Zhongqiang Gong > > Lincoln Lee 于2024年3月18日周一 16:27写道： >> >> The Apache Flink community is very happy to announce the release of

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congrats and thanks all for the efforts! Best Regards, Yu On Tue, 19 Mar 2024 at 11:51, gongzhongqiang wrote: > > Congrats! Thanks to everyone involved! > > Best, > Zhongqiang Gong > > Lincoln Lee 于2024年3月18日周一 16:27写道： >> >> The Apache Flink community is very happy to announce the release of

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congrats! Thanks to everyone involved! Best, Zhongqiang Gong Lincoln Lee 于2024年3月18日周一 16:27写道： > The Apache Flink community is very happy to announce the release of Apache > Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 series. > > Apache Flink® is an open-source stream

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congrats! Thanks to everyone involved! Best, Zhongqiang Gong Lincoln Lee 于2024年3月18日周一 16:27写道： > The Apache Flink community is very happy to announce the release of Apache > Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 series. > > Apache Flink® is an open-source stream

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congratulations Best, Weihua On Tue, Mar 19, 2024 at 10:56 AM Rodrigo Meneses wrote: > Congratulations > > On Mon, Mar 18, 2024 at 7:43 PM Yu Chen wrote: > > > Congratulations! > > Thanks to release managers and everyone involved! > > > > Best, > > Yu Chen > > > > > > > 2024年3月19日

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congratulations Best, Weihua On Tue, Mar 19, 2024 at 10:56 AM Rodrigo Meneses wrote: > Congratulations > > On Mon, Mar 18, 2024 at 7:43 PM Yu Chen wrote: > > > Congratulations! > > Thanks to release managers and everyone involved! > > > > Best, > > Yu Chen > > > > > > > 2024年3月19日

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congratulations! Thanks to release managers and everyone involved! Best, Yu Chen > 2024年3月19日 01:01，Jeyhun Karimov 写道： > > Congrats! > Thanks to release managers and everyone involved. > > Regards, > Jeyhun > > On Mon, Mar 18, 2024 at 9:25 AM Lincoln Lee wrote: > >> The Apache Flink

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congratulations! Thanks to release managers and everyone involved! Best, Yu Chen > 2024年3月19日 01:01，Jeyhun Karimov 写道： > > Congrats! > Thanks to release managers and everyone involved. > > Regards, > Jeyhun > > On Mon, Mar 18, 2024 at 9:25 AM Lincoln Lee wrote: > >> The Apache Flink

Re: Question around manually setting Flink jobId

Thanks for the response, Asimansu. I should have been a bit more clearer and shared some additional context on our internal deployment. Currently, we are running *Flink in YARN application mode* for *batch* execution purposes (we also run it for stream execution as well). In the YARN application

Re: Help with using multiple Windows in the Table API

Thank you! looking though that helped me figure out what i needed to do! at first i was nervous because the docs you sent me are using the SQL approach and i wanted to use the python code i had, but the i noticed the the example SQL it was passing rowtime from one window to another. I

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congrats! Thanks to release managers and everyone involved. Regards, Jeyhun On Mon, Mar 18, 2024 at 9:25 AM Lincoln Lee wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.19.0, which is the fisrt release for the Apache Flink 1.19 series. > > Apache

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congratulations Best, Ron Yanfei Lei 于2024年3月18日周一 20:01写道： > Congrats, thanks for the great work! > > Sergey Nuyanzin 于2024年3月18日周一 19:30写道： > > > > Congratulations, thanks release managers and everyone involved for the > great work! > > > > On Mon, Mar 18, 2024 at 12:15 PM Benchao Li >

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congratulations Best, Ron Yanfei Lei 于2024年3月18日周一 20:01写道： > Congrats, thanks for the great work! > > Sergey Nuyanzin 于2024年3月18日周一 19:30写道： > > > > Congratulations, thanks release managers and everyone involved for the > great work! > > > > On Mon, Mar 18, 2024 at 12:15 PM Benchao Li >

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congrats, thanks for the great work! Sergey Nuyanzin 于2024年3月18日周一 19:30写道： > > Congratulations, thanks release managers and everyone involved for the great > work! > > On Mon, Mar 18, 2024 at 12:15 PM Benchao Li wrote: >> >> Congratulations! And thanks to all release managers and everyone >>

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congrats, thanks for the great work! Sergey Nuyanzin 于2024年3月18日周一 19:30写道： > > Congratulations, thanks release managers and everyone involved for the great > work! > > On Mon, Mar 18, 2024 at 12:15 PM Benchao Li wrote: >> >> Congratulations! And thanks to all release managers and everyone >>

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congratulations, thanks release managers and everyone involved for the great work! On Mon, Mar 18, 2024 at 12:15 PM Benchao Li wrote: > Congratulations! And thanks to all release managers and everyone > involved in this release! > > Yubin Li 于2024年3月18日周一 18:11写道： > > > > Congratulations! > >

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congratulations, thanks release managers and everyone involved for the great work! On Mon, Mar 18, 2024 at 12:15 PM Benchao Li wrote: > Congratulations! And thanks to all release managers and everyone > involved in this release! > > Yubin Li 于2024年3月18日周一 18:11写道： > > > > Congratulations! > >

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congratulations! And thanks to all release managers and everyone involved in this release! Yubin Li 于2024年3月18日周一 18:11写道： > > Congratulations! > > Thanks to release managers and everyone involved. > > On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu wrote: > > > > Congratulations! > > Thanks

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congratulations! And thanks to all release managers and everyone involved in this release! Yubin Li 于2024年3月18日周一 18:11写道： > > Congratulations! > > Thanks to release managers and everyone involved. > > On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu wrote: > > > > Congratulations! > > Thanks

Re: Read configmap data in FlinkDeployment

This looks like an indentation error, just indent everything by two extra spaces after line 13. On Mon, Mar 18, 2024 at 3:19 AM Suyash Jaiswal wrote: > Hi Yaroslav, > > > > Thanks for the swift reply, I understand that it goes under *podTemplate, > *but can I get some sample where configmap is

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congratulations! Thanks to release managers and everyone involved. On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu wrote: > > Congratulations! > Thanks release managers and all involved! > > On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan wrote: > > > Congratulations! > > > > Best, > > Hang > > > >

Re: [ANNOUNCE] Apache Flink 1.19.0 released

Congratulations! Thanks to release managers and everyone involved. On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu wrote: > > Congratulations! > Thanks release managers and all involved! > > On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan wrote: > > > Congratulations! > > > > Best, > > Hang > > > >

Re: Temporal join on rolling aggregate

Hi everyone, Before digging into what it would it take to implement a general solution, I narrowed down the scope to write a fix which makes the query mentioned in the thread work. Here are some findings: - For the temporal join logic, it's not the watermark that matters but having a

<    2   3   4   5   6   7   8   9   10   11   >