Re: [DISCUSS] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction

2024-05-02 Thread Péter Váry
Hi Martijn, We might want to add FLIP-371 [1] to the list. (Or we aim only for higher level FLIPs?) We are in the process of using the new API in Iceberg connector [2] - so far, so good. I know of one minor known issue about the sink [3], which should be ready for the release. All-in-all, I

[jira] [Created] (FLINK-35286) Cannot discover Hive connector outside Hive catalog

2024-05-02 Thread Ryan Goldenberg (Jira)
Ryan Goldenberg created FLINK-35286: --- Summary: Cannot discover Hive connector outside Hive catalog Key: FLINK-35286 URL: https://issues.apache.org/jira/browse/FLINK-35286 Project: Flink

Re: [VOTE] Apache Flink CDC Release 3.1.0, release candidate #1

2024-05-02 Thread Muhammet Orazov
Hey Qingsheng, Thanks a lot! +1 (non-binding) - Checked sha512sum hash - Checked GPG signature - Reviewed release notes - Reviewed GitHub web pr (added minor suggestions) - Built the source with JDK 11 & 8 - Checked that src doesn't contain binary files Best, Muhammet On 2024-04-30 05:11,

[jira] [Created] (FLINK-35285) Autoscaler key group optimization can interfere with scale-down.max-factor

2024-05-02 Thread Trystan (Jira)
Trystan created FLINK-35285: --- Summary: Autoscaler key group optimization can interfere with scale-down.max-factor Key: FLINK-35285 URL: https://issues.apache.org/jira/browse/FLINK-35285 Project: Flink

[jira] [Created] (FLINK-35284) Streaming File Sink end-to-end test times out

2024-05-02 Thread Ryan Skraba (Jira)
Ryan Skraba created FLINK-35284: --- Summary: Streaming File Sink end-to-end test times out Key: FLINK-35284 URL: https://issues.apache.org/jira/browse/FLINK-35284 Project: Flink Issue Type: Bug

Re: [VOTE] FLIP-454: New Apicurio Avro format

2024-05-02 Thread Nic Townsend
+1 I believe expanding the supported list of schema registries is a valuable addition. -- Kind regards Nic Townsend Sofware Developer – IBM Event Automation Slack: @nictownsend X: @nict0wnsend From: David Radley Date: Thursday, 2 May 2024 at 10:43 To: dev@flink.apache.org Subject:

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-02 Thread Aleksandr Pilipenko
Hi Piotr, Thanks for the proposal. How adding a s5cmd will affect memory footprint? Since this is a native binary, memory consumption will not be controlled by JVM or Flink. Thanks, Aleksandr On Thu, 2 May 2024 at 11:12, Hong Liang wrote: > Hi Piotr, > > Thanks for the FLIP! Nice to see work

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-02 Thread Hong Liang
Hi Piotr, Thanks for the FLIP! Nice to see work to improve the filesystem performance. +1 to future work to improve the upload speed as well. This would be useful for jobs with large state and high Async checkpointing times. Some thoughts on the configuration, it might be good for us to

[jira] [Created] (FLINK-35283) Add support unique Kafka producer client ids

2024-05-02 Thread Francis (Jira)
Francis created FLINK-35283: --- Summary: Add support unique Kafka producer client ids Key: FLINK-35283 URL: https://issues.apache.org/jira/browse/FLINK-35283 Project: Flink Issue Type: Improvement

RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-05-02 Thread David Radley
Hi Martijn, I have started a vote thread – please could you update the Flip with the link to the vote thread, Kind regards, David. From: David Radley Date: Thursday, 2 May 2024 at 10:39 To: dev@flink.apache.org Subject: [EXTERNAL] RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

[VOTE] FLIP-454: New Apicurio Avro format

2024-05-02 Thread David Radley
Hi everyone, I'd like to start a vote on the FLIP-454: New Apicurio Avro format [1]. The discussion thread is here [2]. The vote will be open for at least 72 hours unless there is an objection or insufficient votes. [1]

RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-05-02 Thread David Radley
Fabulous, thanks Martijn  From: Martijn Visser Date: Thursday, 2 May 2024 at 10:08 To: dev@flink.apache.org Subject: [EXTERNAL] Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format Done :) On Thu, May 2, 2024 at 11:01 AM David Radley wrote: > Hi Martijn, > Thank you very much for looking at

[jira] [Created] (FLINK-35282) Support for Apache Beam > 2.49

2024-05-02 Thread APA (Jira)
APA created FLINK-35282: --- Summary: Support for Apache Beam > 2.49 Key: FLINK-35282 URL: https://issues.apache.org/jira/browse/FLINK-35282 Project: Flink Issue Type: Improvement Reporter:

Re: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-05-02 Thread Martijn Visser
Done :) On Thu, May 2, 2024 at 11:01 AM David Radley wrote: > Hi Martijn, > Thank you very much for looking at this. In response to your feedback; I > produced a reduced version which is on this link. > > >

RE: FW: RE: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-05-02 Thread David Radley
Hi Martijn, Thank you very much for looking at this. In response to your feedback; I produced a reduced version which is on this link. https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing The original version you have copied is a bit out-dated and

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-02 Thread Muhammet Orazov
Hey Piotr, Thanks for the proposal! It would be great improvement! Some questions from my side: In order to configure s5cmd Flink’s user would need to specify path to the s5cmd binary. Could you please also add the configuration property for this? An example showing how users would set this

Re: [DISCUSS] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction

2024-05-02 Thread Muhammet Orazov
Got it, thanks! On 2024-05-02 06:53, Martijn Visser wrote: Hi Muhammet, Thanks for joining the discussion! The changes in this FLIP would be targeted for Flink 1.19, since it's only a matter of changing the annotation. Best regards, Martijn On Thu, May 2, 2024 at 7:26 AM Muhammet Orazov

Re: [Discuss] FLIP-452: Allow Skipping Invocation of Function Calls While Constant-folding

2024-05-02 Thread Muhammet Orazov
Hey Alan, Thanks for the proposal, +1! The `isDeterministic()`[1] function is mentioned in the documentation, I would suggest to add maybe a section for `supportsConstantFolding()`, with short description and examples use cases (similar to the motivation of the FLIP) where this could be useful

Re: [DISCUSS] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction

2024-05-02 Thread Martijn Visser
Hi Muhammet, Thanks for joining the discussion! The changes in this FLIP would be targeted for Flink 1.19, since it's only a matter of changing the annotation. Best regards, Martijn On Thu, May 2, 2024 at 7:26 AM Muhammet Orazov wrote: > Hello Martijn, > > Thanks for the FLIP and detailed

[jira] [Created] (FLINK-35281) FlinkEnvironmentUtils#addJar add each jar only once

2024-05-02 Thread Hongshun Wang (Jira)
Hongshun Wang created FLINK-35281: - Summary: FlinkEnvironmentUtils#addJar add each jar only once Key: FLINK-35281 URL: https://issues.apache.org/jira/browse/FLINK-35281 Project: Flink Issue