Re: [DISCUSS] Implementing SQL remote functions

Hi Alan I believe that supporting asynchronous UDF is a valuable feature. Currently, there is a similar FLIP[1] available: Can this meet your needs? [1]. https://cwiki.apache.org/confluence/display/FLINK/FLIP-313%3A+Add+support+of+User+Defined+AsyncTableFunction Best, Feng On Thu, Sep 21,

Re: [DISCUSS] Implementing SQL remote functions

Hi Ron, Thanks for your response. I've answered some of your questions below. I think one solution is to support Mini-Batch Lookup Join by the framework > layer, do a RPC call by a batch input row, which can improve throughput. Would the idea be to collect a batch and then do a single RPC (or

Re: [DISCUSS] FLIP-368 Reorganize the exceptions thrown in state interfaces

Hi Martijn, Thanks for the reminder! This FLIP proposes a change to the state API that is annotated as @PublicEvolving and targets version 1.19. I have clarified this in the "Proposed Change" section of the FLIP. Hi Jing, Thanks for sharing your thoughts! Here are my opinions: 1. The

Re: [Re-DISCUSS] FLIP-202: Introduce ClickHouse Connector

I share same concern with Matijn, how about invite the author of unofficial flink-clickhouse-connector itinycheng to co-author the FLIP? @ConradJam And then user would like to use the new official connector from users’ perspective, the maintaining work of unofficial CK connector and the new CK

Re: [VOTE] FLIP-312: Prometheus Sink Connector

Thanks Lorenzo for driving this. +1(binding) Best, Leonard > On Sep 21, 2023, at 11:47 AM, Yun Tang wrote: > > +1 (binding) > > Thanks for driving this, Lorenzo. > > Best > Yun Tang > > From: Hong > Sent: Thursday, September 21, 2023 1:22 > To:

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

Hi, Zakelly, No benchmark tests currently are affected by this issue. We may add benchmarks to guard it later. Thanks. Best, Yuxin Zakelly Lan 于2023年9月21日周四 11:56写道： > Hi Jing, > > Sure, I will run the benchmark with this fix. > > Hi Yunxin, > > I'm not familiar with the hybrid shuffle. Is

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

Hi Jing, Sure, I will run the benchmark with this fix. Hi Yunxin, I'm not familiar with the hybrid shuffle. Is there any specific benchmark test that may be affected by this issue? I will pay special attention to it. Thanks. Best, Zakelly On Thu, Sep 21, 2023 at 10:08 AM Yuxin Tan wrote: >

Re: [VOTE] FLIP-312: Prometheus Sink Connector

+1 (binding) Thanks for driving this, Lorenzo. Best Yun Tang From: Hong Sent: Thursday, September 21, 2023 1:22 To: dev@flink.apache.org Subject: Re: [VOTE] FLIP-312: Prometheus Sink Connector +1 (binding) Thanks Lorenzo. Hong > On 20 Sep 2023, at 17:49,

Re: [Discuss] FLIP-362: Support minimum resource limitation

Thanks for the reply, Shammon. As the example described in my last response, an application could contain multiple jobs, both batch and streaming. I don't lean to disable it in Application mode in case users want to leverage it to accelerate the preceding batch jobs in their application. Best,

Re: [Discuss] FLIP-362: Support minimum resource limitation

Hi, I agree that `minimum resource limitation` will bring values for flink session clusters, but for `Application Mode`, is it useful for streaming and batch jobs? Is it necessary for us to not support the application mode, rather than relying on the default value 0? Best, Shammon FY On Thu,

Re: [Discuss] FLIP-362: Support minimum resource limitation

Hi Jing, Thanks for pointing this out. As described by Yangze, the min resource option will be set to 0 by default and the oscillation check will be disabled at then. In most cases, common streaming jobs won't be affected by this new added option. I've updated the FLIP to explain this. Thx,

[jira] [Created] (FLINK-33127) HeapKeyedStateBackend: use buffered I/O to speed up local recovery

Yangyang ZHANG created FLINK-33127: -- Summary: HeapKeyedStateBackend: use buffered I/O to speed up local recovery Key: FLINK-33127 URL: https://issues.apache.org/jira/browse/FLINK-33127 Project:

Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration

Hi Jane, Thank you for your valuable feedback and suggestions. I agree with your point about differentiating between "flink-config.yaml" and "flink-conf.yaml" to determine the standard syntax at a glance. While I understand your suggestion of using "flink-conf-default.yaml" to represent the

Re: [Discuss] FLIP-362: Support minimum resource limitation

Thanks for the comments, Jing. > Will the minimum resource configuration also take effect for streaming jobs > in application mode? > Since it is not recommended to configure slotmanager.number-of-slots.max for > streaming jobs, does it make sense to disable it for common streaming jobs? > At

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

Hi, Jing, Qingsheng, Thanks a lot. The fix has been backported. Best, Yuxin Jing Ge 于2023年9月21日周四 00:42写道： > Hi Lijie, > > Thanks for reaching out. Please backport it to release-1.18. > > Best regards, > Jing > > On Wed, Sep 20, 2023 at 4:35 PM Lijie Wang > wrote: > > > Hi community and

Re: [Discuss] FLIP-362: Support minimum resource limitation

Hi Xiangyu, Thanks for driving it! There is one thing I am not really sure if I understand you correctly. According to the FLIP: "The minimum resource limitation will be implemented in the DefaultResourceAllocationStrategy of FineGrainedSlotManager. Each time when SlotManager needs to reconcile

Re: [VOTE] FLIP-312: Prometheus Sink Connector

+1 (binding) Thanks Lorenzo. Hong > On 20 Sep 2023, at 17:49, Danny Cranmer wrote: > > +1 binding. > > Thanks for picking this up Lorenzo! > > Danny > > >> On Wed, 20 Sept 2023, 16:33 Jing Ge, wrote: >> >> +1(binding) Thanks! >> >> Best regards, >> Jing >> >> On Wed, Sep 20, 2023 at

Re: [DISCUSS] FLIP-368 Reorganize the exceptions thrown in state interfaces

sorry, typo: It is a known "anti-pattern" instead of "ant-pattern" Best regards, Jing On Wed, Sep 20, 2023 at 7:23 PM Jing Ge wrote: > Hi Zakelly, > > Thanks for driving this topic. From good software engineering's > perspective, I have different thoughts: > > 1. The idea to get rid of all

Re: [DISCUSS] FLIP-368 Reorganize the exceptions thrown in state interfaces

Hi Zakelly, Thanks for driving this topic. From good software engineering's perspective, I have different thoughts: 1. The idea to get rid of all checked Exceptions and replace them with unchecked Exceptions is a known ant-pattern: "Generally speaking, do not throw a RuntimeException or create a

Re: [VOTE] FLIP-312: Prometheus Sink Connector

+1 binding. Thanks for picking this up Lorenzo! Danny On Wed, 20 Sept 2023, 16:33 Jing Ge, wrote: > +1(binding) Thanks! > > Best regards, > Jing > > On Wed, Sep 20, 2023 at 3:20 PM Martijn Visser > wrote: > > > +1 (binding) > > > > Thanks for driving this. Cheers, M > > > > On Mon, Sep 18,

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

Hi Lijie, Thanks for reaching out. Please backport it to release-1.18. Best regards, Jing On Wed, Sep 20, 2023 at 4:35 PM Lijie Wang wrote: > Hi community and release managers: > > We found a critical bug[1] of the rest client a few days ago, which may > cause the inode to be used up. Now the

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

Hi, dear community and release managers, Thanks for bringing this up. When testing the release candidate #0 for the batch scenario, I found an issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and may significantly impact the performance of shuffle writing. The

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

Thanks for the update, Lijie! The patch will be included in the next RC. Best, Qingsheng On Wed, Sep 20, 2023 at 10:12 PM Lijie Wang wrote: > Hi community and release managers: > > We found a critical bug[1] of the rest client a few days ago, which may > cause the inode to be used up. Now the

Re: [DISCUSS] Backport fix to release 1.18

Hi, Jing Thanks for the update. The delay is indeed long. Please ignore it and join the discussion in the original thread. Best, Yuxin Jing Ge 于2023年9月20日周三 22:01写道： > Hi folks, > > Please ignore this email. It seems the dev mail server has some issues > that emails have been sent but will

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

Hi Martijn, This FLIP has passed voting[1]. It is a modification on top of the FLIP-95 interface. Thanks, Jiangjie (Becket) Qin [1] https://lists.apache.org/thread/hysv9y1f48gtpr5vx3x40wtjb6cp9ky6 On Wed, Sep 20, 2023 at 9:29 PM Martijn Visser wrote: > For clarity purposes, this FLIP is

[jira] [Created] (FLINK-33123) Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler and

Zhanghao Chen created FLINK-33123: - Summary: Wrong dynamic replacement of partitioner from FORWARD to REBLANCE for autoscaler and adaptive scheduler and Key: FLINK-33123 URL:

Re: [DISCUSS] FLIP-367: Support Setting Parallelism for Table/SQL Sources

Hi Zhanghao, Thanks for the update. The FLIP now looks good to me in general, and I have two minor comments. 1. Compared with other subclasses like `CacheTransformation` or `PartitionTransformation`, the name `WrapperTransformation` seems too general. What about `SourceTransformationWrapper`,

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

Hi, dear community and release managers, Thanks for bringing this up. When testing the release candidate #0 for the batch scenario, I found an issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and may significantly impact the performance of shuffle writing. The

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

Hi, dear community and release managers, Thanks for bringing this up. When testing the release candidate #0 for the batch scenario, I found an issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and may significantly impact the performance of shuffle writing. The

Re: Future of classical remoting in Pekko

That is a fair point. It not fixes a bug per se, but it would mitigate security vulnerabilities (Netty 3.x CVEs), so my thought was it might qualify it for addressing in a patch release. IMO handling security vulnerabilities is a gray area, if it only requires to bump some deps that are only

Re: [Discuss] FLIP-366: Support standard YAML for FLINK configuration

Hi Junrui, Thanks for driving this FLIP. +1 for adoption of the standard YAML syntax. I just have one minor suggestion. It's a little bit challenging to differentiate between `flink-config.yaml` and `flink-conf.yaml` to determine which one uses the standard syntax at a glance. How about using

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

Hi, dear community and release managers, Thanks for bringing this up. When testing the release candidate #0 for the batch scenario, I found an issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and may significantly impact the performance of shuffle writing. The

Re: [DISCUSS] FLIP-368 Reorganize the exceptions thrown in state interfaces

Hi Yanfei, Thanks for your reply! Yes, this FLIP aims to change all state-related exceptions to unchecked exceptions and remove all exceptions from the signature. So I believe we have come to an agreement to keep the interfaces simple. Best regards, Zakelly On Wed, Sep 20, 2023 at 2:26 PM

Re: [DISCUSS] FLIP-368 Reorganize the exceptions thrown in state interfaces

Hi Hangxiang, Thank you for your response! Here are my thoughts: 1. Regarding the exceptions thrown by internal interfaces, I suggest keeping them as checked exceptions. Since these exceptions will be handled by the internal callers, it is meaningful to throw them as checked ones. If we need to

[jira] [Created] (FLINK-33125) Upgrade JOSDK to 4.4.4

Nicolas Fraison created FLINK-33125: --- Summary: Upgrade JOSDK to 4.4.4 Key: FLINK-33125 URL: https://issues.apache.org/jira/browse/FLINK-33125 Project: Flink Issue Type: Bug

Re: [DISCUSS] Backport fix to release 1.18

Hi folks, Please ignore this email. It seems the dev mail server has some issues that emails have been sent but will be received by dev@flink.apache.org with a big delay at [1]. Participants who have been sent to or cc directly should get the email instantly. Let's consolidate the discussion in

[DISCUSS] Backport fix to release 1.18

Hi, dear community and release managers, Thanks for bringing up the new release candidate for 1.18. When testing for the batch scenario, I found an issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and may significantly impact the performance of shuffle writing.

[jira] [Created] (FLINK-33122) [Benchmark] Null checkpoint directory in rescaling benchmarks

Zakelly Lan created FLINK-33122: --- Summary: [Benchmark] Null checkpoint directory in rescaling benchmarks Key: FLINK-33122 URL: https://issues.apache.org/jira/browse/FLINK-33122 Project: Flink

Re:Re: [ANNOUNCE] Release 1.18.0, release candidate #0

Hi, dear community and release managers, Thanks for bringing this up. When testing the release candidate #0 for the batch scenario, I found an issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and may significantly impact the performance of shuffle writing.

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

Thanks for letting us know, Yuxin. Please go ahead and backport it to 1.18. We will include the patch in the next release candidate. Best, Qingsheng On Wed, Sep 20, 2023 at 8:54 PM Yuxin Tan wrote: > Hi, dear community and release managers, > > Thanks for bringing this up. > > When testing

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

No, this is not abandoned. This is accepted with enough binding votes. I didn't get why you think that this FLIP is abandoned. Could you please clarify? On Wed, Sep 20, 2023, 6:11 AM Martijn Visser wrote: > For clarity purposes, this FLIP is being abandoned because it was part > of FLIP-95? > >

Re: [VOTE] FLIP-312: Prometheus Sink Connector

+1(binding) Thanks! Best regards, Jing On Wed, Sep 20, 2023 at 3:20 PM Martijn Visser wrote: > +1 (binding) > > Thanks for driving this. Cheers, M > > On Mon, Sep 18, 2023 at 1:51 PM Lorenzo Nicora > wrote: > > > > Hi All, > > > > Thanks for the feedback on FLIP-312: Prometheus Sink Connector

Re: [DISCUSS]clean up the savepoints compatibility table

Hi, Thanks for your feedbacks. I will clean up the table and add a link to the table of the previous release. Best regards, Jing On Mon, Sep 18, 2023 at 12:30 PM David Radley wrote: > Hi, > I like the proposal from Martijn. Having an ‘archived’ area for the older > material would make sense

Re: [VOTE] FLIP-307: Flink Connector Redshift

+1 (binding) Best, Leonard > On Sep 18, 2023, at 11:53 PM, Ahmed Hamdy wrote: > > +1 (non-binding) > Best Regards > Ahmed Hamdy > > > On Mon, 18 Sept 2023 at 16:52, Jing Ge wrote: > >> +1(binding). Thanks! >> >> Best regards, >> Jing >> >> On Mon, Sep 18, 2023 at 5:26 PM Danny Cranmer

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

Hi community and release managers: We found a critical bug[1] of the rest client a few days ago, which may cause the inode to be used up. Now the fix-PR[2] is ready for merging, I hope to backport it to release-1.18. Please let me know if you have any concerns. Thanks. [1]

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

Hi Yuxin, Thanks for the update. LGTM. @Zakelly Please please be aware of that and make sure the benchmarks test will include that fix. Thanks! Best regards, Jing On Wed, Sep 20, 2023 at 8:51 AM Yuxin Tan wrote: > Hi, dear community and release managers, > > Thanks for bringing this up. > >

Re: [DISCUSS] FLIP-368 Reorganize the exceptions thrown in state interfaces

Hi Zakelly, Thanks for the FLIP. In the FLIP, it's stated "Since the signature of the public state API has changed, user code that uses these APIs needs to be recompiled". Is this FLIP proposing to change signatures of APIs annotated as @Public and therefore targeting Flink 2.0, or is it

Re: [VOTE] FLIP-312: Prometheus Sink Connector

+1 (binding) Thanks for driving this. Cheers, M On Mon, Sep 18, 2023 at 1:51 PM Lorenzo Nicora wrote: > > Hi All, > > Thanks for the feedback on FLIP-312: Prometheus Sink Connector [1]. > We updated the FLIP accordingly [2]. > > I would like to open the vote on FLIP-312. > The vote will be open

Re: [VOTE] FLIP-307: Flink Connector Redshift

+1 (binding) On Mon, Sep 18, 2023 at 5:55 PM Ahmed Hamdy wrote: > > +1 (non-binding) > Best Regards > Ahmed Hamdy > > > On Mon, 18 Sept 2023 at 16:52, Jing Ge wrote: > > > +1(binding). Thanks! > > > > Best regards, > > Jing > > > > On Mon, Sep 18, 2023 at 5:26 PM Danny Cranmer > > wrote: > > >

[jira] [Created] (FLINK-33124) Kafka Connector not working for table

Aarsh Shah created FLINK-33124: -- Summary: Kafka Connector not working for table Key: FLINK-33124 URL: https://issues.apache.org/jira/browse/FLINK-33124 Project: Flink Issue Type: Bug

Re: [Re-DISCUSS] FLIP-202: Introduce ClickHouse Connector

Hi ConradJam, The FLIP still references the unofficial "flink-clickhouse-connector" which I don't really understand: you want to build a new implementation for this connector, right? Best regards, Martijn On Thu, Sep 7, 2023 at 12:12 PM ConradJam wrote: > > Does anyone else have an opinion on

Re: [DISCUSS] FLIP-356: Support Nested Fields Filter Pushdown

For clarity purposes, this FLIP is being abandoned because it was part of FLIP-95? On Thu, Sep 7, 2023 at 3:01 AM Venkatakrishnan Sowrirajan wrote: > > Hi everyone, > > Posted a PR (https://github.com/apache/flink/pull/23313) to add nested > fields filter pushdown. Please review. Thanks. > >

[jira] [Created] (FLINK-33126) Fix EventTimeAllWindowCheckpointingITCase jobName typo

Yue Ma created FLINK-33126: -- Summary: Fix EventTimeAllWindowCheckpointingITCase jobName typo Key: FLINK-33126 URL: https://issues.apache.org/jira/browse/FLINK-33126 Project: Flink Issue Type:

Re: [VOTE] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI

+1 (non-binding) Thanks for driving this FLIP -- Best, Matt Wang Replied Message | From | Weihua Hu | | Date | 09/19/2023 19:17 | | To | | | Subject | Re: [VOTE] FLIP-363: Unify the Representation of TaskManager Location in REST API and Web UI | +1(binding) Best, Weihua On

Re: Future of classical remoting in Pekko

Just chipping in that I don't think we should add Pekko changes in a patch release, because I think the Pekko related changes don't fix a bug. On Tue, Sep 19, 2023 at 9:06 PM Ferenc Csaky wrote: > > I think that is totally fine, because any Pekko related changes can only be > added to the first

Re: [ANNOUNCE] Release 1.18.0, release candidate #0

Hi, dear community and release managers, Thanks for bringing this up. When testing the release candidate #0 for the batch scenario, I found an issue of frequent flushing in Hybrid shuffle. It is a new bug introduced by 1.18 and may significantly impact the performance of shuffle writing. The