Re: [VOTE] FLIP-295: Support lazy initialization of catalogs and persistence of catalog configurations

2023-06-14 Thread Dong Lin
Thanks Feng for the FLIP. +1(binding) Cheers, Dong On Wed, Jun 14, 2023 at 10:35 AM Feng Jin wrote: > Hi everyone > > Thanks for all the feedback about the FLIP-295: Support lazy initialization > of catalogs and persistence of catalog configurations[1]. > [2] is the discussion thread. > > >

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-14 Thread Xintong Song
I agree that Public APIs should require a longer migration period. I think that is why the FLIP requires at least 2 minor releases (compared to 1 minor release for PublicEvolving and 1 patch release for Experimental). I think the key of stability guarantees is about not breaking the commitments

Re: Re: [VOTE] FLIP-311: Support Call Stored Procedure

2023-06-14 Thread Jane Chan
+1 (non-binding) Best, Jane On Wed, Jun 14, 2023 at 10:41 AM Feng Jin wrote: > +1 (no-binding) > > > Best, > Feng > > > On Wed, Jun 14, 2023 at 7:02 AM Jing Ge > wrote: > > > +1(binding) > > > > Best Regards, > > Jing > > > > On Tue, Jun 13, 2023 at 9:03 AM Mang Zhang wrote: > > > > > +1

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-14 Thread Lijie Wang
Hi Yuxia, Thanks for your feedback. The answers of your questions are as follows: 1. Yes, the row count comes from statistic of underlying table(Or estimated based on the statistic of underlying table, if the build side or probe side is not TableScan). If the statistic unavailable, we will not

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-06-14 Thread Lincoln Lee
Hi Aitozi, Thanks for your reply! Gives sql users more flexibility to get asynchronous processing capabilities via lateral join table function +1 for this For `JavaAsyncTableFunc0` in flip, can you use a scenario like RPC call as an example? For the name of this query hint, 'LATERAL' (include

Re: AsyncFunction vs Async Sink

2023-06-14 Thread Lu Niu
Thanks, Hong! I understand that if the user case is to simply write sth to an external service, Async Sink is a good option that provides features like batching, state management and rate limiting. I have some follow up questions: 1. Is there any problem if we use Async Function for such a user

Re: AsyncFunction vs Async Sink

2023-06-14 Thread Teoh, Hong
Hi Lu, Thanks for your question. See below for my understanding. I would recommend using the Async Sink if you are writing to the external service as the final output of your job graph, and if you don’t have the ordered requirement that updates to the external system must be done before

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2023-06-14 Thread Tzu-Li (Gordon) Tai
Hi Mason, Thanks for addressing my comments. I agree that option 3 seems more reasonable. > Reorganize the metadata in a Map in `KafkaStream` where the String is the proposed `KafkaClusterIdentifier.name` field. Why not just Map? Regarding naming, I like DynamicKafkaSource as that's what I

AsyncFunction vs Async Sink

2023-06-14 Thread Lu Niu
Hi, Flink dev and users If I want to async write to an external service, which API shall I use, AsyncFunction or Async Sink? My understanding after checking the code are: 1. Both APIs guarantee at least once write to external service. As both API internally stores in-flight requests in

[jira] [Created] (FLINK-32341) Fix resource requirements rest API response empty result

2023-06-14 Thread chenyuexin (Jira)
chenyuexin created FLINK-32341: -- Summary: Fix resource requirements rest API response empty result Key: FLINK-32341 URL: https://issues.apache.org/jira/browse/FLINK-32341 Project: Flink Issue

Re: [VOTE] FLIP-295: Support lazy initialization of catalogs and persistence of catalog configurations

2023-06-14 Thread Jing Ge
+1 (binding) Best Regards, Jing On Wed, Jun 14, 2023 at 3:28 PM Rui Fan <1996fan...@gmail.com> wrote: > +1(binding) > > Best, > Rui Fan > > On Wed, Jun 14, 2023 at 16:24 Hang Ruan wrote: > > > +1 (non-binding) > > > > Thanks for Feng driving it. > > > > Best, > > Hang > > > > Feng Jin

Re: [VOTE] FLIP-294: Support Customized Catalog Modification Listener

2023-06-14 Thread Jing Ge
+1 (binding) Best Regards, Jing On Wed, Jun 14, 2023 at 4:07 PM Benchao Li wrote: > +1 (binding) > > Shammon FY 于2023年6月14日周三 19:52写道: > > > Hi all: > > > > Thanks for all the feedback for FLIP-294: Support Customized Catalog > > Modification Listener [1]. I would like to start a vote for it

[jira] [Created] (FLINK-32340) NPE in K8s operator which brakes current and subsequent deployments

2023-06-14 Thread Sergii Nazarov (Jira)
Sergii Nazarov created FLINK-32340: -- Summary: NPE in K8s operator which brakes current and subsequent deployments Key: FLINK-32340 URL: https://issues.apache.org/jira/browse/FLINK-32340 Project:

Re: [VOTE] FLIP-294: Support Customized Catalog Modification Listener

2023-06-14 Thread Benchao Li
+1 (binding) Shammon FY 于2023年6月14日周三 19:52写道: > Hi all: > > Thanks for all the feedback for FLIP-294: Support Customized Catalog > Modification Listener [1]. I would like to start a vote for it according to > the discussion in thread [2]. > > The vote will be open for at least 72

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-14 Thread Becket Qin
Thanks for the explanation, Matthias. In the example you raised, would it be better to just keep both YARN and K8S support in the new major version, but with YARN support deprecated if we want to? We can say for YARN we will only provide bug fixes but no feature development anymore. Given these

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

2023-06-14 Thread Etienne Chauchot
Hi all, @Yukia,I updated the FLIP to include the aggregation of the staked operations that we discussed below PTAL. Best Etienne Le 13/06/2023 à 16:31, Etienne Chauchot a écrit : Hi Yuxia, Thanks for your feedback. The number of potentially stacked operations depends on the configured

Re: [VOTE] FLIP-295: Support lazy initialization of catalogs and persistence of catalog configurations

2023-06-14 Thread Rui Fan
+1(binding) Best, Rui Fan On Wed, Jun 14, 2023 at 16:24 Hang Ruan wrote: > +1 (non-binding) > > Thanks for Feng driving it. > > Best, > Hang > > Feng Jin 于2023年6月14日周三 10:36写道: > > > Hi everyone > > > > Thanks for all the feedback about the FLIP-295: Support lazy > initialization > > of

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-14 Thread Matthias Pohl
One (made-up) example from the top of my head would have been that the community decides to focus fully on Kubernetes without considering Yarn anymore because of some must-have feature on the Kubernetes side. At the same time there are still some users for whom it would be tricky to migrate from

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-14 Thread yuxia
Thanks Lijie for starting this discussion. Excited to see runtime filter is to be implemented in Flink. I have few questions about it: 1: As the FLIP said, `if the ndv cannot be estimated, use row count instead`. So, does row count comes from the statistic from underlying table? What if the

[VOTE] FLIP-294: Support Customized Catalog Modification Listener

2023-06-14 Thread Shammon FY
Hi all: Thanks for all the feedback for FLIP-294: Support Customized Catalog Modification Listener [1]. I would like to start a vote for it according to the discussion in thread [2]. The vote will be open for at least 72 hours(excluding weekends, until June 19, 19:00 PM GMT) unless there is an

[jira] [Created] (FLINK-32339) Align lock usage in DefaultLeaderElectionService

2023-06-14 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-32339: - Summary: Align lock usage in DefaultLeaderElectionService Key: FLINK-32339 URL: https://issues.apache.org/jira/browse/FLINK-32339 Project: Flink Issue

[jira] [Created] (FLINK-32338) Add FailsOnJava17 annotation

2023-06-14 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-32338: Summary: Add FailsOnJava17 annotation Key: FLINK-32338 URL: https://issues.apache.org/jira/browse/FLINK-32338 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-32337) SQL array_union could return wrong result

2023-06-14 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-32337: --- Summary: SQL array_union could return wrong result Key: FLINK-32337 URL: https://issues.apache.org/jira/browse/FLINK-32337 Project: Flink Issue Type:

[jira] [Created] (FLINK-32336) PartitionITCase#ComparablePojo should be public

2023-06-14 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-32336: Summary: PartitionITCase#ComparablePojo should be public Key: FLINK-32336 URL: https://issues.apache.org/jira/browse/FLINK-32336 Project: Flink

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-14 Thread Becket Qin
Hi Matthias, Thanks for the feedback. Do you have an example of behavioral change in mind? Not sure I fully understand the concern for behavioral change here. From what I understand, any user sensible change in an existing API, regardless of its kind (API signature or behavioral change), can

Re: [DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-14 Thread liu ron
Thanks Lijie start this discussion. Runtime Filter is a common optimization to improve the join performance that has been adopted by many computing engines such as Spark, Doris, etc... Flink is a streaming batch computing engine, and we are continuously optimizing the performance of batches.

[DISCUSS] FLIP-324: Introduce Runtime Filter for Flink Batch Jobs

2023-06-14 Thread Lijie Wang
Hi devs Ron Liu, Gen Luo and I would like to start a discussion about FLIP-324: Introduce Runtime Filter for Flink Batch Jobs[1] Runtime Filter is a common optimization to improve join performance. It is designed to dynamically generate filter conditions for certain Join queries at runtime to

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-14 Thread Becket Qin
Hi Xintong, Thanks for the comment. Please see the replies below: 1. Do we allow deprecation & removal of APIs without adding a new one as a > replacement? The examples in the table give me an impression that marking > an API as `@Deprecated` should only happen at the same time of introducing >

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-14 Thread Matthias Pohl
Thanks for starting this discussion, Becket. A few good points were raised. Here's what I want to add: Stefan raised the point of behavioral stability (in contrast to API stability). That might be a reason for users to not be able to go ahead with a major version bump. Working around behavioral

[jira] [Created] (FLINK-32335) Fix the Flink ML unittest failure

2023-06-14 Thread Jiang Xin (Jira)
Jiang Xin created FLINK-32335: - Summary: Fix the Flink ML unittest failure Key: FLINK-32335 URL: https://issues.apache.org/jira/browse/FLINK-32335 Project: Flink Issue Type: New Feature

Re: [VOTE] FLIP-295: Support lazy initialization of catalogs and persistence of catalog configurations

2023-06-14 Thread Hang Ruan
+1 (non-binding) Thanks for Feng driving it. Best, Hang Feng Jin 于2023年6月14日周三 10:36写道: > Hi everyone > > Thanks for all the feedback about the FLIP-295: Support lazy initialization > of catalogs and persistence of catalog configurations[1]. > [2] is the discussion thread. > > > I'd like to

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-06-14 Thread Aitozi
Hi Lincoln Very thanks for your valuable question. I will try to answer your questions inline. >Does the async udtf bring any additional benefits besides a lighter implementation? IMO, async udtf is more than a lighter implementation. It can act as a general way for sql users to use the

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-14 Thread Xintong Song
Thanks for bringing up this discussion, Becket. My two cents: 1. Do we allow deprecation & removal of APIs without adding a new one as a replacement? The examples in the table give me an impression that marking an API as `@Deprecated` should only happen at the same time of introducing a new

[jira] [Created] (FLINK-32334) Operator failed to create taskmanager deployment because it already exist

2023-06-14 Thread Nicolas Fraison (Jira)
Nicolas Fraison created FLINK-32334: --- Summary: Operator failed to create taskmanager deployment because it already exist Key: FLINK-32334 URL: https://issues.apache.org/jira/browse/FLINK-32334

Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

2023-06-14 Thread Becket Qin
Hi Jing, Thanks for the feedback. Please see the answers to your questions below: *"Always add a "Since X.X.X" comment to indicate when was a class / > interface / method marked as deprecated."* > Could you describe it with a code example? Do you mean Java comments? It is just a comment such

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-06-14 Thread Lincoln Lee
Hi Aitozi, Sorry for the lately reply here! Supports async udtf(`AsyncTableFunction`) directly in sql seems like an attractive feature, but there're two issues that need to be addressed before we can be sure to add it: 1. As mentioned in the flip[1], the current lookup function can already