Re: [DISUCSS] Deprecate multiple APIs in 1.18

2023-07-04 Thread Xintong Song
Thanks all for the discussion. It seems to me there's a consensus on marking the following as deprecated in 1.18: - DataSet API - SourceFunction - Queryable State - All Scala APIs More time is needed for deprecating SinkFunction. I'll leave this discussion open for a few more days. And if

Re: [DISUCSS] Deprecate multiple APIs in 1.18

2023-07-04 Thread Xintong Song
Thanks for the input, Jing. I'd also be +1 for option 1. Best, Xintong On Mon, Jul 3, 2023 at 7:20 PM Jing Ge wrote: > Hi Xingtong, > > Option 1, secure plan would be: > > 1. graduate kafka, File, JDBC connectors to @Public > 2. graduate SinkV2 to @Public > 3. remove SinkFunction. > >

[jira] [Created] (FLINK-32539) Archunit violations started to fail in test_misc

2023-07-04 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-32539: --- Summary: Archunit violations started to fail in test_misc Key: FLINK-32539 URL: https://issues.apache.org/jira/browse/FLINK-32539 Project: Flink Issue

RE: Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-07-04 Thread 宇航 李
Hi Aitozi, I think it is necessary to add the following description in FLIP to express the difference between user-defined asynchronous table function and AsyncTableFunction: User-defined asynchronous table functions allow complex parameters (e.g., Row type) to be passed to function, which is

[jira] [Created] (FLINK-32538) CI build failed because node is corrupted when compiling

2023-07-04 Thread Yuxin Tan (Jira)
Yuxin Tan created FLINK-32538: - Summary: CI build failed because node is corrupted when compiling Key: FLINK-32538 URL: https://issues.apache.org/jira/browse/FLINK-32538 Project: Flink Issue

Re: [DISCUSS] Release 2.0 Work Items

2023-07-04 Thread Xintong Song
Hi ConradJam, I think Chesnay has already put his name as the Contributor for the two tasks you listed. Maybe you can reach out to him to see if you can collaborate on this. In general, I don't think contributing to a release 2.0 issue is much different from contributing to a regular issue. We

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

2023-07-04 Thread Dong Lin
Hi Piotr, Any suggestion on how we can practically move forward to address the target use-case? My understanding is that the current proposal does not have any correctness/performance issues. And it allows the extension to support all the extra use-case without having to throw away the proposed

Re: [DISCUSS] Deprecate SourceFunction APIs

2023-07-04 Thread Flavio Pompermaier
Hi all, I've tried to migrate my very simple Elasticsearch SourceFunction (that use scroll API and produce batch of documents) to new Source API, but I gave up because it's too complicated. It should much simpler to migrate that function to a bounded or unbounded source. Before removing

Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

2023-07-04 Thread Jing Ge
Hi Xuannan, Hi Dong Thanks for the Proposal! After reading the FLIP, I'd like to ask some questions: 1. Naming convention for boolean variables. It is recommended to follow JavaBean [1], i.e. objectReuseCompliant as the variable name with isObjectReuseCompliant() and setObjectReuseCompliant() as

Re: [DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems

2023-07-04 Thread Ran Tao
Thanks Dian Fu for sharing the externalized connector usage within flink-python. Looking forward to the solution to this problem, and if possible, can participate in the contribution of this problem, and thank you Martijn for the dependency usage that the externalized connector should follow.

[jira] [Created] (FLINK-32537) Add compatibility annotation for REST API classes

2023-07-04 Thread Hong Liang Teoh (Jira)
Hong Liang Teoh created FLINK-32537: --- Summary: Add compatibility annotation for REST API classes Key: FLINK-32537 URL: https://issues.apache.org/jira/browse/FLINK-32537 Project: Flink

Re: [DISCUSS] FLIP-307: Flink connector Redshift

2023-07-04 Thread Martijn Visser
Hi Samrat, The dependency between JDBC and AWS worries me a lot: we're already seeing that coupling causes a lot of issues down the line. Why can't we decouple these? Best regards, Martijn On Tue, Jul 4, 2023 at 3:35 PM Samrat Deb wrote: > Hi Leonard, > > Sorry for the late reply. > > > 1

Re: [DISCUSS] FLIP-307: Flink connector Redshift

2023-07-04 Thread Samrat Deb
Hi Leonard, Sorry for the late reply. > 1 Reusing the capabilities of JDBC and Filesystem in the Redshift connector generally makes sense to me. However, since all of them are managed in different repositories and depend on Flink dependency, could you explain how you establish the versioning,

[jira] [Created] (FLINK-32536) Python tests fail with Arrow DirectBuffer exception

2023-07-04 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-32536: Summary: Python tests fail with Arrow DirectBuffer exception Key: FLINK-32536 URL: https://issues.apache.org/jira/browse/FLINK-32536 Project: Flink

[jira] [Created] (FLINK-32535) CheckpointingStatisticsHandler periodically returns NullArgumentException after job restarts

2023-07-04 Thread Hong Liang Teoh (Jira)
Hong Liang Teoh created FLINK-32535: --- Summary: CheckpointingStatisticsHandler periodically returns NullArgumentException after job restarts Key: FLINK-32535 URL:

Re: [DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems

2023-07-04 Thread Dian Fu
Thanks Ran Tao for proposing this discussion and Martijn for sharing the thought. > While flink-python now fails the CI, it shouldn't actually depend on the externalized connectors. I'm not sure what PyFlink does with it, but if belongs to the connector code, For each DataStream connector,

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

2023-07-04 Thread David Morávek
> waiting 2 min between 2 requirements push seems ok to me This depends on the workload. Would you care if the cost of rescaling were close to zero (which is for most out-of-the-box workloads)? In that case, it would be desirable to rescale more frequently, for example, if TMs join incrementally.

[jira] [Created] (FLINK-32534) exit code 137 (i.e. OutOfMemoryError) in flink-tests module

2023-07-04 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-32534: --- Summary: exit code 137 (i.e. OutOfMemoryError) in flink-tests module Key: FLINK-32534 URL: https://issues.apache.org/jira/browse/FLINK-32534 Project: Flink

[jira] [Created] (FLINK-32533) exit code 137 (i.e. OutOfMemoryError) in python module

2023-07-04 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-32533: --- Summary: exit code 137 (i.e. OutOfMemoryError) in python module Key: FLINK-32533 URL: https://issues.apache.org/jira/browse/FLINK-32533 Project: Flink

[jira] [Created] (FLINK-32532) exit code 137 (i.e. OutOfMemoryError) in flink-s3-fs-hadoop module

2023-07-04 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-32532: --- Summary: exit code 137 (i.e. OutOfMemoryError) in flink-s3-fs-hadoop module Key: FLINK-32532 URL: https://issues.apache.org/jira/browse/FLINK-32532 Project:

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

2023-07-04 Thread Etienne Chauchot
Hi all, Thanks David for your feedback. My comments are inline Le 04/07/2023 à 09:16, David Morávek a écrit : They will struggle if they add new resources and nothing happens for 5 minutes. The same applies if they start playing with FLIP-291 APIs. I'm wondering if the cooldown makes sense

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

2023-07-04 Thread Chesnay Schepler
I think the cooldown still makes sense with FLIP-291 APIs. If you want to fully control the parallelism and rescale timings then you can set the cooldown to zero. If you don't want complete control but just the target parallelism from time to time, then the cooldown within Flink still makes

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

2023-07-04 Thread Benchao Li
Congratulations! Feng Jin 于2023年7月4日周二 16:17写道: > Congratulations! > > Best, > Feng Jin > > > > On Tue, Jul 4, 2023 at 4:13 PM Yuxin Tan wrote: > > > Congratulations! > > > > Best, > > Yuxin > > > > > > Dunn Bangui 于2023年7月4日周二 16:04写道: > > > > > Congratulations! > > > > > > Best, > > >

[jira] [Created] (FLINK-32531) "Recovery of missing job deployments" feature not working as described

2023-07-04 Thread Ivan Stoiev (Jira)
Ivan Stoiev created FLINK-32531: --- Summary: "Recovery of missing job deployments" feature not working as described Key: FLINK-32531 URL: https://issues.apache.org/jira/browse/FLINK-32531 Project: Flink

[jira] [Created] (FLINK-32530) array_position semantic should align with array_contains instead of spark

2023-07-04 Thread Jacky Lau (Jira)
Jacky Lau created FLINK-32530: - Summary: array_position semantic should align with array_contains instead of spark Key: FLINK-32530 URL: https://issues.apache.org/jira/browse/FLINK-32530 Project: Flink

Re: [VOTE] FLIP-322 Cooldown period for adaptive scheduler

2023-07-04 Thread Etienne Chauchot
Hi all, @David, I just saw your comments on the [DISCUSSION] thread. I'll close this vote for now and open a new voting thread when the discussion reaches a consensus. Best Etienne Le 04/07/2023 à 10:49, Etienne Chauchot a écrit : Hi David, Indeed, this thread was a proper separate

[jira] [Created] (FLINK-32529) Optional startup probe for JM deployment

2023-07-04 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-32529: -- Summary: Optional startup probe for JM deployment Key: FLINK-32529 URL: https://issues.apache.org/jira/browse/FLINK-32529 Project: Flink Issue Type: New Feature

Re: [VOTE] FLIP-322 Cooldown period for adaptive scheduler

2023-07-04 Thread Etienne Chauchot
Hi David, Indeed, this thread was a proper separate [VOTE] thread  You withdrew your -1 vote but did not cast a new vote. Can you do so ? Thanks Etienne Le 04/07/2023 à 08:52, David Morávek a écrit : Hmm, sorry for the confusion; it seems that Google is playing games with me (I see this

[jira] [Created] (FLINK-32528) The RexCall a = a,if a's datatype is nullable, and when a is null, a = a is null, it isn't true in BinaryComparisonExprReducer

2023-07-04 Thread LakeShen (Jira)
LakeShen created FLINK-32528: Summary: The RexCall a = a,if a's datatype is nullable, and when a is null, a = a is null, it isn't true in BinaryComparisonExprReducer Key: FLINK-32528 URL:

[jira] [Created] (FLINK-32527) Build failure on Windows

2023-07-04 Thread Fangbin Sun (Jira)
Fangbin Sun created FLINK-32527: --- Summary: Build failure on Windows Key: FLINK-32527 URL: https://issues.apache.org/jira/browse/FLINK-32527 Project: Flink Issue Type: Bug Components:

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

2023-07-04 Thread Feng Jin
Congratulations! Best, Feng Jin On Tue, Jul 4, 2023 at 4:13 PM Yuxin Tan wrote: > Congratulations! > > Best, > Yuxin > > > Dunn Bangui 于2023年7月4日周二 16:04写道: > > > Congratulations! > > > > Best, > > Bangui Dunn > > > > Yangze Guo 于2023年7月4日周二 15:59写道: > > > > > Congrats everyone! > > > > >

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

2023-07-04 Thread Yuxin Tan
Congratulations! Best, Yuxin Dunn Bangui 于2023年7月4日周二 16:04写道: > Congratulations! > > Best, > Bangui Dunn > > Yangze Guo 于2023年7月4日周二 15:59写道: > > > Congrats everyone! > > > > Best, > > Yangze Guo > > > > On Tue, Jul 4, 2023 at 3:53 PM Rui Fan <1996fan...@gmail.com> wrote: > > > > > >

[jira] [Created] (FLINK-32526) Update apache parquet to 1.31.1

2023-07-04 Thread Sergey Nuyanzin (Jira)
Sergey Nuyanzin created FLINK-32526: --- Summary: Update apache parquet to 1.31.1 Key: FLINK-32526 URL: https://issues.apache.org/jira/browse/FLINK-32526 Project: Flink Issue Type: Technical

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

2023-07-04 Thread Dunn Bangui
Congratulations! Best, Bangui Dunn Yangze Guo 于2023年7月4日周二 15:59写道: > Congrats everyone! > > Best, > Yangze Guo > > On Tue, Jul 4, 2023 at 3:53 PM Rui Fan <1996fan...@gmail.com> wrote: > > > > Congratulations! > > > > Best, > > Rui Fan > > > > On Tue, Jul 4, 2023 at 2:08 PM Zhu Zhu wrote: > >

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

2023-07-04 Thread Yangze Guo
Congrats everyone! Best, Yangze Guo On Tue, Jul 4, 2023 at 3:53 PM Rui Fan <1996fan...@gmail.com> wrote: > > Congratulations! > > Best, > Rui Fan > > On Tue, Jul 4, 2023 at 2:08 PM Zhu Zhu wrote: > > > Congratulations everyone! > > > > Thanks, > > Zhu > > > > Hang Ruan 于2023年7月4日周二 14:06写道: >

Re: Working improving the REST API

2023-07-04 Thread David Morávek
I've left some comments on the PR. On Tue, Jul 4, 2023 at 9:20 AM Martijn Visser wrote: > Hi Hong, > > Given that this changes the REST API, which is a public interface, I'm > wondering if this shouldn't first have had a (small) FLIP if I follow the > guidelines from the overview page [1]. > >

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

2023-07-04 Thread Rui Fan
Congratulations! Best, Rui Fan On Tue, Jul 4, 2023 at 2:08 PM Zhu Zhu wrote: > Congratulations everyone! > > Thanks, > Zhu > > Hang Ruan 于2023年7月4日周二 14:06写道: > > > > Congratulations! > > > > Best, > > Hang > > > > Jingsong Li 于2023年7月4日周二 13:47写道: > > > > > Congratulations! > > > > > >

[jira] [Created] (FLINK-32525) Update commons-beanutils to 1.9.4

2023-07-04 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-32525: -- Summary: Update commons-beanutils to 1.9.4 Key: FLINK-32525 URL: https://issues.apache.org/jira/browse/FLINK-32525 Project: Flink Issue Type: Technical

Re: Working improving the REST API

2023-07-04 Thread Martijn Visser
Hi Hong, Given that this changes the REST API, which is a public interface, I'm wondering if this shouldn't first have had a (small) FLIP if I follow the guidelines from the overview page [1]. Best regards, Martijn [1]

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

2023-07-04 Thread David Morávek
> They will struggle if they add new resources and nothing happens for 5 minutes. The same applies if they start playing with FLIP-291 APIs. I'm wondering if the cooldown makes sense there since it was the user's deliberate choice to push new requirements. 樂 Best, D. On Tue, Jul 4, 2023 at 9:11 

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

2023-07-04 Thread David Morávek
The FLIP reads sane to me. I'm unsure about the default values, though; 5 minutes of wait time between rescales feels rather strict, and we should rethink it to provide a better out-of-the-box experience. I'd focus on newcomers trying AS / Reactive Mode out. They will struggle if they add new

Re: [VOTE] FLIP-322 Cooldown period for adaptive scheduler

2023-07-04 Thread David Morávek
Hmm, sorry for the confusion; it seems that Google is playing games with me (I see this chained under the old [DISCUSS] thread), but it seems correct in the mail archive [1] :/ Just ignore the -1 above. [1] https://lists.apache.org/thread/22fovrkmzcvzblcohhtsp5t96vd64obj On Tue, Jul 4, 2023 at

Re: [VOTE] FLIP-322 Cooldown period for adaptive scheduler

2023-07-04 Thread David Morávek
> > The vote closes within 6 hours and, as for now, there was no vote. This > is a very short FLIP, that takes a few minutes to read. > Maybe because there should have been a dedicated voting thread (marked as [VOTE]), this one was hidden and hard to notice. We should restart the vote with

[jira] [Created] (FLINK-32524) Improve the watermark aggregation performance when enabling the watermark alignment

2023-07-04 Thread Rui Fan (Jira)
Rui Fan created FLINK-32524: --- Summary: Improve the watermark aggregation performance when enabling the watermark alignment Key: FLINK-32524 URL: https://issues.apache.org/jira/browse/FLINK-32524 Project:

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

2023-07-04 Thread Zhu Zhu
Congratulations everyone! Thanks, Zhu Hang Ruan 于2023年7月4日周二 14:06写道: > > Congratulations! > > Best, > Hang > > Jingsong Li 于2023年7月4日周二 13:47写道: > > > Congratulations! > > > > Thank you! All of the Flink community! > > > > Best, > > Jingsong > > > > On Tue, Jul 4, 2023 at 1:24 PM tison

Re: Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

2023-07-04 Thread Hang Ruan
Congratulations! Best, Hang Jingsong Li 于2023年7月4日周二 13:47写道: > Congratulations! > > Thank you! All of the Flink community! > > Best, > Jingsong > > On Tue, Jul 4, 2023 at 1:24 PM tison wrote: > > > > Congrats and with honor :D > > > > Best, > > tison. > > > > > > Mang Zhang 于2023年7月4日周二