Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

2023-07-05 Thread Weihua Hu
Congratulations! Best, Weihua On Wed, Jul 5, 2023 at 5:46 PM Shammon FY wrote: > Congratulations! > > Best, > Shammon FY > > On Wed, Jul 5, 2023 at 2:38 PM Paul Lam wrote: > > > Congrats and cheers! > > > > Best, > > Paul Lam > > > > > 2023年7月4日 18:04,Benchao Li 写道: > > > > > >

[jira] [Created] (FLINK-32547) Add missing doc for Timestamp support in ProtoBuf format

2023-07-05 Thread Benchao Li (Jira)
Benchao Li created FLINK-32547: -- Summary: Add missing doc for Timestamp support in ProtoBuf format Key: FLINK-32547 URL: https://issues.apache.org/jira/browse/FLINK-32547 Project: Flink Issue

Re: Feature requests for Flink protobuf deserialization

2023-07-05 Thread Benchao Li
Thanks for starting the discussion, 1. I'm +1 for this. 2. We have already supported this in [1] 3. I'm not sure about this, could you give more examples except the cases 1&2? 4&5. I think we also have considered this with the option 'protobuf.read-default-values' [2], is this what you want? [1]

Re: [DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems

2023-07-05 Thread Dian Fu
Hi Chesnay, >> The wrapping of connectors is a bit of a maintenance nightmare and doesn't really work with external/custom connectors. Cannot agree with you more. >> Has there ever been thoughts about changing flink-pythons connector setup to use the table api connectors underneath? I'm still

Re: [DISCUSS] FLIP-325: Support configuring end-to-end allowed latency

2023-07-05 Thread Shammon FY
Hi, Thanks for your replay @Dong. I really agree with Piotr's points and I would like to share some thoughts from my side. About the latency for mini-batch mechanism in Flink SQL, I still think the description in the FLIP is not right. If there are N operators and the whole process time for data

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

2023-07-05 Thread Dong Lin
Hi Piotr, I am sorry if you feel unhappy or upset with us for not following/fixing your proposal. It is not my intention to give you this feeling. After all, we are all trying to make Flink better, to support more use-case with the most maintainable code. I hope you understand that just like you,

Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

2023-07-05 Thread Jing Ge
Hi Xuannan, Hi Dong, Thanks for your clarification. @Xuannan A Jira ticket has been created for the doc update: https://issues.apache.org/jira/browse/FLINK-32546 @Dong I don't have a concrete example. I just thought about it from a conceptual or pattern's perspective. Since we have 1.

Re: [DISCUSS] FLIP-325: Support configuring end-to-end allowed latency

2023-07-05 Thread Piotr Nowojski
Hi, Thanks for this proposal, this is a very much needed thing that should be addressed in Flink. I think there is one thing that hasn't been discussed neither here nor in FLIP-309. Given that we have three dimensions: - e2e latency/checkpointing interval - enabling some kind of

[jira] [Created] (FLINK-32546) update Code Style Guide with Java properties naming convention

2023-07-05 Thread Jing Ge (Jira)
Jing Ge created FLINK-32546: --- Summary: update Code Style Guide with Java properties naming convention Key: FLINK-32546 URL: https://issues.apache.org/jira/browse/FLINK-32546 Project: Flink Issue

Re: [DISUCSS] Deprecate multiple APIs in 1.18

2023-07-05 Thread Jing Ge
Hi Alex, > > 3. remove SinkFunction. > Which steps do you imply for the 1.18 release and for the 2.0 release? > for 2.0 release. 1.18 will be released soon. Best regards, Jing On Wed, Jul 5, 2023 at 1:08 PM Alexander Fedulov < alexander.fedu...@gmail.com> wrote: > @Jing > Just to clarify,

Re: [DISCUSS] FLIP-327: Support stream-batch unified operator to improve job throughput when processing backlog data

2023-07-05 Thread Piotr Nowojski
Hi Dong, I have a couple of questions. Could you explain why those properties @Nullable private Boolean isOutputOnEOF = null; @Nullable private Boolean isOutputOnCheckpoint = null; @Nullable private Boolean isInternalSorterSupported = null; must be `@Nullable`, instead of having

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

2023-07-05 Thread Piotr Nowojski
Hi Guys, I would like to ask you again, to spend a bit more effort on trying to find solutions, not just pointing out problems. For 1.5 months, the discussion doesn't go in circle, but I'm suggesting a solution, you are trying to undermine it with some arguments, I'm coming back with a fix, often

[jira] [Created] (FLINK-32545) Removes the expensive Row operations like join

2023-07-05 Thread Jiang Xin (Jira)
Jiang Xin created FLINK-32545: - Summary: Removes the expensive Row operations like join Key: FLINK-32545 URL: https://issues.apache.org/jira/browse/FLINK-32545 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-322 Cooldown period for adaptive scheduler

2023-07-05 Thread Etienne Chauchot
Hi all, Thanks David for your suggestions. Comments inline. Le 04/07/2023 à 13:35, David Morávek a écrit : waiting 2 min between 2 requirements push seems ok to me This depends on the workload. Would you care if the cost of rescaling were close to zero (which is for most out-of-the-box

Re: [DISUCSS] Deprecate multiple APIs in 1.18

2023-07-05 Thread Xintong Song
@Chesnay, shouldn't the decision to deprecate an API be part of the FLIP discussion? > Exactly. I agree that deprecation of an old API should be part of the FLIP where the new API is introduced. And I appreciate that many APIs that are listed to be removed in release 2.0 are already deprecated

Re: [DISUCSS] Deprecate multiple APIs in 1.18

2023-07-05 Thread Alexander Fedulov
@Jing Just to clarify, when you say: > 3. remove SinkFunction. Which steps do you imply for the 1.18 release and for the 2.0 release? @Xintong A side note - with the new Source API we lose the ability to control checkpointing from the source since there is no lock anymore. This functionality is

Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

2023-07-05 Thread Dong Lin
Hi Jing, Thanks for the comments! Please find below my comments, which are based on the offline discussion with Xuannan. On Wed, Jul 5, 2023 at 1:36 AM Jing Ge wrote: > Hi Xuannan, Hi Dong > > Thanks for the Proposal! After reading the FLIP, I'd like to ask some > questions: > > 1. Naming

Re: [DISCUSS] Flink and Externalized connectors leads to block and circular dependency problems

2023-07-05 Thread Chesnay Schepler
Has there ever been thoughts about changing flink-pythons connector setup to use the table api connectors underneath? The wrapping of connectors is a bit of a maintenance nightmare and doesn't really work with external/custom connectors. On 04/07/2023 13:35, Dian Fu wrote: Thanks Ran Tao

[jira] [Created] (FLINK-32544) PythonFunctionFactoryTest fails on Java 17

2023-07-05 Thread Chesnay Schepler (Jira)
Chesnay Schepler created FLINK-32544: Summary: PythonFunctionFactoryTest fails on Java 17 Key: FLINK-32544 URL: https://issues.apache.org/jira/browse/FLINK-32544 Project: Flink Issue

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

2023-07-05 Thread Shammon FY
Congratulations! Best, Shammon FY On Wed, Jul 5, 2023 at 2:38 PM Paul Lam wrote: > Congrats and cheers! > > Best, > Paul Lam > > > 2023年7月4日 18:04,Benchao Li 写道: > > > > Congratulations! > > > > Feng Jin 于2023年7月4日周二 16:17写道: > > > >> Congratulations! > >> > >> Best, > >> Feng Jin > >> > >>

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-07-05 Thread Shammon FY
Hi devs, Thanks for all the feedback. I have discussed with @QingSheng Ren off-line to confirm some questionable points in the FLIP. Thanks for his valuable inputs and I have updated the FLIP according to our discussion. Looking forward to your feedback, thanks, Best, Shammon FY On Wed, Jul

Re: [DISCUSS] FLIP-314: Support Customized Job Lineage Listener

2023-07-05 Thread Shammon FY
Hi Jing, Thanks for your feedback. > 1. TableColumnLineageRelation#sinkColumn() should return TableColumnLineageEntity instead of String, right? The `sinkColumn()` will return `String` which is the column name in the sink connector. I found the name of `TableColumnLineageEntity` may cause

[jira] [Created] (FLINK-32543) The actual behavior of restart-strategy.xxx.delay is inconsistent with the document

2023-07-05 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-32543: -- Summary: The actual behavior of restart-strategy.xxx.delay is inconsistent with the document Key: FLINK-32543 URL: https://issues.apache.org/jira/browse/FLINK-32543

[jira] [Created] (FLINK-32542) 'connector'='starrocks' 如何手动指定starrocks 中__op 隐藏字段,用来实现物理删除

2023-07-05 Thread LaraJiang (Jira)
LaraJiang created FLINK-32542: - Summary: 'connector'='starrocks' 如何手动指定starrocks 中__op 隐藏字段,用来实现物理删除 Key: FLINK-32542 URL: https://issues.apache.org/jira/browse/FLINK-32542 Project: Flink Issue

Re: [DISUCSS] Deprecate multiple APIs in 1.18

2023-07-05 Thread Chesnay Schepler
There's a whole bunch of metric APIs that would need to be deprecated. That is of course if the metric FLIPs are being accepted. Which makes me wonder if we aren't doing things the wrong way around; shouldn't the decision to deprecate an API be part of the FLIP discussion? On 05/07/2023

[jira] [Created] (FLINK-32541) Fix the buffer leaking in buffer accumulators when a failover occurs

2023-07-05 Thread Yuxin Tan (Jira)
Yuxin Tan created FLINK-32541: - Summary: Fix the buffer leaking in buffer accumulators when a failover occurs Key: FLINK-32541 URL: https://issues.apache.org/jira/browse/FLINK-32541 Project: Flink

Re: [DISCUSS] FLIP-329: Add operator attribute to specify support for object-reuse

2023-07-05 Thread Xuannan Su
Hi Jing Ge, Thank you for your valuable comments! 1. I agree with your suggestion regarding following the JavaBean convention. It would be beneficial to incorporate this convention into our Code Style Guide [1]. By doing so, we can ensure consistency and make it easier for developers to adhere

[jira] [Created] (FLINK-32540) The issue of not distributing the last batch of data

2023-07-05 Thread Jira
原来你是小幸运001 created FLINK-32540: -- Summary: The issue of not distributing the last batch of data Key: FLINK-32540 URL: https://issues.apache.org/jira/browse/FLINK-32540 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-309: Enable operators to trigger checkpoints dynamically

2023-07-05 Thread Hang Ruan
Hi, Piotr & Dong. Thanks for the discussion. IMO, I do not think the provided counter proposal is a good idea. There are some concerns from my side. 1. It is hard to find the error checkpoint. If there are other errors causing the checkpoint failure, we have to check every failed checkpoint to

Re: [ANNOUNCE] Apache Flink has won the 2023 SIGMOD Systems Award

2023-07-05 Thread Paul Lam
Congrats and cheers! Best, Paul Lam > 2023年7月4日 18:04,Benchao Li 写道: > > Congratulations! > > Feng Jin 于2023年7月4日周二 16:17写道: > >> Congratulations! >> >> Best, >> Feng Jin >> >> >> >> On Tue, Jul 4, 2023 at 4:13 PM Yuxin Tan wrote: >> >>> Congratulations! >>> >>> Best, >>> Yuxin >>>