Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-06-07 Thread Shammon FY
Hi, To @Jing Ge > Thanks for the clarification. Just out of curiosity, if the context is not part of the event, why should it be the input parameter of each onEvent call? I think it's quite strange to put some information in an Event, such as a factory identifier for catalog, but they will be

Re: [VOTE] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread Aitozi
+1 Looking forward to this feature. Best, Aitozi. Jing Ge 于2023年6月8日周四 04:44写道: > +1 > > Best Regards, > Jing > > On Wed, Jun 7, 2023 at 10:52 AM weijie guo > wrote: > > > +1 (binding) > > > > Best regards, > > > > Weijie > > > > > > Jingsong Li 于2023年6月7日周三 15:59写道: > > > > > +1 > > > > >

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread Aitozi
Hi ron, Thanks you for your explanation, I have no other questions. Best, Atiozi. liu ron 于2023年6月8日周四 10:52写道: > Hi, Atiozi > > Thanks for your feedback. > > > Traverse the ExecNode DAG and create a FusionExecNode for physical > operators that can be fused together. > which kind of

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-06-07 Thread Aitozi
Hi Mason, Thanks for your input. I think if we support the user defined async table function, user will be able to use it to hold a batch data then handle it at one time in the customized function. AsyncSink is meant for the sink operator. I have not figure out how to integrate in this case.

[jira] [Created] (FLINK-32285) Support multiple Flink versions in each Flink ML Java library release

2023-06-07 Thread Jiang Xin (Jira)
Jiang Xin created FLINK-32285: - Summary: Support multiple Flink versions in each Flink ML Java library release Key: FLINK-32285 URL: https://issues.apache.org/jira/browse/FLINK-32285 Project: Flink

Re: [DISCUSS] FLIP-313 Add support of User Defined AsyncTableFunction

2023-06-07 Thread Mason Chen
Hi Aitozi, I think it makes sense to make it easier for SQL users to make RPCs. Do you think your proposal can extend to the ability to batch data for the RPC? This is also another common strategy to increase throughput. Also, have you considered solving this a bit differently by leveraging

Re: [DISCUSS] FLIP-294: Support Customized Job Meta Data Listener

2023-06-07 Thread Mason Chen
Hi Shammon, FLIP generally looks good and I'm excited to see this feature. 1. I'm also curious about default implementations. Would atlas/datahub be supported by default? 2. The FLIP title is confusing to me, especially in distinguishing it from FLIP-314. Would a better FLIP title be "Support

Re: [DISCUSS] FLIP-316: Introduce SQL Driver

2023-06-07 Thread Mason Chen
Hi Paul, Thanks for your response! I agree that utilizing SQL Drivers in Java applications is equally important > as employing them in SQL Gateway. WRT init containers, I think most > users use them just as a workaround. For example, wget a jar from the > maven repo. > > We could implement the

Re: [DISCUSS] Hive dialect shouldn't fall back to Flink's default dialect

2023-06-07 Thread yuxia
Thanks all for the discussion. Since we reach the consensus about abandoning the fall back behavior for Hive dialect, if there's no other concern or objection, I'll close this discussion tomorrow(6/9). Best regards, Yuxia - 原始邮件 - 发件人: "Martijn Visser" 收件人: "dev" 发送时间: 星期一, 2023年 6

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2023-06-07 Thread Mason Chen
Hi Gordon, Thanks for taking a look! Regarding (1), there is a need from the readers to send this event at startup because the reader state may reflect outdated metadata. Thus, the reader should not start without fresh metadata. With fresh metadata, the reader can filter splits from state--this

Re: [DISCUSS] FLIP-311: Support Call Stored Procedure

2023-06-07 Thread yuxia
Hi, all. Thanks everyone for the valuable input. If there are are no further concerns about this FLIP[1], I would like to start voting next monday (6/12). [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-311%3A+Support+Call+Stored+Procedure Best regards, Yuxia - 原始邮件 - 发件人:

Re: Re: Re: [DISCUSS] FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement

2023-06-07 Thread Jingsong Li
Thanks Mang for updating! Looks good to me! Best, Jingsong On Wed, Jun 7, 2023 at 2:31 PM Mang Zhang wrote: > > Hi Jingsong, > > >I have some doubts about the `TwoPhaseCatalogTable`. Generally, our > >Flink design places execution in the TableFactory or directly in the > >Catalog, so

Re: [DISCUSS] FLIP-308: Support Time Travel In Batch Mode

2023-06-07 Thread Feng Jin
Hi all, thanks for your input @Benchao > The type for "TIMESTAMP '2023-04-27 00:00:00'" should be "TIMESTAMP WITHOUT TIME ZONE", converting it to unix timestamp would use UTC timezone, which is not usually expected by users. It was indeed the case before Flink 1.13, but now my understanding

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread liu ron
Hi, Ging Thanks for your valuable input about scala free. Firstly, reply to your question, using java to implement codegen is possible, but we need to utilize some tools. I think the first alternative is to update our jdk version to 13, which provides text block feature[1] makes string format

[jira] [Created] (FLINK-32284) When using session windows, the state of slot 1 keeps growing

2023-06-07 Thread linweijiang (Jira)
linweijiang created FLINK-32284: --- Summary: When using session windows, the state of slot 1 keeps growing Key: FLINK-32284 URL: https://issues.apache.org/jira/browse/FLINK-32284 Project: Flink

[jira] [Created] (FLINK-32283) Implement SortBufferAccumulator

2023-06-07 Thread Yuxin Tan (Jira)
Yuxin Tan created FLINK-32283: - Summary: Implement SortBufferAccumulator Key: FLINK-32283 URL: https://issues.apache.org/jira/browse/FLINK-32283 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread liu ron
Hi, Atiozi Thanks for your feedback. > Traverse the ExecNode DAG and create a FusionExecNode for physical operators that can be fused together. which kind of operators can be fused together ? are the operators in an operator chain? Is this optimization aligned to spark's whole stage codegen ?

Re: [VOTE] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread Jing Ge
+1 Best Regards, Jing On Wed, Jun 7, 2023 at 10:52 AM weijie guo wrote: > +1 (binding) > > Best regards, > > Weijie > > > Jingsong Li 于2023年6月7日周三 15:59写道: > > > +1 > > > > On Wed, Jun 7, 2023 at 3:03 PM Benchao Li wrote: > > > > > > +1, binding > > > > > > Jark Wu 于2023年6月7日周三 14:44写道: > >

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread Jing Ge
Hi Ron, Thanks for raising the proposal. It is a very attractive idea! Since the FLIP is a relatively complex one which contains three papers and a design doc. It deserves more time for the discussion to make sure everyone is on the same page. I have a NIT question which will not block your

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2023-06-07 Thread Tzu-Li (Gordon) Tai
Hi Mason, Thanks for updating the FLIP. In principle, I believe this would be a useful addition. Some comments so far: 1. In this sequence diagram [1], why is there a need for a GetMetadataUpdateEvent from the MultiClusterSourceReader going to the MultiClusterSourceEnumerator? Shouldn't the

Re: [ANNOUNCE] Apache flink-connector-pulsar v3.0.1 released

2023-06-07 Thread Neng Lu
Thank you very much for coordinating this! I think we also need to release 4.0.1 to fix the pulsar-client OAuth2 issue. On Wed, Jun 7, 2023 at 1:48 AM Leonard Xu wrote: > The Apache Flink community is very happy to announce the release of Apache > flink-connector-pulsar v3.0.1. > This release

Re: [DISCUSS] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread Aitozi
Hi Ron: Sorry for the late reply after the voting process. I just want to ask > Traverse the ExecNode DAG and create a FusionExecNode for physical operators that can be fused together. which kind of operators can be fused together ? are the operators in an operator chain? Is this

[jira] [Created] (FLINK-32282) Use kubernetes-webhooks-framework instead of deprecated admission control

2023-06-07 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-32282: -- Summary: Use kubernetes-webhooks-framework instead of deprecated admission control Key: FLINK-32282 URL: https://issues.apache.org/jira/browse/FLINK-32282 Project: Flink

[jira] [Created] (FLINK-32281) Enable two-phase agg default when agg function support adaptive local hashagg

2023-06-07 Thread dalongliu (Jira)
dalongliu created FLINK-32281: - Summary: Enable two-phase agg default when agg function support adaptive local hashagg Key: FLINK-32281 URL: https://issues.apache.org/jira/browse/FLINK-32281 Project:

[jira] [Created] (FLINK-32280) HashAgg support operator fusion codegen

2023-06-07 Thread dalongliu (Jira)
dalongliu created FLINK-32280: - Summary: HashAgg support operator fusion codegen Key: FLINK-32280 URL: https://issues.apache.org/jira/browse/FLINK-32280 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-32279) Shuffle HashJoin support spill to disk when enable operator fusion codegen

2023-06-07 Thread dalongliu (Jira)
dalongliu created FLINK-32279: - Summary: Shuffle HashJoin support spill to disk when enable operator fusion codegen Key: FLINK-32279 URL: https://issues.apache.org/jira/browse/FLINK-32279 Project: Flink

[jira] [Created] (FLINK-32278) HashJoin support operator fusion codegen

2023-06-07 Thread dalongliu (Jira)
dalongliu created FLINK-32278: - Summary: HashJoin support operator fusion codegen Key: FLINK-32278 URL: https://issues.apache.org/jira/browse/FLINK-32278 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-32277) Introduce operator fusion codegen framework

2023-06-07 Thread dalongliu (Jira)
dalongliu created FLINK-32277: - Summary: Introduce operator fusion codegen framework Key: FLINK-32277 URL: https://issues.apache.org/jira/browse/FLINK-32277 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] FLIP-246: Multi Cluster Kafka Source

2023-06-07 Thread Mason Chen
Hi Jing, Thanks for the prompt feedback! I had some confusion with how to resize images in confluence--anyways, I have made the font bigger, added white background, and also made the diagrams themselves bigger. Regarding the exactly once semantics, that's definitely good to point out in the doc.

[jira] [Created] (FLINK-32276) After adding the where condition to the flink lookup left join, the joinType becomes innerJoin

2023-06-07 Thread Xianxun Ye (Jira)
Xianxun Ye created FLINK-32276: -- Summary: After adding the where condition to the flink lookup left join, the joinType becomes innerJoin Key: FLINK-32276 URL: https://issues.apache.org/jira/browse/FLINK-32276

Re: [DISCUSS] FLIP-316: Introduce SQL Driver

2023-06-07 Thread Shengkai Fang
Hi. Paul. Thanks for your update and the update makes me understand the design much better. But I still have some questions about the FLIP. > For SQL Gateway, only DMLs need to be delegated to the SQL server > Driver. I would think about the details and update the FLIP. Do you have some > ideas

[jira] [Created] (FLINK-32275) The return value of PartitionableListState#get is inconsistent with the requirements of AppendingState

2023-06-07 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-32275: -- Summary: The return value of PartitionableListState#get is inconsistent with the requirements of AppendingState Key: FLINK-32275 URL:

Re: [VOTE] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread weijie guo
+1 (binding) Best regards, Weijie Jingsong Li 于2023年6月7日周三 15:59写道: > +1 > > On Wed, Jun 7, 2023 at 3:03 PM Benchao Li wrote: > > > > +1, binding > > > > Jark Wu 于2023年6月7日周三 14:44写道: > > > > > +1 (binding) > > > > > > Best, > > > Jark > > > > > > > 2023年6月7日 14:20,liu ron 写道: > > > > > >

[ANNOUNCE] Apache flink-connector-pulsar v3.0.1 released

2023-06-07 Thread Leonard Xu
The Apache Flink community is very happy to announce the release of Apache flink-connector-pulsar v3.0.1. This release is compatible with Flink 1.16.x series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data

[jira] [Created] (FLINK-32274) kafka class could not be found while using appliction mode.

2023-06-07 Thread Spongebob (Jira)
Spongebob created FLINK-32274: - Summary: kafka class could not be found while using appliction mode. Key: FLINK-32274 URL: https://issues.apache.org/jira/browse/FLINK-32274 Project: Flink Issue

Re: [VOTE] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread Jingsong Li
+1 On Wed, Jun 7, 2023 at 3:03 PM Benchao Li wrote: > > +1, binding > > Jark Wu 于2023年6月7日周三 14:44写道: > > > +1 (binding) > > > > Best, > > Jark > > > > > 2023年6月7日 14:20,liu ron 写道: > > > > > > Hi everyone, > > > > > > Thanks for all the feedback about FLIP-315: Support Operator Fusion > >

[RESULT][VOTE] Release flink-connector-pulsar 3.0.1, release candidate #1

2023-06-07 Thread Leonard Xu
I'm happy to announce that we have unanimously approved this release. There are 7 approving votes, 4 of which are binding: * Zili Chen (non-binding) * Neng Lu (non-binding) * Jing Ge (non-binding) * Jark Wu (binding) * Jingsong Li (binding) * Martijn Visser (binding) * Leonard Xu (binding) There

Re: [VOTE] Release flink-connector-pulsar 3.0.1, release candidate #1

2023-06-07 Thread Leonard Xu
Thanks all ! This vote is now closed, I'll announce the results in a separate thread. Best, Leonard > On Jun 7, 2023, at 3:17 PM, Leonard Xu wrote: > > > +1 (binding) > > - built from source code succeeded > - verified signatures > - verified hashsums > - checked Github release tag > -

Re: [VOTE] Release flink-connector-pulsar 3.0.1, release candidate #1

2023-06-07 Thread Leonard Xu
+1 (binding) - built from source code succeeded - verified signatures - verified hashsums - checked Github release tag - checked release notes - checked the contents contains jar and pom files in apache repo Best, Leonard > On Jun 7, 2023, at 2:58 PM, Martijn Visser wrote: > > +1

Re: [VOTE] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread Benchao Li
+1, binding Jark Wu 于2023年6月7日周三 14:44写道: > +1 (binding) > > Best, > Jark > > > 2023年6月7日 14:20,liu ron 写道: > > > > Hi everyone, > > > > Thanks for all the feedback about FLIP-315: Support Operator Fusion > Codegen > > for Flink SQL[1]. > > [2] is the discussion thread. > > > > I'd like to

Re: [VOTE] Release flink-connector-pulsar 3.0.1, release candidate #1

2023-06-07 Thread Martijn Visser
+1 (binding) - Validated hashes - Verified signature - Verified that no binaries exist in the source archive - Build the source with Maven - Verified licenses - Verified web PRs On Wed, Jun 7, 2023 at 4:34 AM Jingsong Li wrote: > +1 (binding) > > - checked NOTICE and LICENSE > - Verified

Re: [VOTE] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread Jark Wu
+1 (binding) Best, Jark > 2023年6月7日 14:20,liu ron 写道: > > Hi everyone, > > Thanks for all the feedback about FLIP-315: Support Operator Fusion Codegen > for Flink SQL[1]. > [2] is the discussion thread. > > I'd like to start a vote for it. The vote will be open for at least 72 > hours (until

Re:Re: Re: [DISCUSS] FLIP-305: Support atomic for CREATE TABLE AS SELECT(CTAS) statement

2023-06-07 Thread Mang Zhang
Hi Jingsong, >I have some doubts about the `TwoPhaseCatalogTable`. Generally, our >Flink design places execution in the TableFactory or directly in the >Catalog, so introducing an executable table makes me feel a bit >strange. (Spark is this style, but Flink may not be) On this issue, we

[VOTE] FLIP-315: Support Operator Fusion Codegen for Flink SQL

2023-06-07 Thread liu ron
Hi everyone, Thanks for all the feedback about FLIP-315: Support Operator Fusion Codegen for Flink SQL[1]. [2] is the discussion thread. I'd like to start a vote for it. The vote will be open for at least 72 hours (until June 12th, 12:00AM GMT) unless there is an objection or an insufficient