Re: [VOTE] FLIP-297: Improve Auxiliary Sql Statements

2023-03-07 Thread yuxia
+1 (non-binding) Thanks Ran Tao for driving it. Best regards, Yuxia - 原始邮件 - 发件人: "Jark Wu" 收件人: "dev" 发送时间: 星期二, 2023年 3 月 07日 下午 8:50:05 主题: Re: [VOTE] FLIP-297: Improve Auxiliary Sql Statements +1 (binding) Best, Jark > 2023年3月7日 17:07,Jing Ge 写道: > > +1 > Thanks! > > Best

Re: StreamingFileSink's DefaultRollingPolicy

2023-03-07 Thread yuxia
Yes, it's by design. I don't think we should change DefaultRollingPolicy. Can't `OnCheckpointRollingPolicy` meet your requirements? Best regards, Yuxia - 原始邮件 - 发件人: "Lee, Keith" 收件人: "dev" 发送时间: 星期二, 2023年 3 月 07日 下午 10:39:44 主题: StreamingFileSink's DefaultRollingPolicy Hi,

[jira] [Created] (FLINK-31365) Simplify FlinkActionsE2ETest#testMergeInto

2023-03-07 Thread yuzelin (Jira)
yuzelin created FLINK-31365: --- Summary: Simplify FlinkActionsE2ETest#testMergeInto Key: FLINK-31365 URL: https://issues.apache.org/jira/browse/FLINK-31365 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager

2023-03-07 Thread Weihua Hu
Thanks Yangze for your attention, this would be a great help. And thanks Matthias too. FLIP-156 [1] mentions some incompatibility between fine-grained resource > management and reactive mode. I assume that this is independent of the > SlotManager and replacing the DSM with the FGSM wouldn't

[jira] [Created] (FLINK-31364) [Flink] add metrics for TableStore

2023-03-07 Thread Ming Li (Jira)
Ming Li created FLINK-31364: --- Summary: [Flink] add metrics for TableStore Key: FLINK-31364 URL: https://issues.apache.org/jira/browse/FLINK-31364 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-31363) KafkaSink failed to commit transactions under EXACTLY_ONCE semantics

2023-03-07 Thread lightzhao (Jira)
lightzhao created FLINK-31363: - Summary: KafkaSink failed to commit transactions under EXACTLY_ONCE semantics Key: FLINK-31363 URL: https://issues.apache.org/jira/browse/FLINK-31363 Project: Flink

[jira] [Created] (FLINK-31362) Add document about how to use C-style escape strings

2023-03-07 Thread Aitozi (Jira)
Aitozi created FLINK-31362: -- Summary: Add document about how to use C-style escape strings Key: FLINK-31362 URL: https://issues.apache.org/jira/browse/FLINK-31362 Project: Flink Issue Type:

[jira] [Created] (FLINK-31361) job created by sql-client can't authenticate to kafka, can't find org.apache.kafka.common.security.plain.PlainLoginModule

2023-03-07 Thread David Anderson (Jira)
David Anderson created FLINK-31361: -- Summary: job created by sql-client can't authenticate to kafka, can't find org.apache.kafka.common.security.plain.PlainLoginModule Key: FLINK-31361 URL:

filesystem connector InputSplit support for CSV format

2023-03-07 Thread Yasin Celik
*Hi everyone, * *Can anyone please confirm that multi readers are not supported when reading from CSV source files using filesystem connector. I have tried several methodologies but Job Manager always creates a single reader which becomes a bottleneck.Issue summary:Flink provides

Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

2023-03-07 Thread Jing Ge
Hi Lincoln, This FLIP is already in a good shape. +1. Thanks for driving it. I just have a very tiny comment, e.g. NIT. The "wide table" you mentioned in the FLIP should be "denormalized table"[1] and fact tables in data warehouse are commonly denormalized tables[2]. As far as I am concerned,

[jira] [Created] (FLINK-31360) Running HsResultPartitionTest repeatedly causes error with 137 exit code indicating a memory leak

2023-03-07 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-31360: - Summary: Running HsResultPartitionTest repeatedly causes error with 137 exit code indicating a memory leak Key: FLINK-31360 URL:

[jira] [Created] (FLINK-31359) HsResultPartitionTest fails with fatal error

2023-03-07 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-31359: - Summary: HsResultPartitionTest fails with fatal error Key: FLINK-31359 URL: https://issues.apache.org/jira/browse/FLINK-31359 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-298: Unifying the Implementation of SlotManager

2023-03-07 Thread Matthias Pohl
Thanks for updating the FLIP and adding more context to it. Additionally, thanks to Xintong and Yangze for offering your expertise here as contributors to the initial FineGrainedSlotManager implementation. The remark on cutting out functionality was only based on some superficial initial code

Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

2023-03-07 Thread Junrui Lee
Hi Yuxin, This FLIP looks quite reasonable. Flink can solve the problem of Batch shuffle by combining local and remote storage, and can use fixed local disks for better performance in most scenarios, while using remote storage as a supplement when local disks are not sufficient, avoiding

StreamingFileSink's DefaultRollingPolicy

2023-03-07 Thread Lee, Keith
Hi, StreamingFileSink’s DefaultRollingPolicy does not always roll in progress parts to pending parts on checkpoints. Is this by design? It seems counter intuitive to me that part files are not finished on checkpoint by default.

Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

2023-03-07 Thread Lincoln Lee
Hi all, Both the FLIP[1] and the poc[2] has been updated, using Optional to replace the return value of `getTargetColumns` [1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081 [2] https://github.com/apache/flink/pull/22041 Best, Lincoln Lee Jane Chan 于2023年3月7日周二

Re: [VOTE] FLIP-297: Improve Auxiliary Sql Statements

2023-03-07 Thread Jark Wu
+1 (binding) Best, Jark > 2023年3月7日 17:07,Jing Ge 写道: > > +1 > Thanks! > > Best regards, > Jing > > On Tue, Mar 7, 2023 at 9:51 AM ConradJam wrote: > >> +1 (non-binding). >> >> Hang Ruan 于2023年3月7日周二 16:14写道: >> >>> Thanks for Ran's work. >>> +1 (non-binding). >>> >>> Best, >>> Hang

Re: [DISCUSS] String literal behavior in Flink

2023-03-07 Thread Jark Wu
Thank you for sharing this good news! I think we can create an issue for upgrading to Calcite 1.33 and mention the waiting feature. Best, Jark > 2023年3月6日 22:16,Aitozi 写道: > > Hi, Jark > >FYI, this feature has already been supported in Calcite 1.33.0 [1]. > Therefore, I believe we

[SUMMARY] Flink 1.17 Release Sync 3/7/2023

2023-03-07 Thread Leonard Xu
Hi devs and users, I'd like to share some highlights from Flink 1.17 release sync on 3/7/2023. 1.17 Blockers: - Currently, there is one blocker issue (FLINK-31351[1]) that needs to be resolved before we can create a votable RC1. Our contributors are working hard to fix it as soon as

[jira] [Created] (FLINK-31358) Display consumed split/partition/queue info on Web UI

2023-03-07 Thread Zhanghao Chen (Jira)
Zhanghao Chen created FLINK-31358: - Summary: Display consumed split/partition/queue info on Web UI Key: FLINK-31358 URL: https://issues.apache.org/jira/browse/FLINK-31358 Project: Flink

Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL

2023-03-07 Thread Kui Yuan
Hi Jing, Thanks for your advice. In upcoming PR, I will explain all these things (Including flip-182,flip-217,etc.) clearly in the flink doc to make sure the user can understand the behavior behind it. Best, Kui Jing Ge 于2023年3月7日周二 19:42写道: > Hi Kui, > > Thanks for adding it into the Flip.

Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

2023-03-07 Thread Yuxin Tan
Thanks for joining the discussion. @weijie guo > 1. How to optimize the broadcast result partition? For the partitions with multi-consumers, e.g., broadcast result partition, partition reuse, speculative, etc, the processing logic is the same as the original Hybrid Shuffle, that is, using the

Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL

2023-03-07 Thread Jing Ge
Hi Kui, Thanks for adding it into the Flip. There is one more thing wrt this topic you might want to pay attention to, a little bit off-topic, is that Flink SQL users might not be familiar with use cases of low level Datastream API. IMHO, it is highly recommended(mandatory) to write the

Re: [VOTE] Release 1.15.4, release candidate #1

2023-03-07 Thread Danny Cranmer
Hello all, I have been discussing the Python issue with Dian and it appears this issue has a large blast radius. Given we only have 1 binding vote so far I am inclined to cancel this release candidate in favour of an RC2. This will possibly remove the need for a 1.15.5 followup next month. The

Re: [DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

2023-03-07 Thread Jane Chan
Hi Lincoln, Thanks for bringing this to the discussion. +1 for the FLIP. Best, Jane On Tue, Mar 7, 2023 at 3:23 PM Aitozi wrote: > > we can initiate corresponding support issues for > specific connectors to follow up on support after finalizing the API > changes > > Make sense to me! > >

Re: [VOTE] FLIP-297: Improve Auxiliary Sql Statements

2023-03-07 Thread Jing Ge
+1 Thanks! Best regards, Jing On Tue, Mar 7, 2023 at 9:51 AM ConradJam wrote: > +1 (non-binding). > > Hang Ruan 于2023年3月7日周二 16:14写道: > > > Thanks for Ran's work. > > +1 (non-binding). > > > > Best, > > Hang > > > > Ran Tao 于2023年3月7日周二 15:59写道: > > > > > thanks Lau. > > > > > > The vote

[jira] [Created] (FLINK-31357) A record is deleted before being inserted, it will be deleted

2023-03-07 Thread Fangliang Liu (Jira)
Fangliang Liu created FLINK-31357: - Summary: A record is deleted before being inserted, it will be deleted Key: FLINK-31357 URL: https://issues.apache.org/jira/browse/FLINK-31357 Project: Flink

Re: [VOTE] FLIP-297: Improve Auxiliary Sql Statements

2023-03-07 Thread ConradJam
+1 (non-binding). Hang Ruan 于2023年3月7日周二 16:14写道: > Thanks for Ran's work. > +1 (non-binding). > > Best, > Hang > > Ran Tao 于2023年3月7日周二 15:59写道: > > > thanks Lau. > > > > The vote will last for at least 72 hours (03/09, 19:30 UTC+8). > > It needs consensus approval, requiring 3 binding +1

Re:[DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

2023-03-07 Thread Wencong Liu
Hello Yuxin, Thanks for your proposal! Adding remote storage capability to Flink's Hybrid Shuffle is a significant improvement that addresses the issue of local disk storage limitations. This enhancement not only ensures uninterrupted Shuffle, but also enables Flink to handle larger

Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

2023-03-07 Thread weijie guo
Hi Yuxin, As one of the main contributors of hybrid shuffle, I am very happy to see this improvement. IMO, hybrid shuffle should gradually replace the default batch shuffle implementation, because it is theoretically a superset of blocking shuffle(i.e. SortMergeShuffle / BoundedBlockingShuffle).

[jira] [Created] (FLINK-31356) Serialize garbled characters at checkpoint

2023-03-07 Thread John (Jira)
John created FLINK-31356: Summary: Serialize garbled characters at checkpoint Key: FLINK-31356 URL: https://issues.apache.org/jira/browse/FLINK-31356 Project: Flink Issue Type: Bug

Re: [VOTE] FLIP-297: Improve Auxiliary Sql Statements

2023-03-07 Thread Hang Ruan
Thanks for Ran's work. +1 (non-binding). Best, Hang Ran Tao 于2023年3月7日周二 15:59写道: > thanks Lau. > > The vote will last for at least 72 hours (03/09, 19:30 UTC+8). > It needs consensus approval, requiring 3 binding +1 votes and no > binding vetoes. > > > Best Regards, > Ran Tao > > > Jacky Lau