[jira] [Created] (FLINK-31634) FLIP-301: Hybrid Shuffle supports Remote Storage

2023-03-27 Thread Yuxin Tan (Jira)
Yuxin Tan created FLINK-31634: - Summary: FLIP-301: Hybrid Shuffle supports Remote Storage Key: FLINK-31634 URL: https://issues.apache.org/jira/browse/FLINK-31634 Project: Flink Issue Type: New

Re: [DISCUSS] FLIP-288:Enable Dynamic Partition Discovery by Default in Kafka Source

2023-03-27 Thread Shammon FY
Hi hongshun Thanks for updating, the FLIP's pretty good and looks good to me! Best, Shammon FY On Tue, Mar 28, 2023 at 11:16 AM Hongshun Wang wrote: > Hi Shammon, > > > Thanks a lot for your advise. I agree with your opinion now. It seems that > I forgot to consider that it may be at a

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Yuxin Tan
Congratulations! Best, Yuxin Guanghui Zhang 于2023年3月28日周二 11:06写道: > Congratulations! > > Best, > Zhang Guanghui > > Hang Ruan 于2023年3月28日周二 10:29写道: > > > Congratulations! > > > > Best, > > Hang > > > > yu zelin 于2023年3月28日周二 10:27写道: > > > >> Congratulations! > >> > >> Best, > >> Yu Zelin

[jira] [Created] (FLINK-31633) Hive dialect can't order by an unselected column

2023-03-27 Thread luoyuxia (Jira)
luoyuxia created FLINK-31633: Summary: Hive dialect can't order by an unselected column Key: FLINK-31633 URL: https://issues.apache.org/jira/browse/FLINK-31633 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-288:Enable Dynamic Partition Discovery by Default in Kafka Source

2023-03-27 Thread Hongshun Wang
Hi Shammon, Thanks a lot for your advise. I agree with your opinion now. It seems that I forgot to consider that it may be at a certain point in the future. I will modify OffsetsInitializer to provide a different strategy for later discovered partitions, by which users can also customize

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Guanghui Zhang
Congratulations! Best, Zhang Guanghui Hang Ruan 于2023年3月28日周二 10:29写道: > Congratulations! > > Best, > Hang > > yu zelin 于2023年3月28日周二 10:27写道: > >> Congratulations! >> >> Best, >> Yu Zelin >> >> 2023年3月27日 17:23,Yu Li 写道: >> >> Dear Flinkers, >> >> >> >> As you may have noticed, we are

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Yun Tang
Congratulations! Unlike other data-lakes, Paimon might be the first one to act as a stream-first (not batch-first) data-lake. Best Yun Tang From: Xianxun Ye Sent: Tuesday, March 28, 2023 10:52 To: dev@flink.apache.org Cc: Yu Li ; user ; user-zh ;

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Biao Liu
Congrats! Thanks, Biao /'bɪ.aʊ/ On Tue, 28 Mar 2023 at 10:29, Hang Ruan wrote: > Congratulations! > > Best, > Hang > > yu zelin 于2023年3月28日周二 10:27写道: > >> Congratulations! >> >> Best, >> Yu Zelin >> >> 2023年3月27日 17:23,Yu Li 写道: >> >> Dear Flinkers, >> >> >> >> As you may have noticed, we

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Hang Ruan
Congratulations! Best, Hang yu zelin 于2023年3月28日周二 10:27写道: > Congratulations! > > Best, > Yu Zelin > > 2023年3月27日 17:23,Yu Li 写道: > > Dear Flinkers, > > > > As you may have noticed, we are pleased to announce that Flink Table Store > has joined the Apache Incubator as a separate project

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread yu zelin
Congratulations! Best, Yu Zelin > 2023年3月27日 17:23,Yu Li 写道: > > Dear Flinkers, > > As you may have noticed, we are pleased to announce that Flink Table Store > has joined the Apache Incubator as a separate project called Apache > Paimon(incubating) [1] [2] [3]. The new project still aims

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread ConradJam
Congratulations! Yu Li 于2023年3月27日周一 17:24写道: > > Dear Flinkers, > > > As you may have noticed, we are pleased to announce that Flink Table > Store has joined the Apache Incubator as a separate project called > Apache Paimon(incubating) [1] [2] [3]. The new project still aims at > building a

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Jingsong Li
Congratulations! I believe Paimon will work with Flink to build the best streaming data warehouse. Best, Jingsong On Tue, Mar 28, 2023 at 8:27 AM Shammon FY wrote: > > Congratulations! > > > Best, > Shammon FY > > On Mon, Mar 27, 2023 at 11:37 PM Samrat Deb wrote: > > > congratulations > > >

[jira] [Created] (FLINK-31632) watermark aligned idle source can't resume

2023-03-27 Thread haishui (Jira)
haishui created FLINK-31632: --- Summary: watermark aligned idle source can't resume Key: FLINK-31632 URL: https://issues.apache.org/jira/browse/FLINK-31632 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-288:Enable Dynamic Partition Discovery by Default in Kafka Source

2023-03-27 Thread Shammon FY
Hi Hongshun Thanks for your answer. I think the startup offset of Kafka such as timestamp or specific_offset has no relationship with `Window Operator`. Users can freely set the starting position according to their needs, it may be before the latest Kafka data, or it may be at a certain point in

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Shammon FY
Congratulations! Best, Shammon FY On Mon, Mar 27, 2023 at 11:37 PM Samrat Deb wrote: > congratulations > > Bests, > Samrat > On Mon, 27 Mar 2023 at 8:24 PM, Alexander Fedulov < > alexander.fedu...@gmail.com> wrote: > > > Great to see this, congratulations! > > > > Best, > > Alex > > > > On

Re: [DISCUSS] FLIP-292: Support configuring state TTL at operator level for Table API & SQL programs

2023-03-27 Thread Jing Ge
Hi Jane, Thanks for clarifying it. As far as I am concerned, the issue is where to keep the user's job metadata, i.e. SQL script (to make the discussion easier, let's ignore config). As long as FLIP-190 is only used for migration/upgrade, SQL is the single source of truth. Once the compiled plan

Re: Subtask distribution in Flink

2023-03-27 Thread santhosh venkat
Hi, Thank you so much for taking time to answer my questions and pointing me to relevant documentation. Really appreciate it. When the task failover happens, are there internal metrics in Flink at a job level to track the new execution attempt? Is there a way for the application owner to figure

[jira] [Created] (FLINK-31631) Upgrade GCS connector to 2.2.11.

2023-03-27 Thread Chris Nauroth (Jira)
Chris Nauroth created FLINK-31631: - Summary: Upgrade GCS connector to 2.2.11. Key: FLINK-31631 URL: https://issues.apache.org/jira/browse/FLINK-31631 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-31630) Support max-age for checkpoints during last-state upgrades

2023-03-27 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-31630: -- Summary: Support max-age for checkpoints during last-state upgrades Key: FLINK-31630 URL: https://issues.apache.org/jira/browse/FLINK-31630 Project: Flink Issue

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Samrat Deb
congratulations Bests, Samrat On Mon, 27 Mar 2023 at 8:24 PM, Alexander Fedulov < alexander.fedu...@gmail.com> wrote: > Great to see this, congratulations! > > Best, > Alex > > On Mon, 27 Mar 2023 at 11:24, Yu Li wrote: > > > Dear Flinkers, > > > > > > > > As you may have noticed, we are

Re: [ANNOUNCE] Kafka Connector Code Removal from apache/flink:main branch and code freezing

2023-03-27 Thread Tzu-Li (Gordon) Tai
Thanks for the updates. So far the above mentioned issues seem to all have PRs against apache/flink-connector-kafka now. To be clear, this notice isn't about discussing _what_ PRs we will be merging or not merging - we should try to review all of them eventually. The only reason I've made a list

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Alexander Fedulov
Great to see this, congratulations! Best, Alex On Mon, 27 Mar 2023 at 11:24, Yu Li wrote: > Dear Flinkers, > > > > As you may have noticed, we are pleased to announce that Flink Table Store > has joined the Apache Incubator as a separate project called Apache > Paimon(incubating) [1] [2]

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Samrat Deb
congratulations Bests, Samrat On Mon, Mar 27, 2023 at 7:19 PM Yanfei Lei wrote: > Congratulations! > > Best Regards, > Yanfei > > ramkrishna vasudevan 于2023年3月27日周一 21:46写道: > > > > Congratulations !!! > > > > On Mon, Mar 27, 2023 at 2:54 PM Yu Li wrote: > >> > >> Dear Flinkers, > >> > >> >

Re: [NOTICE] Website now has a staging environment

2023-03-27 Thread Chesnay Schepler
It's just another tool in your belt. You can use it, but don't have to. We used it for the Hugo migration, where we could only test some things (like the rewrite rules) in a production-like environment. I'm +0 on whether to move the docs to the repo or not. On 27/03/2023 14:46, Jing Ge

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Yanfei Lei
Congratulations! Best Regards, Yanfei ramkrishna vasudevan 于2023年3月27日周一 21:46写道: > > Congratulations !!! > > On Mon, Mar 27, 2023 at 2:54 PM Yu Li wrote: >> >> Dear Flinkers, >> >> >> As you may have noticed, we are pleased to announce that Flink Table Store >> has joined the Apache

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread ramkrishna vasudevan
Congratulations !!! On Mon, Mar 27, 2023 at 2:54 PM Yu Li wrote: > Dear Flinkers, > > > > As you may have noticed, we are pleased to announce that Flink Table Store > has joined the Apache Incubator as a separate project called Apache > Paimon(incubating) [1] [2] [3]. The new project still

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread feng xiangyu
Congrats Yu! Looking forward to contributing to Paimon! Best Regards, Xiangyu yuxia 于2023年3月27日周一 21:01写道: > congratulations! > > Best regards, > Yuxia > > > 发件人: "Andrew Otto" > 收件人: "Matthias Pohl" > 抄送: "Jing Ge" , "Leonard Xu" , "Yu > Li" , "dev" , "User" < > u...@flink.apache.org>,

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread yuxia
congratulations! Best regards, Yuxia 发件人: "Andrew Otto" 收件人: "Matthias Pohl" 抄送: "Jing Ge" , "Leonard Xu" , "Yu Li" , "dev" , "User" , "user-zh" 发送时间: 星期一, 2023年 3 月 27日 下午 8:57:50 主题: Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating) Exciting!

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Samrat Deb
congratulations 拾 On Mon, 27 Mar 2023 at 6:02 PM, Leonard Xu wrote: > Congratulations! > > > Best, > Leonard > > > On Mar 27, 2023, at 5:23 PM, Yu Li wrote: > > > > Dear Flinkers, > > > > As you may have noticed, we are pleased to announce that Flink Table > Store has joined the Apache

Re: [NOTICE] Website now has a staging environment

2023-03-27 Thread Jing Ge
Wow, that is good news! Thanks Chesnay for your effort. I'd like to ask the same question as what Matthias asked. Speaking of the workflow, should we first push changes to the asf-staging branch and check it at https://flink.staged.apache.org and then cp them to the asf-site? Should there always

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Matthias Pohl
Congratulations and good luck with pushing the project forward. On Mon, Mar 27, 2023 at 2:35 PM Jing Ge via user wrote: > Congrats! > > Best regards, > Jing > > On Mon, Mar 27, 2023 at 2:32 PM Leonard Xu wrote: > >> Congratulations! >> >> >> Best, >> Leonard >> >> On Mar 27, 2023, at 5:23 PM,

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Konstantin Knauf
Congrats! Am Mo., 27. März 2023 um 14:32 Uhr schrieb Leonard Xu : > Congratulations! > > > Best, > Leonard > > On Mar 27, 2023, at 5:23 PM, Yu Li wrote: > > Dear Flinkers, > > > > As you may have noticed, we are pleased to announce that Flink Table Store > has joined the Apache Incubator as a

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Jing Ge
Congrats! Best regards, Jing On Mon, Mar 27, 2023 at 2:32 PM Leonard Xu wrote: > Congratulations! > > > Best, > Leonard > > On Mar 27, 2023, at 5:23 PM, Yu Li wrote: > > Dear Flinkers, > > > > As you may have noticed, we are pleased to announce that Flink Table Store > has joined the Apache

Re: [ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Leonard Xu
Congratulations! Best, Leonard > On Mar 27, 2023, at 5:23 PM, Yu Li wrote: > > Dear Flinkers, > > As you may have noticed, we are pleased to announce that Flink Table Store > has joined the Apache Incubator as a separate project called Apache > Paimon(incubating) [1] [2] [3]. The new

Re: [NOTICE] Website now has a staging environment

2023-03-27 Thread Matthias Pohl
Thanks for sharing that information. Is there a specific workflow expected for pushing changes? Shall we always go through the staging environment first or is it more like a nice-to-have feature? I'm also wondering whether we should move the documentation from the Confluent page into the README

[jira] [Created] (FLINK-31629) Trying to access closed classloader when submit query to restSqlGateway via SqlClient

2023-03-27 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-31629: -- Summary: Trying to access closed classloader when submit query to restSqlGateway via SqlClient Key: FLINK-31629 URL: https://issues.apache.org/jira/browse/FLINK-31629

Re: Subtask distribution in Flink

2023-03-27 Thread Weihua Hu
Hi, 1. Does this mean that each task slot will contain an entire pipeline in > the job? not exactly, each slot will run a subtask of each task. If the job is so simple that there is no keyby logic and we do not enable rebalance shuffle type, each slot could run all the pipeline. But if not we

[jira] [Created] (FLINK-31628) ArrayIndexOutOfBoundsException in watermark processing

2023-03-27 Thread Michael Helmling (Jira)
Michael Helmling created FLINK-31628: Summary: ArrayIndexOutOfBoundsException in watermark processing Key: FLINK-31628 URL: https://issues.apache.org/jira/browse/FLINK-31628 Project: Flink

[ANNOUNCE] Flink Table Store Joins Apache Incubator as Apache Paimon(incubating)

2023-03-27 Thread Yu Li
Dear Flinkers, As you may have noticed, we are pleased to announce that Flink Table Store has joined the Apache Incubator as a separate project called Apache Paimon(incubating) [1] [2] [3]. The new project still aims at building a streaming data lake platform for high-speed data ingestion,

[jira] [Created] (FLINK-31627) docker-build.sh build fails on Linux machines

2023-03-27 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-31627: - Summary: docker-build.sh build fails on Linux machines Key: FLINK-31627 URL: https://issues.apache.org/jira/browse/FLINK-31627 Project: Flink Issue Type:

[jira] [Created] (FLINK-31626) HsSubpartitionFileReaderImpl should recycle skipped read buffers.

2023-03-27 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-31626: -- Summary: HsSubpartitionFileReaderImpl should recycle skipped read buffers. Key: FLINK-31626 URL: https://issues.apache.org/jira/browse/FLINK-31626 Project: Flink

[jira] [Created] (FLINK-31625) Possbile OOM in KBinsDiscretizer

2023-03-27 Thread Fan Hong (Jira)
Fan Hong created FLINK-31625: Summary: Possbile OOM in KBinsDiscretizer Key: FLINK-31625 URL: https://issues.apache.org/jira/browse/FLINK-31625 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-31624) [Connectors/Jdbc] Update flink to 1.17.0

2023-03-27 Thread Jira
João Boto created FLINK-31624: - Summary: [Connectors/Jdbc] Update flink to 1.17.0 Key: FLINK-31624 URL: https://issues.apache.org/jira/browse/FLINK-31624 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-31623) Improvements on DataStreamUtils#sample

2023-03-27 Thread Fan Hong (Jira)
Fan Hong created FLINK-31623: Summary: Improvements on DataStreamUtils#sample Key: FLINK-31623 URL: https://issues.apache.org/jira/browse/FLINK-31623 Project: Flink Issue Type: Improvement

Subtask distribution in Flink

2023-03-27 Thread santhosh venkat
Hi, I am trying to understand how subtask distribution works in Flink. Let's assume a setup of a Flink cluster with a fixed number of TaskManagers in a kubernetes cluster. Let's say I have a flink job with all the operators having the same parallelism and with the same Slot sharing group. The

Re: [External] [DISCUSS] FLIP-292: Support configuring state TTL at operator level for Table API & SQL programs

2023-03-27 Thread Yisha Zhou
Hi guys Thanks for your enlightening opinions which have benefited me greatly. And I have some experiences about configuring something on graphical IDE to share with you. Hopefully, these experiences can give you some useful information. At the FFA 2021, we proposed a solution for state

Re: [DISCUSS] Planning Flink 1.18

2023-03-27 Thread Matthias Pohl
Thanks Konstantin for driving this. I have no objections in regards to the feature freeze happening in July. Thanks everyone for volunteering to drive 1.18. +1 on the people who offered to help PS: I shared learnings from the 1.17 release in the other thread [1] to not spam this thread with

Re: [ANNOUNCE] Apache Flink 1.17.0 released

2023-03-27 Thread Matthias Pohl
Here are a few things I noticed from the 1.17 release retrospectively which I want to share (other release managers might have a different view or might disagree): - Google Meet might not be the best choice for the release sync. We need to be able to invite attendees even if the creator of the

Re: [VOTE] FLIP-284: Making AsyncSinkWriter Flush triggers adjustable

2023-03-27 Thread Danny Cranmer
Thanks for driving this Ahmed. This change will benefit future and current sink implementations by giving the developer more flexibility. A good addition following configurable rate limiting strategies [1]. +1 (binding) Thanks, Danny [1]

Re: [DISCUSS] FLIP-304: Pluggable failure handling for Apache Flink

2023-03-27 Thread Gen Luo
Thanks for the summary! Also +1 to support custom restart strategies in a different FLIP, as long as we can make sure that the plugin interface won't be changed when the restart strategy interface is introduced. To achieve this, maybe we should think well how the handler would cooperate with the

Re: [DISCUSS] FLIP-292: Support configuring state TTL at operator level for Table API & SQL programs

2023-03-27 Thread Jane Chan
Hi Shuo, Thanks for your comments. > The title of this FLIP is "For Table API & SQL", but this proposal actually only covers SQL jobs via compiled json plan Please correct me if I'm wrong, but I'm thinking that the scope of this FLIP should be consistent with the FLIP-190. Here is an example to