Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang

[Yun Tang]

Congratulations, Yong Fang 


Best
Yun Tang

From: Feifan Wang 
Sent: Wednesday, July 26, 2023 10:36
To: dev@flink.apache.org 
Cc: dev@flink.apache.org 
Subject: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang

Congratulations, Yong Fang!

Best,
Feifan Wang


| |
Feifan Wang
|
|
zoltar9...@163.com
|


 Replied Message 
| From | Yuxin Tan |
| Date | 07/26/2023 10:25 |
| To |  |
| Subject | Re: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang |
Congratulations, Yong Fang!

Best,
Yuxin


Yanfei Lei  于2023年7月26日周三 10:18写道：

Congratulations!

Best regards,
Yanfei

weijie guo  于2023年7月26日周三 10:10写道：

Congrats, Yong Fang!

Best regards,

Weijie


Danny Cranmer  于2023年7月26日周三 03:34写道：

Congrats and welcome!

Danny.

On Tue, 25 Jul 2023, 16:48 Matthias Pohl, 
wrote:

Congratulations :)

On Tue, Jul 25, 2023 at 5:13 PM Jing Ge 
wrote:

Congrats, Yong Fang!

Best regards,
Jing

On Tue, Jul 25, 2023 at 7:35 PM Yu Li  wrote:

Congrats, Yong!

Best Regards,
Yu


On Tue, 25 Jul 2023 at 18:03, Sergey Nuyanzin <
snuyan...@gmail.com>
wrote:

Congratulations, Yong Fang!

On Tue, Jul 25, 2023 at 7:53 AM ConradJam  于2023年7月25日周二 12:08写道：

Congratulations, Yong Fang!


--

Best regards,
Mang Zhang





在 2023-07-25 10:30:24，"Jark Wu"  写道：
Congratulations, Yong Fang!

Best,
Jark

On Mon, 24 Jul 2023 at 22:11, Wencong Liu <
liuwencle...@163.com

wrote:

Congratulations!

Best,
Wencong Liu















在 2023-07-24 11:03:30，"Paul Lam" 

Re: [DISCUSS] Feature freeze and deprecation work for 2.0

[Xintong Song]

Thanks for the response, Qingsheng.

I'm fine with not allowing new features after the 1.18 freeze.

Just want to double-check, how about the FLIPs that purely mark things as
`@Deprecated` without adding anything new? Do we agree to treat them as
"not new features"?

Best,

Xintong



On Wed, Jul 26, 2023 at 12:08 PM Qingsheng Ren  wrote:

> (Sorry for resending this. I forgot to cc the dev mailing list)
>
> Hi Matthias and Xintong,
>
> Thanks for raising the question! We brought it to the 1.18 release sync on
> Jul 26th, and we decided to stick to the original schedule of 1.18 and will
> not accept new features, including those deprecation works. We don't see
> benefits to 2.0 clearly about giving another several weeks squeezing these
> deprecations into 1.18. I checked the latest FLIPs and most of them have
> not been voted on yet, so we are a bit concerned about the overhead of
> evaluating these cases and potentially delaying the release of 1.18.
>
> What about we have a better, clearer plan on deprecations at the beginning
> of the next release cycle, considering FLIP-321 and 2.0 working items are
> finalized quite nearing the feature freeze of 1.18?
>
> Best,
> Qingsheng, Jing, Konstantin and Sergey
>
> On Wed, Jul 26, 2023 at 12:06 PM Qingsheng Ren  wrote:
>
>> Hi Matthias and Xintong,
>>
>> Thanks for raising the question! We brought it to the 1.18 release sync
>> on Jul 26th, and we decided to stick to the original schedule of 1.18 and
>> will not accept new features, including those deprecation works. We don't
>> see benefits to 2.0 clearly about giving another several weeks squeezing
>> these deprecations into 1.18. I checked the latest FLIPs and most of them
>> have not been voted on yet, so we are a bit concerned about the overhead of
>> evaluating these cases and potentially delaying the release of 1.18.
>>
>> What about we have a better, clearer plan on deprecations at the
>> beginning of the next release cycle, considering FLIP-321 and 2.0 working
>> items are finalized quite nearing the feature freeze of 1.18?
>>
>> Best,
>> Qingsheng, Jing, Konstantin and Sergey
>>
>> On Fri, Jul 21, 2023 at 5:28 PM Xintong Song 
>> wrote:
>>
>>> Good question. CC-ed the release managers.
>>>
>>> My 2-cents:
>>> I think the purpose of feature freeze is to prevent new feature /
>>> improvement changes from destabilizing the code base, in order to get a
>>> stable and verified release. Based on this, I'd suggest:
>>> - Considering FLIPs that purely mark an API as deprecated and do not
>>> introduce anything new as "not a new feature", because that can hardly
>>> cause any trouble.
>>> - Considering FLIPs that introduce new / replacing APIs in addition to
>>> deprecating old ones as "new features or improvements". Merging codes for
>>> such FLIPs after the feature freeze should be carefully evaluated and
>>> require permissions from the release managers.
>>> - Further extending the feature freeze might also be an option, but I
>>> personally don't think it's necessary to block the release testing. Most of
>>> the recent API deprecation FLIPs require only minor changes that IHMO can
>>> barely affect the stability of the codebase.
>>>
>>> But this should be the release managers' call. Looking forward to your
>>> opinions.
>>>
>>> Best,
>>>
>>> Xintong
>>>
>>>
>>>
>>> On Fri, Jul 21, 2023 at 4:25 PM Matthias Pohl
>>>  wrote:
>>>
 The feature freeze was postponed to July 24 (end of this week in
 Europe/early morning Monday in East Asia) in [1]. What's the 1.18
 release
 managers' take on all the FLIPs that were recently started and require
 some
 deprecation work (which ideally should go into 1.18)? How does that work
 with the feature freeze happening by the end of this week?

 - Do we plan to extend the feature freeze to allow deprecation changes
 to
 go in?
 - Do we consider depreciation work to be "not a new feature" which means
 that we're ok with merging them after the feature freeze?

 Best,
 Matthias

 [1] https://lists.apache.org/thread/9fck1m5llrnv5gx1od05tzx84oy0b4z0

>>>

Re: [DISCUSS] Feature freeze and deprecation work for 2.0

[Qingsheng Ren]

(Sorry for resending this. I forgot to cc the dev mailing list)

Hi Matthias and Xintong,

Thanks for raising the question! We brought it to the 1.18 release sync on
Jul 26th, and we decided to stick to the original schedule of 1.18 and will
not accept new features, including those deprecation works. We don't see
benefits to 2.0 clearly about giving another several weeks squeezing these
deprecations into 1.18. I checked the latest FLIPs and most of them have
not been voted on yet, so we are a bit concerned about the overhead of
evaluating these cases and potentially delaying the release of 1.18.

What about we have a better, clearer plan on deprecations at the beginning
of the next release cycle, considering FLIP-321 and 2.0 working items are
finalized quite nearing the feature freeze of 1.18?

Best,
Qingsheng, Jing, Konstantin and Sergey

On Wed, Jul 26, 2023 at 12:06 PM Qingsheng Ren  wrote:

> Hi Matthias and Xintong,
>
> Thanks for raising the question! We brought it to the 1.18 release sync on
> Jul 26th, and we decided to stick to the original schedule of 1.18 and will
> not accept new features, including those deprecation works. We don't see
> benefits to 2.0 clearly about giving another several weeks squeezing these
> deprecations into 1.18. I checked the latest FLIPs and most of them have
> not been voted on yet, so we are a bit concerned about the overhead of
> evaluating these cases and potentially delaying the release of 1.18.
>
> What about we have a better, clearer plan on deprecations at the
> beginning of the next release cycle, considering FLIP-321 and 2.0 working
> items are finalized quite nearing the feature freeze of 1.18?
>
> Best,
> Qingsheng, Jing, Konstantin and Sergey
>
> On Fri, Jul 21, 2023 at 5:28 PM Xintong Song 
> wrote:
>
>> Good question. CC-ed the release managers.
>>
>> My 2-cents:
>> I think the purpose of feature freeze is to prevent new feature /
>> improvement changes from destabilizing the code base, in order to get a
>> stable and verified release. Based on this, I'd suggest:
>> - Considering FLIPs that purely mark an API as deprecated and do not
>> introduce anything new as "not a new feature", because that can hardly
>> cause any trouble.
>> - Considering FLIPs that introduce new / replacing APIs in addition to
>> deprecating old ones as "new features or improvements". Merging codes for
>> such FLIPs after the feature freeze should be carefully evaluated and
>> require permissions from the release managers.
>> - Further extending the feature freeze might also be an option, but I
>> personally don't think it's necessary to block the release testing. Most of
>> the recent API deprecation FLIPs require only minor changes that IHMO can
>> barely affect the stability of the codebase.
>>
>> But this should be the release managers' call. Looking forward to your
>> opinions.
>>
>> Best,
>>
>> Xintong
>>
>>
>>
>> On Fri, Jul 21, 2023 at 4:25 PM Matthias Pohl
>>  wrote:
>>
>>> The feature freeze was postponed to July 24 (end of this week in
>>> Europe/early morning Monday in East Asia) in [1]. What's the 1.18 release
>>> managers' take on all the FLIPs that were recently started and require
>>> some
>>> deprecation work (which ideally should go into 1.18)? How does that work
>>> with the feature freeze happening by the end of this week?
>>>
>>> - Do we plan to extend the feature freeze to allow deprecation changes to
>>> go in?
>>> - Do we consider depreciation work to be "not a new feature" which means
>>> that we're ok with merging them after the feature freeze?
>>>
>>> Best,
>>> Matthias
>>>
>>> [1] https://lists.apache.org/thread/9fck1m5llrnv5gx1od05tzx84oy0b4z0
>>>
>>

Re: [DISCUSS][2.0] FLIP-349: Move RocksDB statebackend classes to o.a.f.state.rocksdb package

[Yanfei Lei]

+1 for moving all classes in the state-backend-rocksdb module under
the classes to o.a.f.state.rocksdb package.

I have always been curious about the relationship between
o.a.f.contrib.xx and the flink-contrib module. :)

Best,
Yanfei

Jing Ge  于2023年7月25日周二 17:50写道：
>
> make sense.
>
> Best regards,
> Jing
>
> On Tue, Jul 25, 2023 at 4:29 PM Stefan Richter
>  wrote:
>
> >
> > +1
> >
> >
> >
> > > On 24. Jul 2023, at 12:25, Chesnay Schepler  wrote:
> > >
> > > To properly reflect the state of the rocksdb statebackend I propose to
> > move all classes in the state-backend-rocksdb module under the classes to
> > o.a.f.state.rocksdb package.
> > >
> > >
> > >
> > https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/FLINK/FLIP-349%253A%2BMove%2BRocksDB%2Bstatebackend%2Bclasses%2Bto%2Bo.a.f.state.rocksdb%2Bpackage=gmail-imap=169079912800=AOvVaw3OiTwgsLEiTcJpNTVM-Y8y
> >
> >

Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang

[Feifan Wang]

Congratulations, Yong Fang!

Best,
Feifan Wang


| |
Feifan Wang
|
|
zoltar9...@163.com
|


 Replied Message 
| From | Yuxin Tan |
| Date | 07/26/2023 10:25 |
| To |  |
| Subject | Re: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang |
Congratulations, Yong Fang!

Best,
Yuxin


Yanfei Lei  于2023年7月26日周三 10:18写道：

Congratulations!

Best regards,
Yanfei

weijie guo  于2023年7月26日周三 10:10写道：

Congrats, Yong Fang!

Best regards,

Weijie


Danny Cranmer  于2023年7月26日周三 03:34写道：

Congrats and welcome!

Danny.

On Tue, 25 Jul 2023, 16:48 Matthias Pohl, 
wrote:

Congratulations :)

On Tue, Jul 25, 2023 at 5:13 PM Jing Ge 
wrote:

Congrats, Yong Fang!

Best regards,
Jing

On Tue, Jul 25, 2023 at 7:35 PM Yu Li  wrote:

Congrats, Yong!

Best Regards,
Yu


On Tue, 25 Jul 2023 at 18:03, Sergey Nuyanzin <
snuyan...@gmail.com>
wrote:

Congratulations, Yong Fang!

On Tue, Jul 25, 2023 at 7:53 AM ConradJam  于2023年7月25日周二 12:08写道：

Congratulations, Yong Fang!


--

Best regards,
Mang Zhang





在 2023-07-25 10:30:24，"Jark Wu"  写道：
Congratulations, Yong Fang!

Best,
Jark

On Mon, 24 Jul 2023 at 22:11, Wencong Liu <
liuwencle...@163.com

wrote:

Congratulations!

Best,
Wencong Liu















在 2023-07-24 11:03:30，"Paul Lam" 

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang

[Yuxin Tan]

Congratulations, Yong Fang!

Best,
Yuxin


Yanfei Lei  于2023年7月26日周三 10:18写道：

> Congratulations!
>
> Best regards,
> Yanfei
>
> weijie guo  于2023年7月26日周三 10:10写道：
> >
> > Congrats, Yong Fang!
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Danny Cranmer  于2023年7月26日周三 03:34写道：
> >
> > > Congrats and welcome!
> > >
> > > Danny.
> > >
> > > On Tue, 25 Jul 2023, 16:48 Matthias Pohl,  .invalid>
> > > wrote:
> > >
> > > > Congratulations :)
> > > >
> > > > On Tue, Jul 25, 2023 at 5:13 PM Jing Ge 
> > > > wrote:
> > > >
> > > > > Congrats, Yong Fang!
> > > > >
> > > > > Best regards,
> > > > > Jing
> > > > >
> > > > > On Tue, Jul 25, 2023 at 7:35 PM Yu Li  wrote:
> > > > >
> > > > > > Congrats, Yong!
> > > > > >
> > > > > > Best Regards,
> > > > > > Yu
> > > > > >
> > > > > >
> > > > > > On Tue, 25 Jul 2023 at 18:03, Sergey Nuyanzin <
> snuyan...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Congratulations, Yong Fang!
> > > > > > >
> > > > > > > On Tue, Jul 25, 2023 at 7:53 AM ConradJam  >
> > > > wrote:
> > > > > > >
> > > > > > > > Congratulations, Yong Fang
> > > > > > > >
> > > > > > > > Mang Zhang  于2023年7月25日周二 12:08写道：
> > > > > > > >
> > > > > > > > > Congratulations, Yong Fang!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Mang Zhang
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 在 2023-07-25 10:30:24，"Jark Wu"  写道：
> > > > > > > > > >Congratulations, Yong Fang!
> > > > > > > > > >
> > > > > > > > > >Best,
> > > > > > > > > >Jark
> > > > > > > > > >
> > > > > > > > > >On Mon, 24 Jul 2023 at 22:11, Wencong Liu <
> > > liuwencle...@163.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Congratulations!
> > > > > > > > > >>
> > > > > > > > > >> Best,
> > > > > > > > > >> Wencong Liu
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> 在 2023-07-24 11:03:30，"Paul Lam"  >
> > > 写道：
> > > > > > > > > >> >Congrats, Shammon!
> > > > > > > > > >> >
> > > > > > > > > >> >Best,
> > > > > > > > > >> >Paul Lam
> > > > > > > > > >> >
> > > > > > > > > >> >> 2023年7月24日 10:56，Jingsong Li  >
> > > 写道：
> > > > > > > > > >> >>
> > > > > > > > > >> >> Shammon
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Best
> > > > > > > >
> > > > > > > > ConradJam
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best regards,
> > > > > > > Sergey
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>

[jira] [Created] (FLINK-32676) Add doc for catalog modification listener

[Fang Yong (Jira)]

Fang Yong created FLINK-32676:
-

 Summary: Add doc for catalog modification listener
 Key: FLINK-32676
 URL: https://issues.apache.org/jira/browse/FLINK-32676
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Table SQL / Runtime
Affects Versions: 1.18.0
Reporter: Fang Yong


Add doc for catalog modification listener



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang

[Yanfei Lei]

Congratulations!

Best regards,
Yanfei

weijie guo  于2023年7月26日周三 10:10写道：
>
> Congrats, Yong Fang!
>
> Best regards,
>
> Weijie
>
>
> Danny Cranmer  于2023年7月26日周三 03:34写道：
>
> > Congrats and welcome!
> >
> > Danny.
> >
> > On Tue, 25 Jul 2023, 16:48 Matthias Pohl, 
> > wrote:
> >
> > > Congratulations :)
> > >
> > > On Tue, Jul 25, 2023 at 5:13 PM Jing Ge 
> > > wrote:
> > >
> > > > Congrats, Yong Fang!
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Tue, Jul 25, 2023 at 7:35 PM Yu Li  wrote:
> > > >
> > > > > Congrats, Yong!
> > > > >
> > > > > Best Regards,
> > > > > Yu
> > > > >
> > > > >
> > > > > On Tue, 25 Jul 2023 at 18:03, Sergey Nuyanzin 
> > > > wrote:
> > > > >
> > > > > > Congratulations, Yong Fang!
> > > > > >
> > > > > > On Tue, Jul 25, 2023 at 7:53 AM ConradJam 
> > > wrote:
> > > > > >
> > > > > > > Congratulations, Yong Fang
> > > > > > >
> > > > > > > Mang Zhang  于2023年7月25日周二 12:08写道：
> > > > > > >
> > > > > > > > Congratulations, Yong Fang!
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Mang Zhang
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > 在 2023-07-25 10:30:24，"Jark Wu"  写道：
> > > > > > > > >Congratulations, Yong Fang!
> > > > > > > > >
> > > > > > > > >Best,
> > > > > > > > >Jark
> > > > > > > > >
> > > > > > > > >On Mon, 24 Jul 2023 at 22:11, Wencong Liu <
> > liuwencle...@163.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Congratulations!
> > > > > > > > >>
> > > > > > > > >> Best,
> > > > > > > > >> Wencong Liu
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> 在 2023-07-24 11:03:30，"Paul Lam" 
> > 写道：
> > > > > > > > >> >Congrats, Shammon!
> > > > > > > > >> >
> > > > > > > > >> >Best,
> > > > > > > > >> >Paul Lam
> > > > > > > > >> >
> > > > > > > > >> >> 2023年7月24日 10:56，Jingsong Li 
> > 写道：
> > > > > > > > >> >>
> > > > > > > > >> >> Shammon
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Best
> > > > > > >
> > > > > > > ConradJam
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best regards,
> > > > > > Sergey
> > > > > >
> > > > >
> > > >
> > >
> >

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang

[weijie guo]

Congrats, Yong Fang!

Best regards,

Weijie


Danny Cranmer  于2023年7月26日周三 03:34写道：

> Congrats and welcome!
>
> Danny.
>
> On Tue, 25 Jul 2023, 16:48 Matthias Pohl, 
> wrote:
>
> > Congratulations :)
> >
> > On Tue, Jul 25, 2023 at 5:13 PM Jing Ge 
> > wrote:
> >
> > > Congrats, Yong Fang!
> > >
> > > Best regards,
> > > Jing
> > >
> > > On Tue, Jul 25, 2023 at 7:35 PM Yu Li  wrote:
> > >
> > > > Congrats, Yong!
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Tue, 25 Jul 2023 at 18:03, Sergey Nuyanzin 
> > > wrote:
> > > >
> > > > > Congratulations, Yong Fang!
> > > > >
> > > > > On Tue, Jul 25, 2023 at 7:53 AM ConradJam 
> > wrote:
> > > > >
> > > > > > Congratulations, Yong Fang
> > > > > >
> > > > > > Mang Zhang  于2023年7月25日周二 12:08写道：
> > > > > >
> > > > > > > Congratulations, Yong Fang!
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Mang Zhang
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 在 2023-07-25 10:30:24，"Jark Wu"  写道：
> > > > > > > >Congratulations, Yong Fang!
> > > > > > > >
> > > > > > > >Best,
> > > > > > > >Jark
> > > > > > > >
> > > > > > > >On Mon, 24 Jul 2023 at 22:11, Wencong Liu <
> liuwencle...@163.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > >> Congratulations!
> > > > > > > >>
> > > > > > > >> Best,
> > > > > > > >> Wencong Liu
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> 在 2023-07-24 11:03:30，"Paul Lam" 
> 写道：
> > > > > > > >> >Congrats, Shammon!
> > > > > > > >> >
> > > > > > > >> >Best,
> > > > > > > >> >Paul Lam
> > > > > > > >> >
> > > > > > > >> >> 2023年7月24日 10:56，Jingsong Li 
> 写道：
> > > > > > > >> >>
> > > > > > > >> >> Shammon
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best
> > > > > >
> > > > > > ConradJam
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Sergey
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] Release 2.0 must-have work items - Round 2

[Xintong Song]

Hi Konstantin,

It seems the offline discussion has already taken place [1], and part of
the outcome is that removal of SourceFunction would be a *nice-to-have*
item for release 2.0 which may not block this *must-have* vote. Do you have
different opinions about the conclusions in [1]?

If there are still concerns, and the discussion around this topic needs to
be continued, then I'd suggest (as I mentioned in [2]) not to further block
this vote (i.e. the decision on other must-have items). Release 2.0 still
has a long way to go, and it is likely we need to review and update the
list every once in a while. We can update the list with another vote if
later we decide to add the removal of SourceFunction to the must-have list.

WDYT?

Best,

Xintong


[1] https://lists.apache.org/thread/yyw52k45x2sp1jszldtdx7hc98n72w7k
[2] https://lists.apache.org/thread/j5d5022ky8k5t088ffm03727o5g9x9jr

On Tue, Jul 25, 2023 at 8:49 PM Konstantin Knauf  wrote:

> I assume this vote includes a decision to not removing
> SourceFunction/SinkFunction in Flink 2.0 (as it has been removed from the
> table). If this is the case, I don't think, this discussion has concluded.
> There are multiple contributors like myself, Martijn, Alex Fedulov and
> Maximilian Michels, who have indicated they would be in favor of
> deprecating/dropping them. This Source/Sink Function discussion seems to go
> in circles in general. I am wondering if it makes sense to have a call
> about this instead of repeating mailing list discussions.
>
> Am Di., 25. Juli 2023 um 13:38 Uhr schrieb Yu Li :
>
> > +1 (binding)
> >
> > Thanks for driving this, Xintong!
> >
> > Best Regards,
> > Yu
> >
> >
> > On Sun, 23 Jul 2023 at 18:28, Yuan Mei  wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks for driving the discussion through and for all the efforts in
> > > resolving the complexities :-)
> > >
> > > Best
> > > Yuan
> > >
> > > On Thu, Jul 20, 2023 at 5:23 PM Xintong Song 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'd like to start another round of VOTE for the must-have work items
> > for
> > > > release 2.0 [1]. The corresponding discussion thread is [2], and the
> > > > previous voting thread is [3]. All comments from the previous voting
> > > thread
> > > > have been addressed.
> > > >
> > > > Please note that once the vote is approved, any changes to the
> > must-have
> > > > items (adding / removing must-have items, changing the priority)
> > requires
> > > > another vote. Assigning contributors / reviewers, updating
> > descriptions /
> > > > progress, changes to nice-to-have items do not require another vote.
> > > >
> > > > The vote will be open until at least July 25, following the consensus
> > > > voting process. Votes of PMC members are binding.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > > [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> > > >
> > > > [2] https://lists.apache.org/thread/l3dkdypyrovd3txzodn07lgdwtwvhgk4
> > > >
> > > > [3] https://lists.apache.org/thread/r0y9syc6k5nmcxvnd0hj33htdpdj9k6m
> > > >
> > >
> >
>
>
> --
> https://twitter.com/snntrable
> https://github.com/knaufk
>

[jira] [Created] (FLINK-32675) Add doc for the tiered storage of hybrid shuffle

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-32675:
-

 Summary: Add doc for the tiered storage of hybrid shuffle
 Key: FLINK-32675
 URL: https://issues.apache.org/jira/browse/FLINK-32675
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Runtime / Network
Affects Versions: 1.18.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


The new Hybrid Shuffle mode supporting remote storage 
(https://issues.apache.org/jira/browse/FLINK-30469) has finished, we should 
also update the Flink doc of Hybrid Shuffle.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release 2.0 must-have work items

[Xintong Song]

Thanks Leonard for driving this, and thanks everyone for the discussion.
The back-and-force reflects the importance and complexity around this
topic. Glad to see we finally reached consensus.

Best,

Xintong



On Wed, Jul 26, 2023 at 12:42 AM Jing Ge  wrote:

> Thanks Leonard for driving it. We are now on the same page.
>
> Best regards,
> Jing
>
> On Tue, Jul 25, 2023 at 9:19 PM Leonard Xu  wrote:
>
>> We’ve detailed offline discussions with @Alexander and @Jingsong, about
>> “Remove SourceFunction” item, we’ve reached a consensus as following:
>>
>> 1. Deprecate SourceFunction in 1.18 and implement following improvement
>> subtasks of FLINK-28045[1] later is reasonable for all of us.
>>
>> 2. Deleting SourceFunction API depends on future’s work progress, thus
>> “Remove SourceFunction APIs” should be a nice to have item. Alexander has
>> volunteered to take these subtasks and would try to finish them next,
>> thanks again.
>>
>> 3. As a nice to have item, and its READY status depends on  future’s work
>> progress,  this won't block release 2.0 must-have item vote.
>>
>> Thanks again @Alexander, @Jingsong  and @Xintong for driving these things
>> forward.
>>
>> Also CC RMs for 1.18 @QingSheng @Jing @Martijn @Konstantin, I’ve
>> communicated with Alexander and would like to help review the deprecation
>> PR again.
>>
>> Best,
>> Leonard
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-28045
>>
>>
>> On Jul 21, 2023, at 6:09 PM, Chesnay Schepler  wrote:
>>
>> On 21/07/2023 11:45, Leonard Xu wrote:
>>
>> In this way, the user will see the deprecated API firstly but they can
>> not find a candidate if we can not finish all tasks in one minor version .
>>
>>
>> i'm not convinced that this matters. There will be a whole bunch of APIs
>> deprecated in 1.18 (that will remain in 1.x!) without a replacement so we
>> can remove them in 2.0.
>> We already accepted this scenario.
>>
>>
>>

[jira] [Created] (FLINK-32674) Add documentation for the new Context.getTargetColumns

[lincoln lee (Jira)]

lincoln lee created FLINK-32674:
---

 Summary: Add documentation for the new Context.getTargetColumns
 Key: FLINK-32674
 URL: https://issues.apache.org/jira/browse/FLINK-32674
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Affects Versions: 1.18.0
Reporter: lincoln lee
Assignee: lincoln lee
 Fix For: 1.18.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang

[Danny Cranmer]

Congrats and welcome!

Danny.

On Tue, 25 Jul 2023, 16:48 Matthias Pohl, 
wrote:

> Congratulations :)
>
> On Tue, Jul 25, 2023 at 5:13 PM Jing Ge 
> wrote:
>
> > Congrats, Yong Fang!
> >
> > Best regards,
> > Jing
> >
> > On Tue, Jul 25, 2023 at 7:35 PM Yu Li  wrote:
> >
> > > Congrats, Yong!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Tue, 25 Jul 2023 at 18:03, Sergey Nuyanzin 
> > wrote:
> > >
> > > > Congratulations, Yong Fang!
> > > >
> > > > On Tue, Jul 25, 2023 at 7:53 AM ConradJam 
> wrote:
> > > >
> > > > > Congratulations, Yong Fang
> > > > >
> > > > > Mang Zhang  于2023年7月25日周二 12:08写道：
> > > > >
> > > > > > Congratulations, Yong Fang!
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Best regards,
> > > > > > Mang Zhang
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > 在 2023-07-25 10:30:24，"Jark Wu"  写道：
> > > > > > >Congratulations, Yong Fang!
> > > > > > >
> > > > > > >Best,
> > > > > > >Jark
> > > > > > >
> > > > > > >On Mon, 24 Jul 2023 at 22:11, Wencong Liu  >
> > > > wrote:
> > > > > > >
> > > > > > >> Congratulations!
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Wencong Liu
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> 在 2023-07-24 11:03:30，"Paul Lam"  写道：
> > > > > > >> >Congrats, Shammon!
> > > > > > >> >
> > > > > > >> >Best,
> > > > > > >> >Paul Lam
> > > > > > >> >
> > > > > > >> >> 2023年7月24日 10:56，Jingsong Li  写道：
> > > > > > >> >>
> > > > > > >> >> Shammon
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best
> > > > >
> > > > > ConradJam
> > > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Sergey
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-326: Enhance Watermark to Support Processing-Time Temporal Join

[David Anderson]

Dong,

Thank you for the careful analysis of my proposal. Your conclusions
make sense to me.

David

On Mon, Jul 24, 2023 at 8:37 PM Dong Lin  wrote:
>
> Hi David,
>
> Thank you for the detailed comments and the suggestion of this alternative 
> approach.
>
> I agree with you that this alternative can also address the target use-case 
> with the same correctness. In comparison to the current FLIP, this 
> alternative indeed introduces much less complexity to the Flink runtime 
> internal implementation.
>
> At a high level, this alternative is simulating a one-time emission of 
> Watermark(useProcessingTime=true) with periodic emission of 
> Watermark(timestamp=wall-lock-time).
>
> One downside of this alternative is that it can introduce a bit of extra 
> per-record runtime overhead. This is because the ingestion time watermark 
> will be emitted periodically according to pipeline.auto-watermark-interval 
> (200 ms by default). Thus there is still a short period where the watermark 
> from the HybridSource can be lagging behind wall-clock time. For operators 
> whose logic depends on the watermark, such as TemporalRowTimeJoinOperator, 
> they will need to check build-side watermark and delay/buffer records on the 
> probe-side until it receives the next ingestion-time watermark.
>
> The impact of this overhead probably depends on the throughput/watermark of 
> the probe-side records. On the other hand, given that join operator is 
> typically already heavy (due to state backend access and build-side buffer), 
> and the watermark from probe-side (e.g. Kafka) is probably also lagging 
> behind wall-clock time, it is probably not an issue in most cases. Therefore 
> I agree that it is worth trying this approach. We can revisit this issue if 
> we any issues around performance or usability of this approach.
>
> Another potential concern is that it requires the user to use ingestion time. 
> I am not sure we are able to do this in a backward-compatible way yet. We 
> probably need to go through the existing APIs around ingestion time watermark 
> to validate this.
>
> BTW, with the introduction of RecordAttributes(isBacklog=true/false) from 
> FLIP-327, another short-term approach is to let 
> TemporalProcessTimeJoinOperator keep buffering records from 
> MySQL/HybridSource as long as isBacklog=true, and process them in a 
> processing-time manner once it receives isBacklog=false. This should also 
> address the use-case targeted by FLIP-326. The only caveat with this approach 
> is that it is a bit hacky, because it requires JoinOpertor to always buffer 
> records when isBacklog=true, whereas isBacklog's semantics only says it is 
> "optional" to buffer records, which can be an issue in the long term.
>
> Thanks,
> Dong
>
> On Tue, Jul 25, 2023 at 2:37 AM David Anderson  wrote:
>>
>> I'm delighted to see interest in developing support for
>> processing-time temporal joins.
>>
>> The proposed implementation seems rather complex, and I'm not
>> convinced this complexity is justified/necessary. I'd like to outline
>> a simpler alternative that I think would satisfy the key objectives.
>>
>> Key ideas:
>>
>> 1. Limit support to the HybridSource (or a derivative thereof). (E.g.,
>> I'm guessing the MySQL CDC Source could be reworked to be a hybrid
>> source.)
>> 2. Have this HybridSource wait to begin emitting watermarks until it
>> has handled all events from the bounded sources. (I'm not sure how the
>> HybridSource handles this now; if this is an incompatible change, we
>> can find a way to deal with that.)
>> 3. Instruct users to use an ingestion time watermarking strategy for
>> their unbounded source (the source the HybridSource handles last) if
>> they want to do something like a processing time temporal join.
>>
>> One objection to this is the limitation of only supporting the
>> HybridSource -- what about cases where the user has a single source,
>> e.g., a Kafka topic? I'm suggesting the user would divide their
>> build-side stream into two parts -- a bounded component that is fully
>> ingested by the hybrid source before watermarking begins, followed by
>> an unbounded component.
>>
>> I think this alternative handles use cases like processing-time
>> temporal join rather nicely, without requiring any changes to
>> watermarks or the core runtime.
>>
>> David
>>
>> On Thu, Jun 29, 2023 at 1:39 AM Martijn Visser  
>> wrote:
>> >
>> > Hi Dong and Xuannan,
>> >
>> > I'm excited to see this FLIP. I think support for processing-time
>> > temporal joins is something that the Flink users will greatly benefit
>> > off. I specifically want to call-out that it's great to see the use
>> > cases that this enables. From a technical implementation perspective,
>> > I defer to the opinion of others with expertise on this topic.
>> >
>> > Best regards,
>> >
>> > Martijn
>> >
>> > On Sun, Jun 25, 2023 at 9:03 AM Xuannan Su  wrote:
>> > >
>> > > Hi all,
>> > >
>> > > Dong(cc'ed) and I are opening this thread to discuss 

Re: [DISCUSS] Add missing visibility annotation for Table APIs

[Jing Ge]

Hi Jane,

Thanks for your effort of walking through all classes and compiling the
sheet. It is quite helpful. Just out of curiosity, do we really need to
mark some many classes as @Internal? What is the exactly different between
a public class with no annotation and with the @Internal?

Best regards,
Jing


On Tue, Jul 25, 2023 at 11:06 AM Lincoln Lee  wrote:

> Hi Jane,
>
> Thanks for driving this! Overall the proposed annotations looks good to me.
> Some comments for the table[1]:
>
> For the `DynamicFilteringEvent`, tend to keep it 'internal' since it's a
> concreate implementation of `SourceEvent`(and other two implementers are
> not public ones) .
>
> For the `LookupOptions` and `Trigger`s, because they're all in the public
> interfaces of the flip[2], I'm fine with making them all public ones or
> just excluding the Trigger implemantors, cc @Qingsheng can you also help to
> check this?
>
> For the `BuiltInFunctionDefinitions$Builder`, I think it should be
> `BuiltInFunctionDefinition$Builder`.
>
>
> [1].
>
> https://docs.google.com/spreadsheets/d/1e8M0tUtKkZXEd8rCZtZ0C6Ty9QkNaPySsrCgz0vEID4/edit?usp=sharing
> [2].
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-221%3A+Abstraction+for+lookup+source+cache+and+metric#FLIP221:Abstractionforlookupsourcecacheandmetric-PublicInterfaces
>
> Best,
> Lincoln Lee
>
> Jark Wu  于2023年7月25日周二 10:47写道：
>
> > Hi Jane,
> >
> > Thanks for kicking off this work and collecting the detailed list.
> >
> > +1 to add the missing annotation.
> >
> > This often confuses me whether the class can be modified without breaking
> > the compatibility
> >  when looking at classes in table-common and table-api. Explicitly mark
> the
> > visibility can be
> > helpful in this case.
> >
> > I have some additional suggestions wrt the class annotations:
> > - classes in org.apache.flink.table.catalog.stats.* can be
> @PublicEvolving,
> > because all the classes in there are needed to build the @PublicEvolving
> >  CatalogColumnStatistics.
> > - PeriodicCacheReloadTrigger and TimedCacheReloadTrigger can be
> > @PublicEvolving,
> > they are built-in implementations of cache reload trigger and are exposed
> > to connectors.
> > - CoreModule can be @PublicEvolving to allow users use it in
> > TableEnv#loadModule(name, Module).
> >
> > Best,
> > Jark
> >
> >
> > On Fri, 21 Jul 2023 at 00:34, Jane Chan  wrote:
> >
> > > Hi, Devs,
> > >
> > > I would like to start a discussion on adding missing visibility
> > annotation
> > > (PublicEvolving / Internal etc.) for Table APIs.
> > >
> > > The motivation for starting this discussion was during the cleanup of
> > which
> > > Table API to be deprecated for version 2.0, I noticed that some of the
> > APIs
> > > lack visibility annotations, which may lead to users relying on APIs
> that
> > > should have been marked as internal.
> > >
> > > Therefore, I have compiled a sheet[1] listing the currently unmarked
> > > classes under the table-api-java, table-api-java-bridge, and
> table-common
> > > modules and the recommended annotations to be added.
> > >
> > > My thought is that all public classes/interfaces within the three
> modules
> > > mentioned above should be explicitly marked, and we might consider
> > > introducing a new architectural rule to perform auto-check on newly
> added
> > > APIs in the future.
> > >
> > > Let me explain the details.
> > >
> > > 1. Why table-api-java, table-api-java-bridge, and table-common?
> > > Because according to Flink's application project configuration doc[2],
> > > table-api-java and table-api-java-bridge are the leading dependencies
> for
> > > users to develop a table program. Although flink-table-common is not
> > > listed, it is the core dependency for users to implement a User-Defined
> > > Function/Connector[3].
> > >
> > > 2. How are the classes listed in this table compiled?
> > > I use a customized Intellij profile to perform the code inspection
> under
> > > these three modules to find all public classes/interfaces without API
> > > visibility annotations, along with a manual check.
> > >
> > > 3. How is the suggested API visibility to be determined?
> > > For all APIs suggested as PublicEvolving, I added a comment on the cell
> > to
> > > explain the reason. The rest APIs, which are indicated as Internal, are
> > > either util classes or implementations.
> > >
> > > I'm looking forward to your ideas, and it would be great if any
> > interested
> > > developers could help review this list together.
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://docs.google.com/spreadsheets/d/1e8M0tUtKkZXEd8rCZtZ0C6Ty9QkNaPySsrCgz0vEID4/edit?usp=sharing
> > > [2]
> > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/overview/#which-dependencies-do-you-need
> > > [3]
> > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/sourcessinks/#project-configuration
> > >
> > >
> > > Best regards,
> > > Jane
> > >
> >
>

Re: [VOTE] Release 2.0 must-have work items

[Jing Ge]

Thanks Leonard for driving it. We are now on the same page.

Best regards,
Jing

On Tue, Jul 25, 2023 at 9:19 PM Leonard Xu  wrote:

> We’ve detailed offline discussions with @Alexander and @Jingsong, about
> “Remove SourceFunction” item, we’ve reached a consensus as following:
>
> 1. Deprecate SourceFunction in 1.18 and implement following improvement
> subtasks of FLINK-28045[1] later is reasonable for all of us.
>
> 2. Deleting SourceFunction API depends on future’s work progress, thus
> “Remove SourceFunction APIs” should be a nice to have item. Alexander has
> volunteered to take these subtasks and would try to finish them next,
> thanks again.
>
> 3. As a nice to have item, and its READY status depends on  future’s work
> progress,  this won't block release 2.0 must-have item vote.
>
> Thanks again @Alexander, @Jingsong  and @Xintong for driving these things
> forward.
>
> Also CC RMs for 1.18 @QingSheng @Jing @Martijn @Konstantin, I’ve
> communicated with Alexander and would like to help review the deprecation
> PR again.
>
> Best,
> Leonard
>
> [1] https://issues.apache.org/jira/browse/FLINK-28045
>
>
> On Jul 21, 2023, at 6:09 PM, Chesnay Schepler  wrote:
>
> On 21/07/2023 11:45, Leonard Xu wrote:
>
> In this way, the user will see the deprecated API firstly but they can not
> find a candidate if we can not finish all tasks in one minor version .
>
>
> i'm not convinced that this matters. There will be a whole bunch of APIs
> deprecated in 1.18 (that will remain in 1.x!) without a replacement so we
> can remove them in 2.0.
> We already accepted this scenario.
>
>
>

Re: [DISCUSS] Proposing an LTS Release for the 1.x Line

[Jing Ge]

Hi Konstantin,

I might have not made myself clear enough, apologies. The
source-/sink-function was used as a concrete example to discuss the pattern
before we decided to offer LTS. The intention was not to hijack this thread
to discuss how to deprecate them.

We all wish that the only thing users need to migrate from Flink 1.x to 2.0
is some code changes in their repos and we all wish users will migrate, if
LTS has long enough support time. But the question I tried to discuss is
not the wish but the "How?". We might be able to toss the high migration
effort aside(we shouldn't), since it is theoretically still doable if users
have long enough time, even if the effort is extremely high. Another
concern is that if "function regressions" is allowed in 2.0, i.e. if 2.0
has a lack of functionalities or bugs compared to 1.x, there will be no way
for users to do the migration regardless of whether we encourage them to
migrate or they haven been given enough time(how long is enough?) because
LTS has been offered. How could we help users and avoid this happening?

Best regards,
Jing

On Tue, Jul 25, 2023 at 6:57 PM Konstantin Knauf  wrote:

> Hi Jing,
>
> let's not overindex on the Source-/SinkFunction discussion in this thread.
>
> We will generally drop/break a lot of APIs in Flink 2.0. So, naturally
> users will need to make more changes to their code in order to migrate from
> 1.x to Flink 2.0. In order to give them more time to do this, we support
> the last Flink 1.x release for a longer time with bug fix releases.
>
> Of course, we still encourage users to migrate to Flink 2.0, because at
> some point, we will stop support Flink 1.x. For example, if we followed
> Marton's proposal we would support Flink 1.x LTS for about 2 years (roughly
> 4 minor release cycles) instead of about 1 year (2 minor release cycles)
> for regular minor releases. This seems like a reasonable timeframe to me.
> It also gives us more time to discover and address blockers in migrating to
> Flink 2.x that we are not aware of right now.
>
> Best,
>
> Konstantin
>
> Am Di., 25. Juli 2023 um 12:48 Uhr schrieb Jing Ge
> :
>
> > Hi all,
> >
> > Overall, it is a good idea to provide the LTS release, but I'd like to
> > reference a concrete case as an example to understand what restrictions
> the
> > LTS should have.
> >
> > Hypothetically, Source-/Sink- Function have been deprecated in 1.x LTS
> and
> > removed in 2.0 and the issues[1] are not solved in 2.0. This is a typical
> > scenario that the old APIs are widely used in 1.x LTS and the new APIs in
> > 2.0 are not ready yet to take over all users. We will have the following
> > questions:
> >
> > 1. Is this scenario allowed at all? Do we all agree that there could be
> > some features/functionalities that only work in 1.x LTS after 2.0 has
> been
> > released?
> > 2. How long are we going to support 1.x LTS? 1 year? 2 years? As long as
> > the issues that block users from migrating to 2.0 are not solved, we
> can't
> > stop the LTS support, even if the predefined support time expires.
> > 3. What is the intention to release a new version with (or without) LTS?
> Do
> > we still want to engage users to migrate to the new release asap? If  the
> > old APIs 1.x LTS offer more than the new APIs in 2.0 or it is almost
> > impossible to migrate, double effort will be required to maintain those
> > major releases for a very long time. We will be facing many cohorts.
> >
> > IMHO, we should be clear with those questions before we start talking
> about
> > LTS. WDYT?
> >
> > Best regards,
> > Jing
> >
> >
> > [1] https://lists.apache.org/thread/734zhkvs59w2o4d1rsnozr1bfqlr6rgm
> >
> > On Tue, Jul 25, 2023 at 6:08 PM Márton Balassi  >
> > wrote:
> >
> > > Hi team,
> > >
> > > +1 for supporting the last 1.x for a longer than usual period of time
> and
> > > limiting it to bugfixes. I would suggest supporting it for double the
> > usual
> > > amount of time (4 minor releases).
> > >
> > > On Tue, Jul 25, 2023 at 9:25 AM Konstantin Knauf 
> > > wrote:
> > >
> > > > Hi Alex,
> > > >
> > > > yes, I think, it makes sense to support the last 1.x release longer
> > than
> > > > usual. This should be limited to bugfixes in my opinion.
> > > >
> > > > Best,
> > > >
> > > > Konstantin
> > > >
> > > > Am Di., 25. Juli 2023 um 07:07 Uhr schrieb Xintong Song <
> > > > tonysong...@gmail.com>:
> > > >
> > > > > Hi Alex,
> > > > >
> > > > > Providing a longer supporting period for the last 1.x minor release
> > > makes
> > > > > sense to me.
> > > > >
> > > > > I think we need to be more specific about what LTS means here.
> > > > >
> > > > >- IIUC, that means for the last 1.x minor release, we will keep
> > > > >providing 1.x.y / 1.x.z bugfix release. This is a stronger
> support
> > > > > compared
> > > > >to regular minor releases which by default are only supported
> for
> > 2
> > > > > minor
> > > > >release cycles.
> > > > >- Do we only provide bug fixes for the LTS release, or do we
> 

Re: [DISCUSS] FLIP-348: Support System Columns in SQL and Table API

[Jing Ge]

Hi Timo,

Thanks for your proposal. It is a very pragmatic feature. Among all options
in the FLIP, option 3 is one I prefer too and I'd like to ask some
questions to understand your thoughts.

1. I did some research on pseudo columns, just out of curiosity, do you
know why most SQL systems do not need any prefix with their pseudo column?
2. Some platform providers will use ${variable_name} to define their own
configurations and allow them to be embedded into SQL scripts. Will there
be any conflict with option 3?

Best regards,
Jing

On Tue, Jul 25, 2023 at 7:00 PM Konstantin Knauf  wrote:

> Hi Timo,
>
> this makes sense to me. Option 3 seems reasonable, too.
>
> Cheers,
>
> Konstantin
>
> Am Di., 25. Juli 2023 um 12:53 Uhr schrieb Timo Walther <
> twal...@apache.org
> >:
>
> > Hi everyone,
> >
> > I would like to start a discussion about introducing the concept of
> > "System Columns" in SQL and Table API.
> >
> > The subject sounds bigger than it actually is. Luckily, Flink SQL
> > already exposes the concept of metadata columns. And this proposal is
> > just a slight adjustment for how metadata columns can be used as system
> > columns.
> >
> > The biggest problem of metadata columns currently is that a catalog
> > implementation can't provide them by default because they would affect
> > `SELECT *` when adding another one.
> >
> > Looking forward to your feedback on FLIP-348:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-348%3A+Support+System+Columns+in+SQL+and+Table+API
> >
> > Thanks,
> > Timo
> >
>
>
> --
> https://twitter.com/snntrable
> https://github.com/knaufk
>

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang

[Matthias Pohl]

Congratulations :)

On Tue, Jul 25, 2023 at 5:13 PM Jing Ge  wrote:

> Congrats, Yong Fang!
>
> Best regards,
> Jing
>
> On Tue, Jul 25, 2023 at 7:35 PM Yu Li  wrote:
>
> > Congrats, Yong!
> >
> > Best Regards,
> > Yu
> >
> >
> > On Tue, 25 Jul 2023 at 18:03, Sergey Nuyanzin 
> wrote:
> >
> > > Congratulations, Yong Fang!
> > >
> > > On Tue, Jul 25, 2023 at 7:53 AM ConradJam  wrote:
> > >
> > > > Congratulations, Yong Fang
> > > >
> > > > Mang Zhang  于2023年7月25日周二 12:08写道：
> > > >
> > > > > Congratulations, Yong Fang!
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best regards,
> > > > > Mang Zhang
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 在 2023-07-25 10:30:24，"Jark Wu"  写道：
> > > > > >Congratulations, Yong Fang!
> > > > > >
> > > > > >Best,
> > > > > >Jark
> > > > > >
> > > > > >On Mon, 24 Jul 2023 at 22:11, Wencong Liu 
> > > wrote:
> > > > > >
> > > > > >> Congratulations!
> > > > > >>
> > > > > >> Best,
> > > > > >> Wencong Liu
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> 在 2023-07-24 11:03:30，"Paul Lam"  写道：
> > > > > >> >Congrats, Shammon!
> > > > > >> >
> > > > > >> >Best,
> > > > > >> >Paul Lam
> > > > > >> >
> > > > > >> >> 2023年7月24日 10:56，Jingsong Li  写道：
> > > > > >> >>
> > > > > >> >> Shammon
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > > >
> > > > --
> > > > Best
> > > >
> > > > ConradJam
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Sergey
> > >
> >
>

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang

[Jing Ge]

Congrats, Yong Fang!

Best regards,
Jing

On Tue, Jul 25, 2023 at 7:35 PM Yu Li  wrote:

> Congrats, Yong!
>
> Best Regards,
> Yu
>
>
> On Tue, 25 Jul 2023 at 18:03, Sergey Nuyanzin  wrote:
>
> > Congratulations, Yong Fang!
> >
> > On Tue, Jul 25, 2023 at 7:53 AM ConradJam  wrote:
> >
> > > Congratulations, Yong Fang
> > >
> > > Mang Zhang  于2023年7月25日周二 12:08写道：
> > >
> > > > Congratulations, Yong Fang!
> > > >
> > > >
> > > > --
> > > >
> > > > Best regards,
> > > > Mang Zhang
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 在 2023-07-25 10:30:24，"Jark Wu"  写道：
> > > > >Congratulations, Yong Fang!
> > > > >
> > > > >Best,
> > > > >Jark
> > > > >
> > > > >On Mon, 24 Jul 2023 at 22:11, Wencong Liu 
> > wrote:
> > > > >
> > > > >> Congratulations!
> > > > >>
> > > > >> Best,
> > > > >> Wencong Liu
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> 在 2023-07-24 11:03:30，"Paul Lam"  写道：
> > > > >> >Congrats, Shammon!
> > > > >> >
> > > > >> >Best,
> > > > >> >Paul Lam
> > > > >> >
> > > > >> >> 2023年7月24日 10:56，Jingsong Li  写道：
> > > > >> >>
> > > > >> >> Shammon
> > > > >> >
> > > > >>
> > > >
> > >
> > >
> > > --
> > > Best
> > >
> > > ConradJam
> > >
> >
> >
> > --
> > Best regards,
> > Sergey
> >
>

[jira] [Created] (FLINK-32673) Migrage Google PubSub connector to V2

[Alexander Fedulov (Jira)]

Alexander Fedulov created FLINK-32673:
-

 Summary: Migrage Google PubSub connector to V2
 Key: FLINK-32673
 URL: https://issues.apache.org/jira/browse/FLINK-32673
 Project: Flink
  Issue Type: Sub-task
Reporter: Alexander Fedulov






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32672) Migrate RabbitMQ connector to Source V2 API

[Alexander Fedulov (Jira)]

Alexander Fedulov created FLINK-32672:
-

 Summary: Migrate RabbitMQ connector to Source V2 API
 Key: FLINK-32672
 URL: https://issues.apache.org/jira/browse/FLINK-32672
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors/ RabbitMQ
Reporter: Alexander Fedulov






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release 2.0 must-have work items

[Leonard Xu]

We’ve detailed offline discussions with @Alexander and @Jingsong, about “Remove 
SourceFunction” item, we’ve reached a consensus as following:

1. Deprecate SourceFunction in 1.18 and implement following improvement 
subtasks of FLINK-28045[1] later is reasonable for all of us.

2. Deleting SourceFunction API depends on future’s work progress, thus “Remove 
SourceFunction APIs” should be a nice to have item. Alexander has volunteered 
to take these subtasks and would try to finish them next, thanks again. 

3. As a nice to have item, and its READY status depends on  future’s work 
progress,  this won't block release 2.0 must-have item vote.

Thanks again @Alexander, @Jingsong  and @Xintong for driving these things 
forward. 

Also CC RMs for 1.18 @QingSheng @Jing @Martijn @Konstantin, I’ve communicated 
with Alexander and would like to help review the deprecation PR again.

Best,
Leonard

[1] https://issues.apache.org/jira/browse/FLINK-28045 


> On Jul 21, 2023, at 6:09 PM, Chesnay Schepler  wrote:
> 
> On 21/07/2023 11:45, Leonard Xu wrote:
>> In this way, the user will see the deprecated API firstly but they can not 
>> find a candidate if we can not finish all tasks in one minor version .
> 
> i'm not convinced that this matters. There will be a whole bunch of APIs 
> deprecated in 1.18 (that will remain in 1.x!) without a replacement so we can 
> remove them in 2.0.
> We already accepted this scenario.

Re: Flink suspected BUG

[Matthias Pohl]

Hi liuwb9,
thanks for reporting this issue. Please keep a few things in mind, though:
- The community only supports the two most-recently released versions of
Flink. Your version 1.14 is too old (issues might have been fixed already
in newer releases). Trying to reproduce the behavior in a supported Flink
version is appreciated.
- The dev mailing list is usually used for development/architectural
discussions (user questions could be asked in the user mailing list) [1]
- The description of the issue sounds a bit vague. Is your conclusion of a
bug based on the fact that the submission took longer than expected? That
information is not enough to start an investigation. Usually, providing
Flink logs helps to support your case and makes it easier for others to
analyze what's going on.
- Jira [2] might be the best channel if you are certain that the behavior
you observed indicates a bug. Additional artifacts (like logs) can be
attached to your bug report there as well.

Thanks,
Matthias

[1] https://flink.apache.org/community/#mailing-lists
[2] https://flink.apache.org/community/#issue-tracker

On Thu, Jul 20, 2023 at 11:27 AM liu...@midea.com  wrote:

>
> Hi all,
>
> I encountered a suspected bug in Flink.
> A  flink job associates with 30+ HBase2.2 asynchronous dimension 
> tables（Temporal
> Join AND lookup.async = 'true'),
> When I submitted this job, it took a very long time to submit to the YARN
> cluster.
> During the generation of the job graph and optimization of the operator
> chain, recursion occurs in the process of locating and discovering.
>
> Flink source code location: version 1.14
>
>
> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator#areOperatorsChainable
>
>
> liu...@midea.com
>

Re: [DISCUSSION] Revival of FLIP-154 (SQL Implicit Type Coercion)

[Benchao Li]

Hi Sergey,

There is a discussion[1] for this FLIP before.

I like this feature, thanks for driving this. I hope it does not need to
much customization (if there is any improvement which would benefit both
Calcite and Flink, it would be great to push those also to upstream
project, and I would like to help on that)

[1] https://lists.apache.org/thread/mglop6mgd36rrbjt7ojyd5sj8tyoc3c2

Sergey Nuyanzin  于2023年7月19日周三 06:18写道：

> Hello everyone
>
> I'd like to revive FLIP-154[1] a bit.
>
> I failed with finding any discussion/vote thread about it (please point me
> to that if it is present somewhere)
>
> Also FLIP itself looks abandoned (no activity for a couple of years),
> however it seems to be useful.
>
> I did a bit investigation about that
>
> From one side Calcite provides its own coercion... However, sometimes it
> behaves strangely and is not ready to use in Flink.
> for instance
> 1) it can not handle cases with `with` subqueries and fails with NPE (even
> if it's fixed it will come not earlier than with 1.36.0)
> 2) under the hood it uses hardcoded `cast`, especially it is an issue for
> equals where `cast` is invoked without fallback to `null`. In addition it
> tries to cast `string` to `date`. All this leads to failure for such
> queries like `select uuid() = null;` where it tries to cast the result of
> `uuid()` to date without a fallback option.
>
> The good thing is that Calcite also provides a custom TypeCoercionFactory
> which could be used during coercion (in case it is enabled). This could
> allow for instance to use `try_cast` instead of `cast`, embed the fix for
> aforementioned NPE, enable only required coercion rules. Also it will
> enable coercions rule by rule instead of big bang.
>
> I would volunteer to update the FLIP page with usage of a custom factory if
> there are no objections and come back with a discussion thread to revive
> the work on it.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-154%3A+SQL+Implicit+Type+Coercion
> --
> Best regards,
> Sergey
>


-- 

Best,
Benchao Li

Re: [DISCUSS] Connectors, Formats, and even User Code should also be pluggable.

[Benchao Li]

I agree with Jing that the current doc is quite preliminary, and needs to
be turned into a FLIP.

I'll be very interested in this FLIP, and looking forward to it. We've
suffered from reshading/relocating heavy connector dependencies for a long
time.

Besides, we've implemented a plugin mechanism to load hive udf in our batch
SQL scenario internally, it saves us a lot of effort to handle the
dependency conflicts. The most challenging part would be the
(de)serialization of those classes loaded through plugins according to our
experience. (I noticed you have already considered this)

Jing Ge  于2023年7月21日周五 06:44写道：

> Hi Zhiqiang,
>
> Thanks for your proposal. The idea is very interesting! Since almost all
> connectors are or will be externalized, the pluggable design you suggested
> could help reduce the development effort.
>
> As you mentioned, the attached doc contains only your preliminary idea. I
> would suggest that you could turn it into a FLIP with more details and do
> the followings:
>
> 1. Describe a real conflict case that will benefit from the pluggable
> design
> 2. Describe where you want to modify the code, or provide a POC branch
> 3. Guideline to tell users how to use it, i.e. where (the plugins dir)
> should the connector jar be located, how does it look like with an example,
> etc.
>
> WDYT?
>
> Best regards,
> Jing
>
>
> On Fri, Jul 14, 2023 at 3:00 PM zhiqiang li 
> wrote:
>
> > Hi devs,
> >
> > I have observed that in [1], connectors and formats are pluggable,
> > allowing user code to be easily integrated. The advantages of having
> > pluggable connectors are evident, as it helps avoid conflicts between
> > different versions of jar packages. If classloader isolation is not used,
> > shading becomes necessary to resolve conflicts, resulting in a
> significant
> > waste of development time. I understand that implementing this change may
> > require numerous API modifications, so I would like to discuss in this
> > email.
> >
> > > Plugins cannot access classes from other plugins or from Flink that
> have
> > not been specifically whitelisted.
> > > This strict isolation allows plugins to contain conflicting versions of
> > the same library without the need to relocate classes or to converge to
> > common versions.
> > > Currently, file systems and metric reporters are pluggable but, in the
> > future, connectors, formats, and even user code should also be pluggable.
> >
> > [2] It is my preliminary idea.
> >
> > [1]
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/overview/
> > [2]
> >
> https://docs.google.com/document/d/1XP2fBpcntK0YIdQ_Ax7JV2MhNdebvkFxSiNJRp6WQ24/edit?usp=sharing
> >
> >
> > Best,
> > Zhiqiang
> >
> >
>


-- 

Best,
Benchao Li

Re: [VOTE] Release 2.0 must-have work items - Round 2

[Konstantin Knauf]

I assume this vote includes a decision to not removing
SourceFunction/SinkFunction in Flink 2.0 (as it has been removed from the
table). If this is the case, I don't think, this discussion has concluded.
There are multiple contributors like myself, Martijn, Alex Fedulov and
Maximilian Michels, who have indicated they would be in favor of
deprecating/dropping them. This Source/Sink Function discussion seems to go
in circles in general. I am wondering if it makes sense to have a call
about this instead of repeating mailing list discussions.

Am Di., 25. Juli 2023 um 13:38 Uhr schrieb Yu Li :

> +1 (binding)
>
> Thanks for driving this, Xintong!
>
> Best Regards,
> Yu
>
>
> On Sun, 23 Jul 2023 at 18:28, Yuan Mei  wrote:
>
> > +1 (binding)
> >
> > Thanks for driving the discussion through and for all the efforts in
> > resolving the complexities :-)
> >
> > Best
> > Yuan
> >
> > On Thu, Jul 20, 2023 at 5:23 PM Xintong Song 
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to start another round of VOTE for the must-have work items
> for
> > > release 2.0 [1]. The corresponding discussion thread is [2], and the
> > > previous voting thread is [3]. All comments from the previous voting
> > thread
> > > have been addressed.
> > >
> > > Please note that once the vote is approved, any changes to the
> must-have
> > > items (adding / removing must-have items, changing the priority)
> requires
> > > another vote. Assigning contributors / reviewers, updating
> descriptions /
> > > progress, changes to nice-to-have items do not require another vote.
> > >
> > > The vote will be open until at least July 25, following the consensus
> > > voting process. Votes of PMC members are binding.
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > > [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> > >
> > > [2] https://lists.apache.org/thread/l3dkdypyrovd3txzodn07lgdwtwvhgk4
> > >
> > > [3] https://lists.apache.org/thread/r0y9syc6k5nmcxvnd0hj33htdpdj9k6m
> > >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

[jira] [Created] (FLINK-32671) Document Externalized Declarative Resource Management

[Konstantin Knauf (Jira)]

Konstantin Knauf created FLINK-32671:


 Summary: Document Externalized Declarative Resource Management
 Key: FLINK-32671
 URL: https://issues.apache.org/jira/browse/FLINK-32671
 Project: Flink
  Issue Type: Sub-task
Reporter: Konstantin Knauf






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release 2.0 must-have work items - Round 2

[Yu Li]

+1 (binding)

Thanks for driving this, Xintong!

Best Regards,
Yu


On Sun, 23 Jul 2023 at 18:28, Yuan Mei  wrote:

> +1 (binding)
>
> Thanks for driving the discussion through and for all the efforts in
> resolving the complexities :-)
>
> Best
> Yuan
>
> On Thu, Jul 20, 2023 at 5:23 PM Xintong Song 
> wrote:
>
> > Hi all,
> >
> > I'd like to start another round of VOTE for the must-have work items for
> > release 2.0 [1]. The corresponding discussion thread is [2], and the
> > previous voting thread is [3]. All comments from the previous voting
> thread
> > have been addressed.
> >
> > Please note that once the vote is approved, any changes to the must-have
> > items (adding / removing must-have items, changing the priority) requires
> > another vote. Assigning contributors / reviewers, updating descriptions /
> > progress, changes to nice-to-have items do not require another vote.
> >
> > The vote will be open until at least July 25, following the consensus
> > voting process. Votes of PMC members are binding.
> >
> > Best,
> >
> > Xintong
> >
> >
> > [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> >
> > [2] https://lists.apache.org/thread/l3dkdypyrovd3txzodn07lgdwtwvhgk4
> >
> > [3] https://lists.apache.org/thread/r0y9syc6k5nmcxvnd0hj33htdpdj9k6m
> >
>

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang

[Yu Li]

Congrats, Yong!

Best Regards,
Yu


On Tue, 25 Jul 2023 at 18:03, Sergey Nuyanzin  wrote:

> Congratulations, Yong Fang!
>
> On Tue, Jul 25, 2023 at 7:53 AM ConradJam  wrote:
>
> > Congratulations, Yong Fang
> >
> > Mang Zhang  于2023年7月25日周二 12:08写道：
> >
> > > Congratulations, Yong Fang!
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Mang Zhang
> > >
> > >
> > >
> > >
> > >
> > > 在 2023-07-25 10:30:24，"Jark Wu"  写道：
> > > >Congratulations, Yong Fang!
> > > >
> > > >Best,
> > > >Jark
> > > >
> > > >On Mon, 24 Jul 2023 at 22:11, Wencong Liu 
> wrote:
> > > >
> > > >> Congratulations!
> > > >>
> > > >> Best,
> > > >> Wencong Liu
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> 在 2023-07-24 11:03:30，"Paul Lam"  写道：
> > > >> >Congrats, Shammon!
> > > >> >
> > > >> >Best,
> > > >> >Paul Lam
> > > >> >
> > > >> >> 2023年7月24日 10:56，Jingsong Li  写道：
> > > >> >>
> > > >> >> Shammon
> > > >> >
> > > >>
> > >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>
>
> --
> Best regards,
> Sergey
>

[jira] [Created] (FLINK-32670) Annotate interfaces that inherit from SourceFunction as deprecated

[Alexander Fedulov (Jira)]

Alexander Fedulov created FLINK-32670:
-

 Summary: Annotate interfaces that inherit from SourceFunction as 
deprecated 
 Key: FLINK-32670
 URL: https://issues.apache.org/jira/browse/FLINK-32670
 Project: Flink
  Issue Type: Sub-task
Reporter: Alexander Fedulov


 ProcessFunction, RichParallelSourceFunction, ExternallyInducedSource



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-348: Support System Columns in SQL and Table API

[Konstantin Knauf]

Hi Timo,

this makes sense to me. Option 3 seems reasonable, too.

Cheers,

Konstantin

Am Di., 25. Juli 2023 um 12:53 Uhr schrieb Timo Walther :

> Hi everyone,
>
> I would like to start a discussion about introducing the concept of
> "System Columns" in SQL and Table API.
>
> The subject sounds bigger than it actually is. Luckily, Flink SQL
> already exposes the concept of metadata columns. And this proposal is
> just a slight adjustment for how metadata columns can be used as system
> columns.
>
> The biggest problem of metadata columns currently is that a catalog
> implementation can't provide them by default because they would affect
> `SELECT *` when adding another one.
>
> Looking forward to your feedback on FLIP-348:
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-348%3A+Support+System+Columns+in+SQL+and+Table+API
>
> Thanks,
> Timo
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

Re: [DISCUSS] Proposing an LTS Release for the 1.x Line

[Konstantin Knauf]

Hi Jing,

let's not overindex on the Source-/SinkFunction discussion in this thread.

We will generally drop/break a lot of APIs in Flink 2.0. So, naturally
users will need to make more changes to their code in order to migrate from
1.x to Flink 2.0. In order to give them more time to do this, we support
the last Flink 1.x release for a longer time with bug fix releases.

Of course, we still encourage users to migrate to Flink 2.0, because at
some point, we will stop support Flink 1.x. For example, if we followed
Marton's proposal we would support Flink 1.x LTS for about 2 years (roughly
4 minor release cycles) instead of about 1 year (2 minor release cycles)
for regular minor releases. This seems like a reasonable timeframe to me.
It also gives us more time to discover and address blockers in migrating to
Flink 2.x that we are not aware of right now.

Best,

Konstantin

Am Di., 25. Juli 2023 um 12:48 Uhr schrieb Jing Ge
:

> Hi all,
>
> Overall, it is a good idea to provide the LTS release, but I'd like to
> reference a concrete case as an example to understand what restrictions the
> LTS should have.
>
> Hypothetically, Source-/Sink- Function have been deprecated in 1.x LTS and
> removed in 2.0 and the issues[1] are not solved in 2.0. This is a typical
> scenario that the old APIs are widely used in 1.x LTS and the new APIs in
> 2.0 are not ready yet to take over all users. We will have the following
> questions:
>
> 1. Is this scenario allowed at all? Do we all agree that there could be
> some features/functionalities that only work in 1.x LTS after 2.0 has been
> released?
> 2. How long are we going to support 1.x LTS? 1 year? 2 years? As long as
> the issues that block users from migrating to 2.0 are not solved, we can't
> stop the LTS support, even if the predefined support time expires.
> 3. What is the intention to release a new version with (or without) LTS? Do
> we still want to engage users to migrate to the new release asap? If  the
> old APIs 1.x LTS offer more than the new APIs in 2.0 or it is almost
> impossible to migrate, double effort will be required to maintain those
> major releases for a very long time. We will be facing many cohorts.
>
> IMHO, we should be clear with those questions before we start talking about
> LTS. WDYT?
>
> Best regards,
> Jing
>
>
> [1] https://lists.apache.org/thread/734zhkvs59w2o4d1rsnozr1bfqlr6rgm
>
> On Tue, Jul 25, 2023 at 6:08 PM Márton Balassi 
> wrote:
>
> > Hi team,
> >
> > +1 for supporting the last 1.x for a longer than usual period of time and
> > limiting it to bugfixes. I would suggest supporting it for double the
> usual
> > amount of time (4 minor releases).
> >
> > On Tue, Jul 25, 2023 at 9:25 AM Konstantin Knauf 
> > wrote:
> >
> > > Hi Alex,
> > >
> > > yes, I think, it makes sense to support the last 1.x release longer
> than
> > > usual. This should be limited to bugfixes in my opinion.
> > >
> > > Best,
> > >
> > > Konstantin
> > >
> > > Am Di., 25. Juli 2023 um 07:07 Uhr schrieb Xintong Song <
> > > tonysong...@gmail.com>:
> > >
> > > > Hi Alex,
> > > >
> > > > Providing a longer supporting period for the last 1.x minor release
> > makes
> > > > sense to me.
> > > >
> > > > I think we need to be more specific about what LTS means here.
> > > >
> > > >- IIUC, that means for the last 1.x minor release, we will keep
> > > >providing 1.x.y / 1.x.z bugfix release. This is a stronger support
> > > > compared
> > > >to regular minor releases which by default are only supported for
> 2
> > > > minor
> > > >release cycles.
> > > >- Do we only provide bug fixes for the LTS release, or do we also
> > > allow
> > > >backporting features to that release?
> > > >- How long exactly shall we support the LTS release?
> > > >
> > > > And maybe we can make this a general convention for last minor
> releases
> > > for
> > > > all major releases, rather than only discuss it for the 2.0 version
> > bump.
> > > >
> > > > @Leonard,
> > > >
> > > > I'd like to clarify that there are no community decisions yet on
> > release
> > > > 2.0 after 1.19. It is possible to have 1.20 before 2.0.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Tue, Jul 25, 2023 at 11:54 AM Leonard Xu 
> wrote:
> > > >
> > > > > +1, it’s pretty necessary especially we deprecated so many APIs in
> > 1.18
> > > > > and plan to remove in 2.0.
> > > > >
> > > > > The 1.19 should be a proper version for LTS Release.
> > > > >
> > > > > Best,
> > > > > Leonard
> > > > >
> > > > >
> > > > > > On Jul 25, 2023, at 3:30 AM, Alexander Fedulov <
> > > > > alexander.fedu...@gmail.com> wrote:
> > > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > Recently, there were a lot of discussions about the deprecation
> of
> > > > > various
> > > > > > APIs for the upcoming 2.0 release. It appears there are two main
> > > > > motivations
> > > > > > with opposing directions, causing these discussions to remain
> > > > unsettled.
> > > > > On
> 

[DISCUSS] FLIP-348: Support System Columns in SQL and Table API

[Timo Walther]

Hi everyone,

I would like to start a discussion about introducing the concept of 
"System Columns" in SQL and Table API.


The subject sounds bigger than it actually is. Luckily, Flink SQL 
already exposes the concept of metadata columns. And this proposal is 
just a slight adjustment for how metadata columns can be used as system 
columns.


The biggest problem of metadata columns currently is that a catalog 
implementation can't provide them by default because they would affect 
`SELECT *` when adding another one.


Looking forward to your feedback on FLIP-348:

https://cwiki.apache.org/confluence/display/FLINK/FLIP-348%3A+Support+System+Columns+in+SQL+and+Table+API

Thanks,
Timo

Re: [DISCUSS] Proposing an LTS Release for the 1.x Line

[Jing Ge]

Hi all,

Overall, it is a good idea to provide the LTS release, but I'd like to
reference a concrete case as an example to understand what restrictions the
LTS should have.

Hypothetically, Source-/Sink- Function have been deprecated in 1.x LTS and
removed in 2.0 and the issues[1] are not solved in 2.0. This is a typical
scenario that the old APIs are widely used in 1.x LTS and the new APIs in
2.0 are not ready yet to take over all users. We will have the following
questions:

1. Is this scenario allowed at all? Do we all agree that there could be
some features/functionalities that only work in 1.x LTS after 2.0 has been
released?
2. How long are we going to support 1.x LTS? 1 year? 2 years? As long as
the issues that block users from migrating to 2.0 are not solved, we can't
stop the LTS support, even if the predefined support time expires.
3. What is the intention to release a new version with (or without) LTS? Do
we still want to engage users to migrate to the new release asap? If  the
old APIs 1.x LTS offer more than the new APIs in 2.0 or it is almost
impossible to migrate, double effort will be required to maintain those
major releases for a very long time. We will be facing many cohorts.

IMHO, we should be clear with those questions before we start talking about
LTS. WDYT?

Best regards,
Jing


[1] https://lists.apache.org/thread/734zhkvs59w2o4d1rsnozr1bfqlr6rgm

On Tue, Jul 25, 2023 at 6:08 PM Márton Balassi 
wrote:

> Hi team,
>
> +1 for supporting the last 1.x for a longer than usual period of time and
> limiting it to bugfixes. I would suggest supporting it for double the usual
> amount of time (4 minor releases).
>
> On Tue, Jul 25, 2023 at 9:25 AM Konstantin Knauf 
> wrote:
>
> > Hi Alex,
> >
> > yes, I think, it makes sense to support the last 1.x release longer than
> > usual. This should be limited to bugfixes in my opinion.
> >
> > Best,
> >
> > Konstantin
> >
> > Am Di., 25. Juli 2023 um 07:07 Uhr schrieb Xintong Song <
> > tonysong...@gmail.com>:
> >
> > > Hi Alex,
> > >
> > > Providing a longer supporting period for the last 1.x minor release
> makes
> > > sense to me.
> > >
> > > I think we need to be more specific about what LTS means here.
> > >
> > >- IIUC, that means for the last 1.x minor release, we will keep
> > >providing 1.x.y / 1.x.z bugfix release. This is a stronger support
> > > compared
> > >to regular minor releases which by default are only supported for 2
> > > minor
> > >release cycles.
> > >- Do we only provide bug fixes for the LTS release, or do we also
> > allow
> > >backporting features to that release?
> > >- How long exactly shall we support the LTS release?
> > >
> > > And maybe we can make this a general convention for last minor releases
> > for
> > > all major releases, rather than only discuss it for the 2.0 version
> bump.
> > >
> > > @Leonard,
> > >
> > > I'd like to clarify that there are no community decisions yet on
> release
> > > 2.0 after 1.19. It is possible to have 1.20 before 2.0.
> > >
> > > Best,
> > >
> > > Xintong
> > >
> > >
> > >
> > > On Tue, Jul 25, 2023 at 11:54 AM Leonard Xu  wrote:
> > >
> > > > +1, it’s pretty necessary especially we deprecated so many APIs in
> 1.18
> > > > and plan to remove in 2.0.
> > > >
> > > > The 1.19 should be a proper version for LTS Release.
> > > >
> > > > Best,
> > > > Leonard
> > > >
> > > >
> > > > > On Jul 25, 2023, at 3:30 AM, Alexander Fedulov <
> > > > alexander.fedu...@gmail.com> wrote:
> > > > >
> > > > > Hello everyone,
> > > > >
> > > > > Recently, there were a lot of discussions about the deprecation of
> > > > various
> > > > > APIs for the upcoming 2.0 release. It appears there are two main
> > > > motivations
> > > > > with opposing directions, causing these discussions to remain
> > > unsettled.
> > > > On
> > > > > one hand, there's a desire to finally trim a wide range of legacy
> > APIs,
> > > > some
> > > > > lingering around since the beginning of the 1.x release line (as
> far
> > > > back as
> > > > > 2016). On the other hand, there is a commitment to uphold our
> > > guarantees
> > > > to
> > > > > the users, ensuring a smooth transition.
> > > > >
> > > > > I believe we could reconcile these two motivations. My proposition
> is
> > > to
> > > > > designate the final release of the 1.x timeline as a Long-Term
> > Support
> > > > (LTS)
> > > > > release. By doing so, we would:
> > > > >
> > > > > 1. Enable more efficient cleanup and be liberated to introduce more
> > > > breaking
> > > > >   changes, paving the way for greater innovation in the 2.0
> release.
> > > > > 2. Sustain a positive user experience by granting enough time for
> the
> > > > > changes
> > > > >   introduced in 2.0 to stabilize, allowing users to confidently
> > > > transition
> > > > >   their production code to the new release.
> > > > >
> > > > > I look forward to hearing your thoughts on this proposal.
> > > > >
> > > > > Best Regards,
> > > > > Alex
> > > >
> > > >
> > >
> >
> 

Re: [DISCUSS][2.0] FLIP-343: Remove parameter in WindowAssigner#getDefaultTrigger()

[Junrui Lee]

+1

Best,
Junrui

weijie guo  于2023年7月24日周一 10:25写道：

> +1 for this.
>
> Best regards,
>
> Weijie
>
>
> liu ron  于2023年7月24日周一 09:58写道：
>
> > +1
> >
> > Best,
> > Ron
> >
> > Yuxin Tan  于2023年7月21日周五 16:21写道：
> >
> > > +1
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Jing Ge  于2023年7月21日周五 15:41写道：
> > >
> > > > +1
> > > >
> > > > NIT: the release in the FLIP is still empty, it should be 2.0
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Fri, Jul 21, 2023 at 6:03 AM Xintong Song 
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Best,
> > > > >
> > > > > Xintong
> > > > >
> > > > >
> > > > >
> > > > > On Fri, Jul 21, 2023 at 10:53 AM Wencong Liu  >
> > > > wrote:
> > > > >
> > > > > > Hi devs,
> > > > > >
> > > > > > I would like to start a discussion on FLIP-343: Remove parameter
> in
> > > > > > WindowAssigner#getDefaultTrigger() [1].
> > > > > >
> > > > > >
> > > > > > The method getDefaultTrigger() in WindowAssigner takes a
> > > > > > StreamExecutionEnvironment
> > > > > > parameter, but this parameter is not actually used for any
> > subclasses
> > > > of
> > > > > > WindowAssigner.
> > > > > > Therefore, it is unnecessary to include this parameter.
> > > > > > As such I propose to remove the StreamExecutionEnvironment field
> > from
> > > > > > WindowAssigner#getDefaultTrigger(StreamExecutionEnvironment env).
> > > > > > Looking forward to your feedback.
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425229
> > > > > > Best regards,
> > > > > >
> > > > > >
> > > > > > Wencong Liu
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Proposing an LTS Release for the 1.x Line

[Márton Balassi]

Hi team,

+1 for supporting the last 1.x for a longer than usual period of time and
limiting it to bugfixes. I would suggest supporting it for double the usual
amount of time (4 minor releases).

On Tue, Jul 25, 2023 at 9:25 AM Konstantin Knauf  wrote:

> Hi Alex,
>
> yes, I think, it makes sense to support the last 1.x release longer than
> usual. This should be limited to bugfixes in my opinion.
>
> Best,
>
> Konstantin
>
> Am Di., 25. Juli 2023 um 07:07 Uhr schrieb Xintong Song <
> tonysong...@gmail.com>:
>
> > Hi Alex,
> >
> > Providing a longer supporting period for the last 1.x minor release makes
> > sense to me.
> >
> > I think we need to be more specific about what LTS means here.
> >
> >- IIUC, that means for the last 1.x minor release, we will keep
> >providing 1.x.y / 1.x.z bugfix release. This is a stronger support
> > compared
> >to regular minor releases which by default are only supported for 2
> > minor
> >release cycles.
> >- Do we only provide bug fixes for the LTS release, or do we also
> allow
> >backporting features to that release?
> >- How long exactly shall we support the LTS release?
> >
> > And maybe we can make this a general convention for last minor releases
> for
> > all major releases, rather than only discuss it for the 2.0 version bump.
> >
> > @Leonard,
> >
> > I'd like to clarify that there are no community decisions yet on release
> > 2.0 after 1.19. It is possible to have 1.20 before 2.0.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Tue, Jul 25, 2023 at 11:54 AM Leonard Xu  wrote:
> >
> > > +1, it’s pretty necessary especially we deprecated so many APIs in 1.18
> > > and plan to remove in 2.0.
> > >
> > > The 1.19 should be a proper version for LTS Release.
> > >
> > > Best,
> > > Leonard
> > >
> > >
> > > > On Jul 25, 2023, at 3:30 AM, Alexander Fedulov <
> > > alexander.fedu...@gmail.com> wrote:
> > > >
> > > > Hello everyone,
> > > >
> > > > Recently, there were a lot of discussions about the deprecation of
> > > various
> > > > APIs for the upcoming 2.0 release. It appears there are two main
> > > motivations
> > > > with opposing directions, causing these discussions to remain
> > unsettled.
> > > On
> > > > one hand, there's a desire to finally trim a wide range of legacy
> APIs,
> > > some
> > > > lingering around since the beginning of the 1.x release line (as far
> > > back as
> > > > 2016). On the other hand, there is a commitment to uphold our
> > guarantees
> > > to
> > > > the users, ensuring a smooth transition.
> > > >
> > > > I believe we could reconcile these two motivations. My proposition is
> > to
> > > > designate the final release of the 1.x timeline as a Long-Term
> Support
> > > (LTS)
> > > > release. By doing so, we would:
> > > >
> > > > 1. Enable more efficient cleanup and be liberated to introduce more
> > > breaking
> > > >   changes, paving the way for greater innovation in the 2.0 release.
> > > > 2. Sustain a positive user experience by granting enough time for the
> > > > changes
> > > >   introduced in 2.0 to stabilize, allowing users to confidently
> > > transition
> > > >   their production code to the new release.
> > > >
> > > > I look forward to hearing your thoughts on this proposal.
> > > >
> > > > Best Regards,
> > > > Alex
> > >
> > >
> >
>
>
> --
> https://twitter.com/snntrable
> https://github.com/knaufk
>

Re: Re: [DISCUSS][2.0] FLIP-344: Remove parameter in RichFunction#open

[Junrui Lee]

+1

Best,
Junrui

Wencong Liu  于2023年7月24日周一 20:12写道：

> Hi Timo,
>
>
> Thanks for you reply. I think adding an empty OpenContext to keep the
> signature is
> reasonable. I'll modify the FLIP at a later time.
>
>
> Best,
> Wencong Liu
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> At 2023-07-24 17:11:44, "Timo Walther"  wrote:
> >+1
> >
> >But instead we should add a OpenContext there to keep the signature
> >stable but still be able to add parameters.
> >
> >Regards,
> >Timo
> >
> >On 21.07.23 12:24, Jing Ge wrote:
> >> +1
> >>
> >> On Fri, Jul 21, 2023 at 10:22 AM Yuxin Tan 
> wrote:
> >>
> >>> +1
> >>>
> >>> Best,
> >>> Yuxin
> >>>
> >>>
> >>> Xintong Song  于2023年7月21日周五 12:04写道：
> >>>
>  +1
> 
>  Best,
> 
>  Xintong
> 
> 
> 
>  On Fri, Jul 21, 2023 at 10:52 AM Wencong Liu 
> >>> wrote:
> 
> > Hi devs,
> >
> > I would like to start a discussion on FLIP-344: Remove parameter in
> > RichFunction#open [1].
> >
> > The open() method in RichFunction requires a Configuration instance
> as
> >>> an
> > argument,
> > which is always passed as a new instance without any configuration
> > parameters in
> > AbstractUdfStreamOperator#open. Thus, it is unnecessary to include
> this
> > parameter
> > in the open() method.
> > As such I propose to remove the Configuration field from
> > RichFunction#open(Configuration parameters).
> > Looking forward to your feedback.
> > [1]
> >
> 
> >>>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425231
> > Best regards,
> >
> >
> > Wencong Liu
> 
> >>>
> >>
>

Re: [DISCUSS][2.0] FLIP-352: Use camelCast for all REST API fields/parameters

[Xintong Song]

Thanks for the explanation. I'd be fine with not bumping the version, if it
requires that much effort.


Best,

Xintong



On Tue, Jul 25, 2023 at 4:44 PM Chesnay Schepler  wrote:

> On 25/07/2023 04:09, Xintong Song wrote:
> > In this case, I wonder if it makes sense to bump the REST API version to
> > V2. That should allow us to gradually phase out the old APIs without the
> > above mentioned problems.
> It's ultimately possible do do that; it just implies duplicating _a lot_
> classes and setting up a class hierarchy between the messages such that
> we don't need to duplicate all workers and slapping v2 onto all handlers.
> It's quite a lot of monkey work really, with quite a maintenance cost on
> the 1.x side.
> Personally I'd like to avoid it; if we go with a v2 for this one all
> other changes that I'm proposing will also be pushed into v2, which will
> make the efforts more work and way harder to maintain in 1.x (and imo
> the point of 2.0 is to be able to break things hard).
>

Re: Re: Re: [DISCUSS][2.0] FLIP-347: Remove IOReadableWritable serialization in Path

[Junrui Lee]

+1

Best,
Junrui

Jing Ge  于2023年7月24日周一 23:28写道：

> agree, since we want to try our best to deprecate APIs in 1.18, it makes
> sense.
>
>
> Best regards,
> Jing
>
> On Mon, Jul 24, 2023 at 12:11 PM Wencong Liu  wrote:
>
> > Hi Jing and Matthias,
> >
> >
> > I believe it is reasonable to examine all classes that implement the
> > IOReadableWritable
> > interface and summarize their actual usage. However, due to time
> > constraints, I suggest
> > we minimize the scope of this FLIP to focus on the Path class. As for
> > other components
> > that implement IOReadableWritable, we can make an effort to investigate
> > them
> > in the future. WDYT?
> >
> >
> > Best regards,
> > Wencong Liu
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > At 2023-07-22 00:46:45, "Jing Ge"  wrote:
> > >Hi Wencong,
> > >
> > >Thanks for the clarification. I got your point. It makes sense.
> > >
> > >Wrt IOReadableWritable, the suggestion was to check all classes that
> > >implemented it, e.g. BlockInfo, Value, Configuration, etc. Not limited
> to
> > >the Path.
> > >
> > >Best regards,
> > >Jing
> > >
> > >On Fri, Jul 21, 2023 at 4:31 PM Wencong Liu 
> wrote:
> > >
> > >> Hello Jing,
> > >>
> > >>
> > >> Thanks for your reply. The URI field should be final and the
> > >> Path will be immutable.The static method deserializeFromDataInputView
> > >> will create a new Path object instead of replacing the URI field
> > >> in a existed Path Object.
> > >>
> > >>
> > >> For the crossing multiple modules issue, I've explained it in the
> reply
> > >> to Matthias.
> > >>
> > >>
> > >> Best regards,
> > >>
> > >>
> > >> Wencong Liu
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> At 2023-07-21 18:05:26, "Jing Ge"  wrote:
> > >> >Hi Wencong,
> > >> >
> > >> >Just out of curiosity, will the newly introduced
> > >> >deserializeFromDataInputView() method make the Path mutable again?
> > >> >
> > >> >What Matthias suggested makes sense, although the extension might
> make
> > >> this
> > >> >FLIP cross multiple modules.
> > >> >
> > >> >Best regards,
> > >> >Jing
> > >> >
> > >> >On Fri, Jul 21, 2023 at 10:23 AM Matthias Pohl
> > >> > wrote:
> > >> >
> > >> >> There's a kind-of-related issue FLINK-4758 [1] that proposes
> removing
> > >> the
> > >> >> IOReadableWritable interface from more classes. It was briefly
> > >> mentioned in
> > >> >> the must-have work items discussion [2].
> > >> >>
> > >> >> I'm not too sure about the usage of IOReadableWritable: ...whether
> it
> > >> would
> > >> >> go away with the removal of the DataSet API in general (the Jira
> > issue
> > >> has
> > >> >> DataSet as a component), anyway.
> > >> >>
> > >> >> Otherwise, might it make sense to extend the scope of this FLIP?
> > >> >>
> > >> >> [1] https://issues.apache.org/jira/browse/FLINK-4758
> > >> >> [2]
> https://lists.apache.org/thread/gf0h4gh3xfsj78cpdsxsnj70nhzcmv9r
> > >> >>
> > >> >> On Fri, Jul 21, 2023 at 6:04 AM Xintong Song <
> tonysong...@gmail.com>
> > >> >> wrote:
> > >> >>
> > >> >> > +1
> > >> >> >
> > >> >> > Best,
> > >> >> >
> > >> >> > Xintong
> > >> >> >
> > >> >> >
> > >> >> >
> > >> >> > On Fri, Jul 21, 2023 at 10:54 AM Wencong Liu <
> liuwencle...@163.com
> > >
> > >> >> wrote:
> > >> >> >
> > >> >> > > Hi devs,
> > >> >> > >
> > >> >> > > I would like to start a discussion on FLIP-347: Remove
> > >> >> IOReadableWritable
> > >> >> > > serialization in Path [1].
> > >> >> > >
> > >> >> > >
> > >> >> > > The Path class is currently mutable to support
> IOReadableWritable
> > >> >> > > serialization. However, many parts
> > >> >> > > of the code assume that the Path is immutable. By making the
> Path
> > >> class
> > >> >> > > immutable, we can ensure
> > >> >> > > that paths are stored correctly without the possibility of
> > mutation
> > >> and
> > >> >> > > eliminate the occurrence of subtle errors.
> > >> >> > > As such I propose to modify the Path class to no longer
> implement
> > >> the
> > >> >> > > IOReadableWritable interface.
> > >> >> > > Looking forward to your feedback.
> > >> >> > > [1]
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-347%3A+Remove+IOReadableWritable+serialization+in+Path
> > >> >> > > Best regards,
> > >> >> > >
> > >> >> > >
> > >> >> > > Wencong Liu
> > >> >> >
> > >> >>
> > >>
> >
>

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yong Fang

[Sergey Nuyanzin]

Congratulations, Yong Fang!

On Tue, Jul 25, 2023 at 7:53 AM ConradJam  wrote:

> Congratulations, Yong Fang
>
> Mang Zhang  于2023年7月25日周二 12:08写道：
>
> > Congratulations, Yong Fang!
> >
> >
> > --
> >
> > Best regards,
> > Mang Zhang
> >
> >
> >
> >
> >
> > 在 2023-07-25 10:30:24，"Jark Wu"  写道：
> > >Congratulations, Yong Fang!
> > >
> > >Best,
> > >Jark
> > >
> > >On Mon, 24 Jul 2023 at 22:11, Wencong Liu  wrote:
> > >
> > >> Congratulations!
> > >>
> > >> Best,
> > >> Wencong Liu
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> 在 2023-07-24 11:03:30，"Paul Lam"  写道：
> > >> >Congrats, Shammon!
> > >> >
> > >> >Best,
> > >> >Paul Lam
> > >> >
> > >> >> 2023年7月24日 10:56，Jingsong Li  写道：
> > >> >>
> > >> >> Shammon
> > >> >
> > >>
> >
>
>
> --
> Best
>
> ConradJam
>


-- 
Best regards,
Sergey

Re: [DISCUSS][2.0] FLIP-349: Move RocksDB statebackend classes to o.a.f.state.rocksdb package

[Jing Ge]

make sense.

Best regards,
Jing

On Tue, Jul 25, 2023 at 4:29 PM Stefan Richter
 wrote:

>
> +1
>
>
>
> > On 24. Jul 2023, at 12:25, Chesnay Schepler  wrote:
> >
> > To properly reflect the state of the rocksdb statebackend I propose to
> move all classes in the state-backend-rocksdb module under the classes to
> o.a.f.state.rocksdb package.
> >
> >
> >
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/FLINK/FLIP-349%253A%2BMove%2BRocksDB%2Bstatebackend%2Bclasses%2Bto%2Bo.a.f.state.rocksdb%2Bpackage=gmail-imap=169079912800=AOvVaw3OiTwgsLEiTcJpNTVM-Y8y
>
>

[jira] [Created] (FLINK-32669) Support port-range for taskmanager data port

[chenyuzhi (Jira)]

chenyuzhi created FLINK-32669:
-

 Summary: Support port-range for taskmanager data port
 Key: FLINK-32669
 URL: https://issues.apache.org/jira/browse/FLINK-32669
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Network
Reporter: chenyuzhi


We can setup range-port for taskmanager rpc port to avoid occupying an 
unexpected port(such as the port of datanode service).
 
However, we can't setup range-port for taskmanager data port(config-key: 
taskmanager.data.port).
In production env, it's unreasonable to setup a specify port, thus we usually 
not setup this configuration key. 
 
The problem is without setup taskmanager data port, it can conflict with port 
of other services .
It means still can be port conflict 
不合理



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS][2.0] FLIP-352: Use camelCast for all REST API fields/parameters

[Chesnay Schepler]

On 25/07/2023 04:09, Xintong Song wrote:

In this case, I wonder if it makes sense to bump the REST API version to
V2. That should allow us to gradually phase out the old APIs without the
above mentioned problems.
It's ultimately possible do do that; it just implies duplicating _a lot_ 
classes and setting up a class hierarchy between the messages such that 
we don't need to duplicate all workers and slapping v2 onto all handlers.
It's quite a lot of monkey work really, with quite a maintenance cost on 
the 1.x side.
Personally I'd like to avoid it; if we go with a v2 for this one all 
other changes that I'm proposing will also be pushed into v2, which will 
make the efforts more work and way harder to maintain in 1.x (and imo 
the point of 2.0 is to be able to break things hard).

Re: [DISCUSS][2.0] FLIP-351: REST API normalizes +/-Inf / NaN to 0

[Chesnay Schepler]

Then we'd break the API for users that did already apply workarounds 
although the user hasn't done anything wrong.


On 25/07/2023 04:31, Xintong Song wrote:

we should treat these cases as errors


Looking at the fields listed in the FLIP, I'd agree with this argument. And
based on this, shouldn't we fail the request with e.g., a status code 500,
rather than responding with a fallback value silently?

Best,

Xintong



On Tue, Jul 25, 2023 at 12:22 AM Jing Ge  wrote:


We might consider using 0 as null for values that never return 0. But null
is still not equal to 0 and it will be very difficult to let every
contributor in this community follow this rule, especially for future
unknown APIs, which means there will be some cases that still need null.
Personally, I would choose accuracy over convenience and consistency over
convenience. Therefore, null is recommended.

Best regards,
Jing

On Mon, Jul 24, 2023 at 11:48 PM Chesnay Schepler 
wrote:


The downside to null is that it again forces users to handle this case
themselves.

The reality is that there is no good default value.

Ideally we fix all cases where we return such values, such that the
fallback to 0 isn't even used.
Arguably the same could be said for null, but I'd think that 0 is less
of a surprise.

On 24/07/2023 17:21, Gyula Fóra wrote:

I agree that it's a bit strange to have 0 as a fallback value because

it

can also naturally occur for many metrics.
If we want to omit the value null would be probably better as Matthias
suggested.

Gyula

On Mon, Jul 24, 2023 at 4:02 PM Matthias Pohl
 wrote:


What was the reason you decided to go for 0 as the fallback value

instead

of null? Wouldn't that be a more reasonable value for error cases?

On Mon, Jul 24, 2023 at 12:51 PM Chesnay Schepler 
There are a number of cases where the REST API can return infinity or
NaN for certain double fields.

This is problematic because the JSON spec does not allow such values,
and tooling working against that spec may run into issues when
encountering such a value.

Specifically we've seen this become an issue in clients generated

from

the OpenAPI spec.





https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425797

Re: [DISCUSS][2.0] FLIP-349: Move RocksDB statebackend classes to o.a.f.state.rocksdb package

[Stefan Richter]

+1



> On 24. Jul 2023, at 12:25, Chesnay Schepler  wrote:
> 
> To properly reflect the state of the rocksdb statebackend I propose to move 
> all classes in the state-backend-rocksdb module under the classes to 
> o.a.f.state.rocksdb package.
> 
> 
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/display/FLINK/FLIP-349%253A%2BMove%2BRocksDB%2Bstatebackend%2Bclasses%2Bto%2Bo.a.f.state.rocksdb%2Bpackage=gmail-imap=169079912800=AOvVaw3OiTwgsLEiTcJpNTVM-Y8y
>  

[jira] [Created] (FLINK-32668) fix up watchdog timeout bug in common.sh(e2e test) ?

[Hongshun Wang (Jira)]

Hongshun Wang created FLINK-32668:
-

 Summary: fix up watchdog timeout bug in common.sh(e2e test) ?
 Key: FLINK-32668
 URL: https://issues.apache.org/jira/browse/FLINK-32668
 Project: Flink
  Issue Type: Improvement
  Components: Build System / CI
Affects Versions: 1.17.1
Reporter: Hongshun Wang
 Fix For: 1.17.2
 Attachments: image-2023-07-25-15-27-37-441.png

When run e2e test, an error like this occrurs:

!image-2023-07-25-15-27-37-441.png|width=733,height=115!

then I find a problem in the corresponding code:

 
{code:java}
kill_test_watchdog() {
    local watchdog_pid=$(cat $TEST_DATA_DIR/job_watchdog.pid)
    echo "Stopping job timeout watchdog (with pid=$watchdog_pid)"
    kill $watchdog_pid
} 
internal_run_with_timeout() {
    local timeout_in_seconds="$1"
    local on_failure="$2"
    local command_label="$3"
    local command="${@:4}"

    on_exit kill_test_watchdog
   (
           command_pid=$BASHPID
           (sleep "${timeout_in_seconds}" # set a timeout for this command
            echo "${command_label:-"The command '${command}'"} (pid: 
$command_pid) did not finish after $timeout_in_seconds seconds."
eval "${on_failure}"
           kill "$command_pid") & watchdog_pid=$!
           echo $watchdog_pid > $TEST_DATA_DIR/job_watchdog.pid
           # invoke
          $command
  )

}{code}
 

When {{$command}} completes before the timeout, the watchdog process is killed 
successfully. However, when {{$command}} times out, the watchdog process kills 
{{$command}} and then exits itself, leaving behind an error message when trying 
to kill its own process ID with {{{}kill $watchdog_pid{}}}.

 

So, I will modify like this:

 
{code:java}
kill_test_watchdog() {
      local watchdog_pid=$(cat $TEST_DATA_DIR/job_watchdog.pid)
      if kill -0 $watchdog_pid > /dev/null 2>&1; then
           echo "Stopping job timeout watchdog (with pid=$watchdog_pid)"
           kill $watchdog_pid
      else
            echo "watchdog (with pid=$watchdog_pid) does not exist now"
      fi
} {code}
 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE][2.0] FLIP-336: Remove "now" timestamp field from REST responses

[Zhu Zhu]

+1 (binding)

Thanks,
Zhu

Xintong Song  于2023年7月25日周二 09:46写道：
>
> +1 (binding)
>
> Best,
>
> Xintong
>
>
>
> On Mon, Jul 24, 2023 at 11:26 PM Jing Ge  wrote:
>
> > +1(binding)
> >
> > On Mon, Jul 24, 2023 at 8:55 PM Matthias Pohl
> >  wrote:
> >
> > > +1 (binding)
> > >
> > > On Mon, Jul 24, 2023 at 2:24 PM Konstantin Knauf 
> > > wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > Am Mo., 24. Juli 2023 um 14:15 Uhr schrieb Martijn Visser <
> > > > martijnvis...@apache.org>:
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > On Mon, Jul 24, 2023 at 1:08 PM Chesnay Schepler  > >
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I'd like to start a vote on FLIP-336.
> > > > > >
> > > > > > Discussion thread:
> > > > > > https://lists.apache.org/thread/ms3sk0p21n7q2oq0fjtq43koqj2pmwv4
> > > > > > FLIP:
> > > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263424789
> > > > > >
> > > > > > Regards,
> > > > > > Chesnay
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > https://twitter.com/snntrable
> > > > https://github.com/knaufk
> > > >
> > >
> >

Re: [DISCUSS] Proposing an LTS Release for the 1.x Line

[Konstantin Knauf]

Hi Alex,

yes, I think, it makes sense to support the last 1.x release longer than
usual. This should be limited to bugfixes in my opinion.

Best,

Konstantin

Am Di., 25. Juli 2023 um 07:07 Uhr schrieb Xintong Song <
tonysong...@gmail.com>:

> Hi Alex,
>
> Providing a longer supporting period for the last 1.x minor release makes
> sense to me.
>
> I think we need to be more specific about what LTS means here.
>
>- IIUC, that means for the last 1.x minor release, we will keep
>providing 1.x.y / 1.x.z bugfix release. This is a stronger support
> compared
>to regular minor releases which by default are only supported for 2
> minor
>release cycles.
>- Do we only provide bug fixes for the LTS release, or do we also allow
>backporting features to that release?
>- How long exactly shall we support the LTS release?
>
> And maybe we can make this a general convention for last minor releases for
> all major releases, rather than only discuss it for the 2.0 version bump.
>
> @Leonard,
>
> I'd like to clarify that there are no community decisions yet on release
> 2.0 after 1.19. It is possible to have 1.20 before 2.0.
>
> Best,
>
> Xintong
>
>
>
> On Tue, Jul 25, 2023 at 11:54 AM Leonard Xu  wrote:
>
> > +1, it’s pretty necessary especially we deprecated so many APIs in 1.18
> > and plan to remove in 2.0.
> >
> > The 1.19 should be a proper version for LTS Release.
> >
> > Best,
> > Leonard
> >
> >
> > > On Jul 25, 2023, at 3:30 AM, Alexander Fedulov <
> > alexander.fedu...@gmail.com> wrote:
> > >
> > > Hello everyone,
> > >
> > > Recently, there were a lot of discussions about the deprecation of
> > various
> > > APIs for the upcoming 2.0 release. It appears there are two main
> > motivations
> > > with opposing directions, causing these discussions to remain
> unsettled.
> > On
> > > one hand, there's a desire to finally trim a wide range of legacy APIs,
> > some
> > > lingering around since the beginning of the 1.x release line (as far
> > back as
> > > 2016). On the other hand, there is a commitment to uphold our
> guarantees
> > to
> > > the users, ensuring a smooth transition.
> > >
> > > I believe we could reconcile these two motivations. My proposition is
> to
> > > designate the final release of the 1.x timeline as a Long-Term Support
> > (LTS)
> > > release. By doing so, we would:
> > >
> > > 1. Enable more efficient cleanup and be liberated to introduce more
> > breaking
> > >   changes, paving the way for greater innovation in the 2.0 release.
> > > 2. Sustain a positive user experience by granting enough time for the
> > > changes
> > >   introduced in 2.0 to stabilize, allowing users to confidently
> > transition
> > >   their production code to the new release.
> > >
> > > I look forward to hearing your thoughts on this proposal.
> > >
> > > Best Regards,
> > > Alex
> >
> >
>


-- 
https://twitter.com/snntrable
https://github.com/knaufk

[jira] [Created] (FLINK-32667) Use standalone store and embedding writer for jobs with no-restart-strategy in session cluster

[Fang Yong (Jira)]

Fang Yong created FLINK-32667:
-

 Summary: Use standalone store and embedding writer for jobs with 
no-restart-strategy in session cluster
 Key: FLINK-32667
 URL: https://issues.apache.org/jira/browse/FLINK-32667
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Affects Versions: 1.18.0
Reporter: Fang Yong


When a flink session cluster use zk or k8s high availability service, it will 
store jobs in zk or ConfigMap. When we submit flink olap jobs to the session 
cluster, they always turn off restart strategy. These jobs with 
no-restart-strategy should not be stored in zk or ConfigMap in k8s



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-32666) ASM rewrite class lead to package failed.

[lizhiqiang (Jira)]

lizhiqiang created FLINK-32666:
--

 Summary: ASM rewrite class lead to package failed.
 Key: FLINK-32666
 URL: https://issues.apache.org/jira/browse/FLINK-32666
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Affects Versions: 1.18.0
Reporter: lizhiqiang


{code:java}
[DEBUG] Processing JAR 
/Users/lzq/Desktop/Projects/Flink/flink/flink-master/flink-table/flink-table-planner/target/flink-table-planner_2.12-1.17-SNAPSHOT.jar
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/interpreter/JaninoRexCompiler.class
[DEBUG] Rewrote class bytecode: org/apache/calcite/tools/RelBuilder$Frame.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/tools/ImmutableRelBuilder$Config.class
[DEBUG] Rewrote class bytecode: org/apache/calcite/tools/RelBuilder.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/tools/RelBuilder$OverCallImpl.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/tools/RelBuilder$GroupKey.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/tools/RelBuilder$OverCall.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/tools/RelBuilder$RelOptTableFinder.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/tools/ImmutableRelBuilder$Config$Builder.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/tools/RelBuilder$Registrar.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/tools/RelBuilder$OverCallImpl$1.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/tools/RelBuilder$AggCallImpl.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/tools/ImmutableRelBuilder.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/tools/RelBuilder$AggCall.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/tools/ImmutableRelBuilder$1.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/tools/RelBuilder$1.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/tools/RelBuilder$GroupKeyImpl.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/tools/RelBuilder$Config.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/tools/ImmutableRelBuilder$Config$InitShim.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/tools/RelBuilder$AggCallPlus.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/tools/RelBuilder$AggCallImpl2.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/tools/RelBuilder$Shifter.class
[DEBUG] Rewrote class bytecode: org/apache/calcite/tools/RelBuilder$Field.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/tools/RelBuilder$2.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/util/javac/JaninoCompiler$JaninoCompilerArgs.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/util/javac/JaninoCompiler.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/util/javac/JaninoCompiler$AccountingClassLoader.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/adapter/enumerable/EnumerableInterpretable$1$1.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/adapter/enumerable/EnumerableInterpretable$EnumerableNode.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/adapter/enumerable/EnumerableInterpretable$1.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/adapter/enumerable/EnumerableInterpretable$StaticFieldDetector.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/adapter/enumerable/EnumerableInterpretable.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/adapter/enumerable/EnumerableInterpretable$1$1$1.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/jdbc/CalciteSchemaBuilder.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/rex/RexSimplify$SafeRexVisitor.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/rex/RexSimplify$SargCollector.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/rex/RexSimplify$RexSargBuilder.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/rex/RexSimplify$IsPredicate.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/rex/RexSimplify$Predicate.class
[DEBUG] Rewrote class bytecode: 
org/apache/calcite/rex/RexSimplify$Comparison.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/rex/RexSimplify$CaseBranch.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/rex/RexSimplify$1.class
[DEBUG] Rewrote class bytecode: org/apache/calcite/rex/RexSimplify.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/rel/hint/NodeTypeHintPredicate$NodeType.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/rel/hint/NodeTypeHintPredicate$1.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/rel/hint/HintPredicates.class
[DEBUG] Keeping original class bytecode: 
org/apache/calcite/rel/hint/NodeTypeHintPredicate.class
[DEBUG] Rewrote class bytecode: 

[jira] [Created] (FLINK-32665) Support read null value for csv format

[Fang Yong (Jira)]

Fang Yong created FLINK-32665:
-

 Summary: Support read null value for csv format
 Key: FLINK-32665
 URL: https://issues.apache.org/jira/browse/FLINK-32665
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.18.0
Reporter: Fang Yong


when there is null column in a file with csv format, it will throw exception 
when flink job try to parse these data



--
This message was sent by Atlassian Jira
(v8.20.10#820010)