[VOTE] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

[Shammon FY]

Hi Devs,

I'd like to start the vote on FLIP-293: Introduce Flink Jdbc Driver For Sql
Gateway [1].

The FLIP was discussed in thread [2], and it aims to introduce Flink Jdbc
Driver module in Flink.

The vote will last for at least 72 hours (03/16, 15:00 UTC+8) unless there
is an objection or insufficient vote. Thank you all.


[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway
[2] https://lists.apache.org/thread/d1owrg8zh77v0xygcpb93fxt0jpjdkb3


Best,
Shammon.FY

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Jane Chan]

Congratulations, Yuxia!

Best,
Jane

On Mon, Mar 13, 2023 at 12:26 PM Lincoln Lee  wrote:

> Congratulations, Yuxia!
>
> Best,
> Lincoln Lee
>
>
> Shammon FY  于2023年3月13日周一 12:16写道：
>
> > Congratulations, Yuxia
> >
> > On Mon, Mar 13, 2023 at 12:11 PM Jingsong Li 
> > wrote:
> >
> > > Congratulations, Yuxia!
> > >
> > > On Mon, Mar 13, 2023 at 11:49 AM Juntao Hu 
> wrote:
> > > >
> > > > Congratulations, Yuxia!
> > > >
> > > > Best,
> > > > Juntao
> > > >
> > > >
> > > > Wencong Liu  于2023年3月13日周一 11:33写道：
> > > >
> > > > > Congratulations, Yuxia!
> > > > >
> > > > > Best,
> > > > > Wencong Liu
> > > > >
> > > > >
> > > > > At 2023-03-13 11:20:21, "Qingsheng Ren" 
> wrote:
> > > > > >Congratulations, Yuxia!
> > > > > >
> > > > > >Best,
> > > > > >Qingsheng
> > > > > >
> > > > > >On Mon, Mar 13, 2023 at 10:27 AM Jark Wu 
> wrote:
> > > > > >
> > > > > >> Hi, everyone
> > > > > >>
> > > > > >> On behalf of the PMC, I'm very happy to announce Yuxia Luo as a
> > new
> > > > > Flink
> > > > > >> Committer.
> > > > > >>
> > > > > >> Yuxia has been continuously contributing to the Flink project
> for
> > > almost
> > > > > >> two
> > > > > >> years, authored and reviewed hundreds of PRs over this time. He
> is
> > > > > >> currently
> > > > > >> the core maintainer of the Hive component, where he contributed
> > many
> > > > > >> valuable
> > > > > >> features, including the Hive dialect with 95% compatibility and
> > > small
> > > > > file
> > > > > >> compaction.
> > > > > >> In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to
> better
> > > > > >> integrate
> > > > > >> Flink with data lakes. He actively participated in dev
> discussions
> > > and
> > > > > >> answered
> > > > > >> many questions on the user mailing list.
> > > > > >>
> > > > > >> Please join me in congratulating Yuxia Luo for becoming a Flink
> > > > > Committer!
> > > > > >>
> > > > > >> Best,
> > > > > >> Jark Wu (on behalf of the Flink PMC)
> > > > > >>
> > > > >
> > >
> >
>

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Lincoln Lee]

Congratulations, Yuxia!

Best,
Lincoln Lee


Shammon FY  于2023年3月13日周一 12:16写道：

> Congratulations, Yuxia
>
> On Mon, Mar 13, 2023 at 12:11 PM Jingsong Li 
> wrote:
>
> > Congratulations, Yuxia!
> >
> > On Mon, Mar 13, 2023 at 11:49 AM Juntao Hu  wrote:
> > >
> > > Congratulations, Yuxia!
> > >
> > > Best,
> > > Juntao
> > >
> > >
> > > Wencong Liu  于2023年3月13日周一 11:33写道：
> > >
> > > > Congratulations, Yuxia!
> > > >
> > > > Best,
> > > > Wencong Liu
> > > >
> > > >
> > > > At 2023-03-13 11:20:21, "Qingsheng Ren"  wrote:
> > > > >Congratulations, Yuxia!
> > > > >
> > > > >Best,
> > > > >Qingsheng
> > > > >
> > > > >On Mon, Mar 13, 2023 at 10:27 AM Jark Wu  wrote:
> > > > >
> > > > >> Hi, everyone
> > > > >>
> > > > >> On behalf of the PMC, I'm very happy to announce Yuxia Luo as a
> new
> > > > Flink
> > > > >> Committer.
> > > > >>
> > > > >> Yuxia has been continuously contributing to the Flink project for
> > almost
> > > > >> two
> > > > >> years, authored and reviewed hundreds of PRs over this time. He is
> > > > >> currently
> > > > >> the core maintainer of the Hive component, where he contributed
> many
> > > > >> valuable
> > > > >> features, including the Hive dialect with 95% compatibility and
> > small
> > > > file
> > > > >> compaction.
> > > > >> In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> > > > >> integrate
> > > > >> Flink with data lakes. He actively participated in dev discussions
> > and
> > > > >> answered
> > > > >> many questions on the user mailing list.
> > > > >>
> > > > >> Please join me in congratulating Yuxia Luo for becoming a Flink
> > > > Committer!
> > > > >>
> > > > >> Best,
> > > > >> Jark Wu (on behalf of the Flink PMC)
> > > > >>
> > > >
> >
>

[jira] [Created] (FLINK-31410) ListStateWithCache Should support incremental snapshot

[Zhipeng Zhang (Jira)]

Zhipeng Zhang created FLINK-31410:
-

 Summary: ListStateWithCache Should support incremental snapshot
 Key: FLINK-31410
 URL: https://issues.apache.org/jira/browse/FLINK-31410
 Project: Flink
  Issue Type: Bug
  Components: Library / Machine Learning
Affects Versions: ml-2.2.0
Reporter: Zhipeng Zhang


In Flink ML, we used ListStateWithCache [2] to enable caching data in memory 
and filesystem. However, it does not support incremental snapshot now — It 
writes all the data to checkpoint stream when calling snapshot [1], which could 
be inefficient.

 

 
[1][https://github.com/apache/flink-ml/blob/master/flink-ml-iteration/src/main/java/org/apache/flink/iteration/datacache/nonkeyed/DataCacheSnapshot.java#L116]

 
[2][https://github.com/apache/flink-ml/blob/master/flink-ml-iteration/src/main/java/org/apache/flink/iteration/datacache/nonkeyed/ListStateWithCache.java]
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Shammon FY]

Congratulations, Yuxia

On Mon, Mar 13, 2023 at 12:11 PM Jingsong Li  wrote:

> Congratulations, Yuxia!
>
> On Mon, Mar 13, 2023 at 11:49 AM Juntao Hu  wrote:
> >
> > Congratulations, Yuxia!
> >
> > Best,
> > Juntao
> >
> >
> > Wencong Liu  于2023年3月13日周一 11:33写道：
> >
> > > Congratulations, Yuxia!
> > >
> > > Best,
> > > Wencong Liu
> > >
> > >
> > > At 2023-03-13 11:20:21, "Qingsheng Ren"  wrote:
> > > >Congratulations, Yuxia!
> > > >
> > > >Best,
> > > >Qingsheng
> > > >
> > > >On Mon, Mar 13, 2023 at 10:27 AM Jark Wu  wrote:
> > > >
> > > >> Hi, everyone
> > > >>
> > > >> On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new
> > > Flink
> > > >> Committer.
> > > >>
> > > >> Yuxia has been continuously contributing to the Flink project for
> almost
> > > >> two
> > > >> years, authored and reviewed hundreds of PRs over this time. He is
> > > >> currently
> > > >> the core maintainer of the Hive component, where he contributed many
> > > >> valuable
> > > >> features, including the Hive dialect with 95% compatibility and
> small
> > > file
> > > >> compaction.
> > > >> In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> > > >> integrate
> > > >> Flink with data lakes. He actively participated in dev discussions
> and
> > > >> answered
> > > >> many questions on the user mailing list.
> > > >>
> > > >> Please join me in congratulating Yuxia Luo for becoming a Flink
> > > Committer!
> > > >>
> > > >> Best,
> > > >> Jark Wu (on behalf of the Flink PMC)
> > > >>
> > >
>

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Weihua Hu]

Congratulations, Yuxia!

Best,
Weihua


On Mon, Mar 13, 2023 at 12:11 PM Jingsong Li  wrote:

> Congratulations, Yuxia!
>
> On Mon, Mar 13, 2023 at 11:49 AM Juntao Hu  wrote:
> >
> > Congratulations, Yuxia!
> >
> > Best,
> > Juntao
> >
> >
> > Wencong Liu  于2023年3月13日周一 11:33写道：
> >
> > > Congratulations, Yuxia!
> > >
> > > Best,
> > > Wencong Liu
> > >
> > >
> > > At 2023-03-13 11:20:21, "Qingsheng Ren"  wrote:
> > > >Congratulations, Yuxia!
> > > >
> > > >Best,
> > > >Qingsheng
> > > >
> > > >On Mon, Mar 13, 2023 at 10:27 AM Jark Wu  wrote:
> > > >
> > > >> Hi, everyone
> > > >>
> > > >> On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new
> > > Flink
> > > >> Committer.
> > > >>
> > > >> Yuxia has been continuously contributing to the Flink project for
> almost
> > > >> two
> > > >> years, authored and reviewed hundreds of PRs over this time. He is
> > > >> currently
> > > >> the core maintainer of the Hive component, where he contributed many
> > > >> valuable
> > > >> features, including the Hive dialect with 95% compatibility and
> small
> > > file
> > > >> compaction.
> > > >> In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> > > >> integrate
> > > >> Flink with data lakes. He actively participated in dev discussions
> and
> > > >> answered
> > > >> many questions on the user mailing list.
> > > >>
> > > >> Please join me in congratulating Yuxia Luo for becoming a Flink
> > > Committer!
> > > >>
> > > >> Best,
> > > >> Jark Wu (on behalf of the Flink PMC)
> > > >>
> > >
>

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Jingsong Li]

Congratulations, Yuxia!

On Mon, Mar 13, 2023 at 11:49 AM Juntao Hu  wrote:
>
> Congratulations, Yuxia!
>
> Best,
> Juntao
>
>
> Wencong Liu  于2023年3月13日周一 11:33写道：
>
> > Congratulations, Yuxia!
> >
> > Best,
> > Wencong Liu
> >
> >
> > At 2023-03-13 11:20:21, "Qingsheng Ren"  wrote:
> > >Congratulations, Yuxia!
> > >
> > >Best,
> > >Qingsheng
> > >
> > >On Mon, Mar 13, 2023 at 10:27 AM Jark Wu  wrote:
> > >
> > >> Hi, everyone
> > >>
> > >> On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new
> > Flink
> > >> Committer.
> > >>
> > >> Yuxia has been continuously contributing to the Flink project for almost
> > >> two
> > >> years, authored and reviewed hundreds of PRs over this time. He is
> > >> currently
> > >> the core maintainer of the Hive component, where he contributed many
> > >> valuable
> > >> features, including the Hive dialect with 95% compatibility and small
> > file
> > >> compaction.
> > >> In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> > >> integrate
> > >> Flink with data lakes. He actively participated in dev discussions and
> > >> answered
> > >> many questions on the user mailing list.
> > >>
> > >> Please join me in congratulating Yuxia Luo for becoming a Flink
> > Committer!
> > >>
> > >> Best,
> > >> Jark Wu (on behalf of the Flink PMC)
> > >>
> >

Re: Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Juntao Hu]

Congratulations, Yuxia!

Best,
Juntao


Wencong Liu  于2023年3月13日周一 11:33写道：

> Congratulations, Yuxia!
>
> Best,
> Wencong Liu
>
>
> At 2023-03-13 11:20:21, "Qingsheng Ren"  wrote:
> >Congratulations, Yuxia!
> >
> >Best,
> >Qingsheng
> >
> >On Mon, Mar 13, 2023 at 10:27 AM Jark Wu  wrote:
> >
> >> Hi, everyone
> >>
> >> On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new
> Flink
> >> Committer.
> >>
> >> Yuxia has been continuously contributing to the Flink project for almost
> >> two
> >> years, authored and reviewed hundreds of PRs over this time. He is
> >> currently
> >> the core maintainer of the Hive component, where he contributed many
> >> valuable
> >> features, including the Hive dialect with 95% compatibility and small
> file
> >> compaction.
> >> In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> >> integrate
> >> Flink with data lakes. He actively participated in dev discussions and
> >> answered
> >> many questions on the user mailing list.
> >>
> >> Please join me in congratulating Yuxia Luo for becoming a Flink
> Committer!
> >>
> >> Best,
> >> Jark Wu (on behalf of the Flink PMC)
> >>
>

Re:Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Wencong Liu]

Congratulations, Yuxia!

Best,
Wencong Liu


At 2023-03-13 11:20:21, "Qingsheng Ren"  wrote:
>Congratulations, Yuxia!
>
>Best,
>Qingsheng
>
>On Mon, Mar 13, 2023 at 10:27 AM Jark Wu  wrote:
>
>> Hi, everyone
>>
>> On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new Flink
>> Committer.
>>
>> Yuxia has been continuously contributing to the Flink project for almost
>> two
>> years, authored and reviewed hundreds of PRs over this time. He is
>> currently
>> the core maintainer of the Hive component, where he contributed many
>> valuable
>> features, including the Hive dialect with 95% compatibility and small file
>> compaction.
>> In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
>> integrate
>> Flink with data lakes. He actively participated in dev discussions and
>> answered
>> many questions on the user mailing list.
>>
>> Please join me in congratulating Yuxia Luo for becoming a Flink Committer!
>>
>> Best,
>> Jark Wu (on behalf of the Flink PMC)
>>

Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Qingsheng Ren]

Congratulations, Yuxia!

Best,
Qingsheng

On Mon, Mar 13, 2023 at 10:27 AM Jark Wu  wrote:

> Hi, everyone
>
> On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new Flink
> Committer.
>
> Yuxia has been continuously contributing to the Flink project for almost
> two
> years, authored and reviewed hundreds of PRs over this time. He is
> currently
> the core maintainer of the Hive component, where he contributed many
> valuable
> features, including the Hive dialect with 95% compatibility and small file
> compaction.
> In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> integrate
> Flink with data lakes. He actively participated in dev discussions and
> answered
> many questions on the user mailing list.
>
> Please join me in congratulating Yuxia Luo for becoming a Flink Committer!
>
> Best,
> Jark Wu (on behalf of the Flink PMC)
>

Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[weijie guo]

Congratulations, Yuxia

Best regards,

Weijie


Geng Biao  于2023年3月13日周一 11:02写道：

> Congrats, yuxia.
> Best,
> Biao Geng
>
> 获取 Outlook for iOS
> 
> 发件人: Guowei Ma 
> 发送时间: Monday, March 13, 2023 10:54:24 AM
> 收件人: dev@flink.apache.org 
> 主题: Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo
>
> congratulations Yuxia
> Best,
> Guowei
>
>
> On Mon, Mar 13, 2023 at 10:43 AM Junrui Lee  wrote:
>
> > Congratulations, Yuxia!
> >
> > Best,
> > Junrui
> >
> > Yanfei Lei  于2023年3月13日周一 10:42写道：
> >
> > > Congratulations, Yuxia!
> > >
> > > Best,
> > > Yanfei
> > >
> > >
> > > Samrat Deb  于2023年3月13日周一 10:41写道：
> > > >
> > > > congratulations Yuxia
> > > >
> > > > Bests,
> > > > Samrat
> > > >
> > > > On Mon, 13 Mar 2023 at 8:06 AM, Yuxin Tan 
> > > wrote:
> > > >
> > > > > Congratulations, Yuxia!
> > > > >
> > > > > Best,
> > > > > Yuxin
> > > > >
> > > > >
> > > > > Jark Wu  于2023年3月13日周一 10:26写道：
> > > > >
> > > > > > Hi, everyone
> > > > > >
> > > > > > On behalf of the PMC, I'm very happy to announce Yuxia Luo as a
> new
> > > Flink
> > > > > > Committer.
> > > > > >
> > > > > > Yuxia has been continuously contributing to the Flink project for
> > > almost
> > > > > > two
> > > > > > years, authored and reviewed hundreds of PRs over this time. He
> is
> > > > > > currently
> > > > > > the core maintainer of the Hive component, where he contributed
> > many
> > > > > > valuable
> > > > > > features, including the Hive dialect with 95% compatibility and
> > small
> > > > > file
> > > > > > compaction.
> > > > > > In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to
> better
> > > > > > integrate
> > > > > > Flink with data lakes. He actively participated in dev
> discussions
> > > and
> > > > > > answered
> > > > > > many questions on the user mailing list.
> > > > > >
> > > > > > Please join me in congratulating Yuxia Luo for becoming a Flink
> > > > > Committer!
> > > > > >
> > > > > > Best,
> > > > > > Jark Wu (on behalf of the Flink PMC)
> > > > > >
> > > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Geng Biao]

Congrats, yuxia.
Best,
Biao Geng

获取 Outlook for iOS

发件人: Guowei Ma 
发送时间: Monday, March 13, 2023 10:54:24 AM
收件人: dev@flink.apache.org 
主题: Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

congratulations Yuxia
Best,
Guowei


On Mon, Mar 13, 2023 at 10:43 AM Junrui Lee  wrote:

> Congratulations, Yuxia!
>
> Best,
> Junrui
>
> Yanfei Lei  于2023年3月13日周一 10:42写道：
>
> > Congratulations, Yuxia!
> >
> > Best,
> > Yanfei
> >
> >
> > Samrat Deb  于2023年3月13日周一 10:41写道：
> > >
> > > congratulations Yuxia
> > >
> > > Bests,
> > > Samrat
> > >
> > > On Mon, 13 Mar 2023 at 8:06 AM, Yuxin Tan 
> > wrote:
> > >
> > > > Congratulations, Yuxia!
> > > >
> > > > Best,
> > > > Yuxin
> > > >
> > > >
> > > > Jark Wu  于2023年3月13日周一 10:26写道：
> > > >
> > > > > Hi, everyone
> > > > >
> > > > > On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new
> > Flink
> > > > > Committer.
> > > > >
> > > > > Yuxia has been continuously contributing to the Flink project for
> > almost
> > > > > two
> > > > > years, authored and reviewed hundreds of PRs over this time. He is
> > > > > currently
> > > > > the core maintainer of the Hive component, where he contributed
> many
> > > > > valuable
> > > > > features, including the Hive dialect with 95% compatibility and
> small
> > > > file
> > > > > compaction.
> > > > > In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> > > > > integrate
> > > > > Flink with data lakes. He actively participated in dev discussions
> > and
> > > > > answered
> > > > > many questions on the user mailing list.
> > > > >
> > > > > Please join me in congratulating Yuxia Luo for becoming a Flink
> > > > Committer!
> > > > >
> > > > > Best,
> > > > > Jark Wu (on behalf of the Flink PMC)
> > > > >
> > > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Hang Ruan]

Congratulations, Yuxia!

Best,
Hang

Guowei Ma  于2023年3月13日周一 10:55写道：

> congratulations Yuxia
> Best,
> Guowei
>
>
> On Mon, Mar 13, 2023 at 10:43 AM Junrui Lee  wrote:
>
> > Congratulations, Yuxia!
> >
> > Best,
> > Junrui
> >
> > Yanfei Lei  于2023年3月13日周一 10:42写道：
> >
> > > Congratulations, Yuxia!
> > >
> > > Best,
> > > Yanfei
> > >
> > >
> > > Samrat Deb  于2023年3月13日周一 10:41写道：
> > > >
> > > > congratulations Yuxia
> > > >
> > > > Bests,
> > > > Samrat
> > > >
> > > > On Mon, 13 Mar 2023 at 8:06 AM, Yuxin Tan 
> > > wrote:
> > > >
> > > > > Congratulations, Yuxia!
> > > > >
> > > > > Best,
> > > > > Yuxin
> > > > >
> > > > >
> > > > > Jark Wu  于2023年3月13日周一 10:26写道：
> > > > >
> > > > > > Hi, everyone
> > > > > >
> > > > > > On behalf of the PMC, I'm very happy to announce Yuxia Luo as a
> new
> > > Flink
> > > > > > Committer.
> > > > > >
> > > > > > Yuxia has been continuously contributing to the Flink project for
> > > almost
> > > > > > two
> > > > > > years, authored and reviewed hundreds of PRs over this time. He
> is
> > > > > > currently
> > > > > > the core maintainer of the Hive component, where he contributed
> > many
> > > > > > valuable
> > > > > > features, including the Hive dialect with 95% compatibility and
> > small
> > > > > file
> > > > > > compaction.
> > > > > > In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to
> better
> > > > > > integrate
> > > > > > Flink with data lakes. He actively participated in dev
> discussions
> > > and
> > > > > > answered
> > > > > > many questions on the user mailing list.
> > > > > >
> > > > > > Please join me in congratulating Yuxia Luo for becoming a Flink
> > > > > Committer!
> > > > > >
> > > > > > Best,
> > > > > > Jark Wu (on behalf of the Flink PMC)
> > > > > >
> > > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Yun Tang]

Congratulations, Yuxia!


Best
Yun Tang

From: Guowei Ma 
Sent: Monday, March 13, 2023 10:54
To: dev@flink.apache.org 
Subject: Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

congratulations Yuxia
Best,
Guowei


On Mon, Mar 13, 2023 at 10:43 AM Junrui Lee  wrote:

> Congratulations, Yuxia!
>
> Best,
> Junrui
>
> Yanfei Lei  于2023年3月13日周一 10:42写道：
>
> > Congratulations, Yuxia!
> >
> > Best,
> > Yanfei
> >
> >
> > Samrat Deb  于2023年3月13日周一 10:41写道：
> > >
> > > congratulations Yuxia
> > >
> > > Bests,
> > > Samrat
> > >
> > > On Mon, 13 Mar 2023 at 8:06 AM, Yuxin Tan 
> > wrote:
> > >
> > > > Congratulations, Yuxia!
> > > >
> > > > Best,
> > > > Yuxin
> > > >
> > > >
> > > > Jark Wu  于2023年3月13日周一 10:26写道：
> > > >
> > > > > Hi, everyone
> > > > >
> > > > > On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new
> > Flink
> > > > > Committer.
> > > > >
> > > > > Yuxia has been continuously contributing to the Flink project for
> > almost
> > > > > two
> > > > > years, authored and reviewed hundreds of PRs over this time. He is
> > > > > currently
> > > > > the core maintainer of the Hive component, where he contributed
> many
> > > > > valuable
> > > > > features, including the Hive dialect with 95% compatibility and
> small
> > > > file
> > > > > compaction.
> > > > > In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> > > > > integrate
> > > > > Flink with data lakes. He actively participated in dev discussions
> > and
> > > > > answered
> > > > > many questions on the user mailing list.
> > > > >
> > > > > Please join me in congratulating Yuxia Luo for becoming a Flink
> > > > Committer!
> > > > >
> > > > > Best,
> > > > > Jark Wu (on behalf of the Flink PMC)
> > > > >
> > > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Guowei Ma]

congratulations Yuxia
Best,
Guowei


On Mon, Mar 13, 2023 at 10:43 AM Junrui Lee  wrote:

> Congratulations, Yuxia!
>
> Best,
> Junrui
>
> Yanfei Lei  于2023年3月13日周一 10:42写道：
>
> > Congratulations, Yuxia!
> >
> > Best,
> > Yanfei
> >
> >
> > Samrat Deb  于2023年3月13日周一 10:41写道：
> > >
> > > congratulations Yuxia
> > >
> > > Bests,
> > > Samrat
> > >
> > > On Mon, 13 Mar 2023 at 8:06 AM, Yuxin Tan 
> > wrote:
> > >
> > > > Congratulations, Yuxia!
> > > >
> > > > Best,
> > > > Yuxin
> > > >
> > > >
> > > > Jark Wu  于2023年3月13日周一 10:26写道：
> > > >
> > > > > Hi, everyone
> > > > >
> > > > > On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new
> > Flink
> > > > > Committer.
> > > > >
> > > > > Yuxia has been continuously contributing to the Flink project for
> > almost
> > > > > two
> > > > > years, authored and reviewed hundreds of PRs over this time. He is
> > > > > currently
> > > > > the core maintainer of the Hive component, where he contributed
> many
> > > > > valuable
> > > > > features, including the Hive dialect with 95% compatibility and
> small
> > > > file
> > > > > compaction.
> > > > > In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> > > > > integrate
> > > > > Flink with data lakes. He actively participated in dev discussions
> > and
> > > > > answered
> > > > > many questions on the user mailing list.
> > > > >
> > > > > Please join me in congratulating Yuxia Luo for becoming a Flink
> > > > Committer!
> > > > >
> > > > > Best,
> > > > > Jark Wu (on behalf of the Flink PMC)
> > > > >
> > > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Junrui Lee]

Congratulations, Yuxia!

Best,
Junrui

Yanfei Lei  于2023年3月13日周一 10:42写道：

> Congratulations, Yuxia!
>
> Best,
> Yanfei
>
>
> Samrat Deb  于2023年3月13日周一 10:41写道：
> >
> > congratulations Yuxia
> >
> > Bests,
> > Samrat
> >
> > On Mon, 13 Mar 2023 at 8:06 AM, Yuxin Tan 
> wrote:
> >
> > > Congratulations, Yuxia!
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Jark Wu  于2023年3月13日周一 10:26写道：
> > >
> > > > Hi, everyone
> > > >
> > > > On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new
> Flink
> > > > Committer.
> > > >
> > > > Yuxia has been continuously contributing to the Flink project for
> almost
> > > > two
> > > > years, authored and reviewed hundreds of PRs over this time. He is
> > > > currently
> > > > the core maintainer of the Hive component, where he contributed many
> > > > valuable
> > > > features, including the Hive dialect with 95% compatibility and small
> > > file
> > > > compaction.
> > > > In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> > > > integrate
> > > > Flink with data lakes. He actively participated in dev discussions
> and
> > > > answered
> > > > many questions on the user mailing list.
> > > >
> > > > Please join me in congratulating Yuxia Luo for becoming a Flink
> > > Committer!
> > > >
> > > > Best,
> > > > Jark Wu (on behalf of the Flink PMC)
> > > >
> > >
>

Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Yanfei Lei]

Congratulations, Yuxia!

Best,
Yanfei


Samrat Deb  于2023年3月13日周一 10:41写道：
>
> congratulations Yuxia
>
> Bests,
> Samrat
>
> On Mon, 13 Mar 2023 at 8:06 AM, Yuxin Tan  wrote:
>
> > Congratulations, Yuxia!
> >
> > Best,
> > Yuxin
> >
> >
> > Jark Wu  于2023年3月13日周一 10:26写道：
> >
> > > Hi, everyone
> > >
> > > On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new Flink
> > > Committer.
> > >
> > > Yuxia has been continuously contributing to the Flink project for almost
> > > two
> > > years, authored and reviewed hundreds of PRs over this time. He is
> > > currently
> > > the core maintainer of the Hive component, where he contributed many
> > > valuable
> > > features, including the Hive dialect with 95% compatibility and small
> > file
> > > compaction.
> > > In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> > > integrate
> > > Flink with data lakes. He actively participated in dev discussions and
> > > answered
> > > many questions on the user mailing list.
> > >
> > > Please join me in congratulating Yuxia Luo for becoming a Flink
> > Committer!
> > >
> > > Best,
> > > Jark Wu (on behalf of the Flink PMC)
> > >
> >

Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Benchao Li]

Congratulations, Yuxia!

Samrat Deb  于2023年3月13日周一 10:40写道：

> congratulations Yuxia
>
> Bests,
> Samrat
>
> On Mon, 13 Mar 2023 at 8:06 AM, Yuxin Tan  wrote:
>
> > Congratulations, Yuxia!
> >
> > Best,
> > Yuxin
> >
> >
> > Jark Wu  于2023年3月13日周一 10:26写道：
> >
> > > Hi, everyone
> > >
> > > On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new
> Flink
> > > Committer.
> > >
> > > Yuxia has been continuously contributing to the Flink project for
> almost
> > > two
> > > years, authored and reviewed hundreds of PRs over this time. He is
> > > currently
> > > the core maintainer of the Hive component, where he contributed many
> > > valuable
> > > features, including the Hive dialect with 95% compatibility and small
> > file
> > > compaction.
> > > In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> > > integrate
> > > Flink with data lakes. He actively participated in dev discussions and
> > > answered
> > > many questions on the user mailing list.
> > >
> > > Please join me in congratulating Yuxia Luo for becoming a Flink
> > Committer!
> > >
> > > Best,
> > > Jark Wu (on behalf of the Flink PMC)
> > >
> >
>


-- 

Best,
Benchao Li

Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Samrat Deb]

congratulations Yuxia

Bests,
Samrat

On Mon, 13 Mar 2023 at 8:06 AM, Yuxin Tan  wrote:

> Congratulations, Yuxia!
>
> Best,
> Yuxin
>
>
> Jark Wu  于2023年3月13日周一 10:26写道：
>
> > Hi, everyone
> >
> > On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new Flink
> > Committer.
> >
> > Yuxia has been continuously contributing to the Flink project for almost
> > two
> > years, authored and reviewed hundreds of PRs over this time. He is
> > currently
> > the core maintainer of the Hive component, where he contributed many
> > valuable
> > features, including the Hive dialect with 95% compatibility and small
> file
> > compaction.
> > In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> > integrate
> > Flink with data lakes. He actively participated in dev discussions and
> > answered
> > many questions on the user mailing list.
> >
> > Please join me in congratulating Yuxia Luo for becoming a Flink
> Committer!
> >
> > Best,
> > Jark Wu (on behalf of the Flink PMC)
> >
>

Re: 退订

[Yuxin Tan]

退订请发任意内容到 dev-unsubscr...@flink.apache.org [1]

[1] https://flink.apache.org/community/

Best,
Yuxin


MuChen <9329...@qq.com.invalid> 于2023年3月13日周一 08:48写道：

> 退订
>
>
>
>
> MuChen
> 9329...@qq.com
>
>
>
> 

Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Aitozi]

Congratulations, Yuxia

Best,
Aitozi

Yuxin Tan  于2023年3月13日周一 10:36写道：

> Congratulations, Yuxia!
>
> Best,
> Yuxin
>
>
> Jark Wu  于2023年3月13日周一 10:26写道：
>
> > Hi, everyone
> >
> > On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new Flink
> > Committer.
> >
> > Yuxia has been continuously contributing to the Flink project for almost
> > two
> > years, authored and reviewed hundreds of PRs over this time. He is
> > currently
> > the core maintainer of the Hive component, where he contributed many
> > valuable
> > features, including the Hive dialect with 95% compatibility and small
> file
> > compaction.
> > In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> > integrate
> > Flink with data lakes. He actively participated in dev discussions and
> > answered
> > many questions on the user mailing list.
> >
> > Please join me in congratulating Yuxia Luo for becoming a Flink
> Committer!
> >
> > Best,
> > Jark Wu (on behalf of the Flink PMC)
> >
>

Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

[Benchao Li]

Hi Jing & Shammon,

IMO, Jdbc is a Java standard and Flink-JDBC-Driver is an implementation to
interact with Flink, so in theory, it would be doable to use
Flink-JDBC-Driver in Flink-JDBC-Connector if we add it as a dialect.

However, I don't see a scenario which requires this yet. All in all, it
would be an orthogonal topic, which does not block current FLIP discussion.

Shammon FY  于2023年3月13日周一 08:47写道：

> Hi Jing
>
> Flink-Jdbc-Driver is mainly used to facilitate users to submit flink sql
> jobs. I think we will not use Flink-Jdbc-Driver in Flink-Jdbc-Connector to
> read/write data in general. If users can read/write specific tables by
> Flink-Jdbc-Driver, they can use these tables in their flink jobs directly.
> Thanks
>
> Best,
> Shammon
>
>
> On Mon, Mar 13, 2023 at 6:21 AM Jing Ge 
> wrote:
>
> > Hi, Shammon,
> >
> > This FLIP is very interesting. Thanks for driving it! Just out of
> > curiosity, Could it be used in the Flink JDBC connector[1] after
> > doing the appropriate implementation there?
> >
> > Best regards,
> > Jing
> >
> > [1]
> >
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/
> >
> > On Fri, Mar 3, 2023 at 8:56 AM Shammon FY  wrote:
> >
> > > Hi,
> > >
> > > Thanks jinsong. I think implementing JDBC for batch mode first sounds
> > good.
> > > This will simplify the implementation and we can also remove the row
> kind
> > > first. We can claim this in the FLIP and docs, I will update the FLIP.
> > >
> > > Best,
> > > Shammon
> > >
> > >
> > > On Fri, Mar 3, 2023 at 2:36 PM Jingsong Li 
> > wrote:
> > >
> > > > Hi, Shammon,
> > > >
> > > > I took a look at JDBC `ResultSet` and `Statement`.  They are
> > > > complicated and have many interfaces. Some of the interfaces may not
> > > > be very suitable for streaming.
> > > >
> > > > I think maybe we can just implement JDBC for batch/olap only. It is
> > > > hard to have an integration for JDBC and streaming...
> > > >
> > > > Do you need to use JDBC in streaming mode? Or do we just implement
> > > > JDBC for batch only first?
> > > >
> > > > Best,
> > > > Jingsong
> > > >
> > > >
> > > > On Thu, Mar 2, 2023 at 6:22 PM Shammon FY  wrote:
> > > > >
> > > > > Hi
> > > > >
> > > > > Thanks for the feedback from Jingsong and Benchao.
> > > > >
> > > > > For @Jingsong
> > > > > > If the user does not cast into a FlinkResultSet, will there be
> > > > > serious consequences here (RowKind is ignored)?
> > > > >
> > > > > I agree with you that it's indeed a big deal if users ignore the
> row
> > > kind
> > > > > when they must know it. One idea that comes to my mind is we can
> add
> > an
> > > > > option such as `table.result.changelog-mode`, users can set it
> > through
> > > > > connection properties or set dynamic parameters. The option value
> can
> > > be
> > > > > `insert-only`, `upset` or `all` and the default value is
> > `insert-only`.
> > > > >
> > > > > If the result does not conform to the changelog mode, the jdbc
> driver
> > > > > throws an exception. What do you think?
> > > > >
> > > > >
> > > > > For @Benchao
> > > > > > Besides `java.sql.Driver`, have you considered also adding
> support
> > > for
> > > > > `javax.sql.DataSource` interface?
> > > > >
> > > > > I missed the `javax.sql.DataSource` and I have added it to the
> FLIP,
> > > > thanks
> > > > > Benchao
> > > > >
> > > > >
> > > > > Best,
> > > > > Shammon
> > > > >
> > > > > On Wed, Mar 1, 2023 at 7:57 PM Benchao Li 
> > > wrote:
> > > > >
> > > > > > +1 for the FLIP, thanks Shammon for driving this.
> > > > > >
> > > > > > JDBC is quite useful in OLAP scenarios, supporting JDBC would
> > enable
> > > > Flink
> > > > > > to be used with existing tools, such as Tableau.
> > > > > >
> > > > > > Regarding the JDBC interfaces listed in the FLIP, I think they
> > looks
> > > > good
> > > > > > already. Besides `java.sql.Driver`, have you considered also
> adding
> > > > support
> > > > > > for `javax.sql.DataSource` interface?
> > > > > >
> > > > > > Jingsong Li  于2023年3月1日周三 17:53写道：
> > > > > >
> > > > > > > Thanks Shammon for driving.
> > > > > > >
> > > > > > > Big +1 for this.
> > > > > > >
> > > > > > > I heard that many users want to use FlinkGateway + JDBC to do
> > some
> > > > > > > queries, but at present, only Hive JDBC can be used. It is Hive
> > > > > > > dialect by default, and the experience is also different from
> > > > > > > FlinkSQL. We need to have our own JDBC.
> > > > > > >
> > > > > > > I took a look at your `Public Interface` part, only
> > > > > > > `FlinkResultSet.getRowKind` is a true new interface, others are
> > > just
> > > > > > > implementations.
> > > > > > >
> > > > > > > If the user does not cast into a FlinkResultSet, will there be
> > > > serious
> > > > > > > consequences here (RowKind is ignored)?
> > > > > > >
> > > > > > > Best,
> > > > > > > Jingsong
> > > > > > >
> > > > > > > On Wed, Mar 1, 2023 at 4:59 PM Shammon FY 
> > > wrote:
> > > > > > > >
> > > > > > > > Hi devs,
> > > > > > > >
> > > 

Re: [ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Yuxin Tan]

Congratulations, Yuxia!

Best,
Yuxin


Jark Wu  于2023年3月13日周一 10:26写道：

> Hi, everyone
>
> On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new Flink
> Committer.
>
> Yuxia has been continuously contributing to the Flink project for almost
> two
> years, authored and reviewed hundreds of PRs over this time. He is
> currently
> the core maintainer of the Hive component, where he contributed many
> valuable
> features, including the Hive dialect with 95% compatibility and small file
> compaction.
> In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
> integrate
> Flink with data lakes. He actively participated in dev discussions and
> answered
> many questions on the user mailing list.
>
> Please join me in congratulating Yuxia Luo for becoming a Flink Committer!
>
> Best,
> Jark Wu (on behalf of the Flink PMC)
>

[ANNOUNCE] New Apache Flink Committer - Yuxia Luo

[Jark Wu]

Hi, everyone

On behalf of the PMC, I'm very happy to announce Yuxia Luo as a new Flink
Committer.

Yuxia has been continuously contributing to the Flink project for almost
two
years, authored and reviewed hundreds of PRs over this time. He is
currently
the core maintainer of the Hive component, where he contributed many
valuable
features, including the Hive dialect with 95% compatibility and small file
compaction.
In addition, Yuxia driven FLIP-282 (DELETE & UPDATE API) to better
integrate
Flink with data lakes. He actively participated in dev discussions and
answered
many questions on the user mailing list.

Please join me in congratulating Yuxia Luo for becoming a Flink Committer!

Best,
Jark Wu (on behalf of the Flink PMC)

[jira] [Created] (FLINK-31409) Hive connector should use public interfaces for Hive dialct

[luoyuxia (Jira)]

luoyuxia created FLINK-31409:


 Summary: Hive connector should use public interfaces for Hive 
dialct
 Key: FLINK-31409
 URL: https://issues.apache.org/jira/browse/FLINK-31409
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive
Reporter: luoyuxia


Currently, for the Hive dialect part in Hive connector, it depends much 
internal interfaces in flink-table-planner or other module. We should avoid it 
and use public interfaces proposed in  
[FLIP-216|[https://cwiki.apache.org/confluence/display/FLINK/FLIP-216%3A++Introduce+pluggable+dialect+and+plan+for+migrating+Hive+dialect]]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Flink metric collection

[Atul Lal]

Hi everyone,

Thank you very much for the prompt responses and help. I was able to get
the MetricFetcher connected with my thread and am successfully able to
collect metrics needed for my research and experiments.

Thank you,
Atul

On Thu, Mar 9, 2023 at 7:53 PM Atul Lal  wrote:

> Hi everyone,
>
> I also think the reporter won't work for my use case, it could work but
> then I am trying to modify the placement of operators on taskmanagers
> manually based on my some decisions made from metric collection, reporter
> makes it harder to do so. Right now, I have REST APIs, but I was thinking
> if in some way I could create a new MetricFetcher object and use that for
> collecting metrics instead of REST APIs.
>
> Is it possible to use MetricFetcher?
>
> Thanks a lot for the help,
> Atul
>
> On Thu, Mar 9, 2023 at 7:39 PM Mason Chen  wrote:
>
>> Hi all,
>>
>> Metric reporter may be useful if you only need per component level metrics
>> like jobmanager and each taskmanager since the metric reporter runs in
>> each
>> component. However, for an aggregated job level view of metrics, there is
>> no better out-of-the-box/user-facing way to get metrics than the REST API.
>> A good example in using the REST API is the k8s operator autoscaler which
>> needs to scrape metrics:
>>
>> https://github.com/apache/flink-kubernetes-operator/tree/main/flink-kubernetes-operator-autoscaler/src/main/java/org/apache/flink/kubernetes/operator/autoscaler
>> .
>>
>> Best,
>> Mason
>>
>> On Thu, Mar 9, 2023 at 4:34 AM Hang Ruan  wrote:
>>
>> > Hi, Atul,
>> >
>> > I think the metric reporter[1] will be helpful for you.
>> >
>> > Best,
>> > Hang
>> >
>> > [1]
>> >
>> >
>> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/deployment/metric_reporters/
>> >
>> > Atul Lal  于2023年3月9日周四 17:49写道：
>> >
>> > > Hi everyone,
>> > >
>> > > I am trying to do some experiments with Flink. I am trying to modify
>> the
>> > > source code of Flink for this experiment, and I am starting a process
>> > > thread from JobMaster.java constructor whenever a new job is started
>> in
>> > > Flink. In this thread, I want to monitor a few metrics related to the
>> job
>> > > and make some decisions based on it.
>> > >
>> > > Is there any way to collect metrics related to a job without using
>> REST
>> > > endpoints? Because I think using REST endpoints here is pointless as
>> this
>> > > is internal code running on JobMaster. If there is no other way than
>> > using
>> > > REST endpoints, is there any easy way to serialize or parse JSON
>> > responses
>> > > from those endpoints as the response structures are already defined in
>> > the
>> > > classes.
>> > >
>> > > I would really appreciate it if someone could help me with this.
>> > >
>> > > Thank you,
>> > > Atul
>> > >
>> >
>>
>

Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

[Shammon FY]

Hi Jing

Flink-Jdbc-Driver is mainly used to facilitate users to submit flink sql
jobs. I think we will not use Flink-Jdbc-Driver in Flink-Jdbc-Connector to
read/write data in general. If users can read/write specific tables by
Flink-Jdbc-Driver, they can use these tables in their flink jobs directly.
Thanks

Best,
Shammon


On Mon, Mar 13, 2023 at 6:21 AM Jing Ge  wrote:

> Hi, Shammon,
>
> This FLIP is very interesting. Thanks for driving it! Just out of
> curiosity, Could it be used in the Flink JDBC connector[1] after
> doing the appropriate implementation there?
>
> Best regards,
> Jing
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/
>
> On Fri, Mar 3, 2023 at 8:56 AM Shammon FY  wrote:
>
> > Hi,
> >
> > Thanks jinsong. I think implementing JDBC for batch mode first sounds
> good.
> > This will simplify the implementation and we can also remove the row kind
> > first. We can claim this in the FLIP and docs, I will update the FLIP.
> >
> > Best,
> > Shammon
> >
> >
> > On Fri, Mar 3, 2023 at 2:36 PM Jingsong Li 
> wrote:
> >
> > > Hi, Shammon,
> > >
> > > I took a look at JDBC `ResultSet` and `Statement`.  They are
> > > complicated and have many interfaces. Some of the interfaces may not
> > > be very suitable for streaming.
> > >
> > > I think maybe we can just implement JDBC for batch/olap only. It is
> > > hard to have an integration for JDBC and streaming...
> > >
> > > Do you need to use JDBC in streaming mode? Or do we just implement
> > > JDBC for batch only first?
> > >
> > > Best,
> > > Jingsong
> > >
> > >
> > > On Thu, Mar 2, 2023 at 6:22 PM Shammon FY  wrote:
> > > >
> > > > Hi
> > > >
> > > > Thanks for the feedback from Jingsong and Benchao.
> > > >
> > > > For @Jingsong
> > > > > If the user does not cast into a FlinkResultSet, will there be
> > > > serious consequences here (RowKind is ignored)?
> > > >
> > > > I agree with you that it's indeed a big deal if users ignore the row
> > kind
> > > > when they must know it. One idea that comes to my mind is we can add
> an
> > > > option such as `table.result.changelog-mode`, users can set it
> through
> > > > connection properties or set dynamic parameters. The option value can
> > be
> > > > `insert-only`, `upset` or `all` and the default value is
> `insert-only`.
> > > >
> > > > If the result does not conform to the changelog mode, the jdbc driver
> > > > throws an exception. What do you think?
> > > >
> > > >
> > > > For @Benchao
> > > > > Besides `java.sql.Driver`, have you considered also adding support
> > for
> > > > `javax.sql.DataSource` interface?
> > > >
> > > > I missed the `javax.sql.DataSource` and I have added it to the FLIP,
> > > thanks
> > > > Benchao
> > > >
> > > >
> > > > Best,
> > > > Shammon
> > > >
> > > > On Wed, Mar 1, 2023 at 7:57 PM Benchao Li 
> > wrote:
> > > >
> > > > > +1 for the FLIP, thanks Shammon for driving this.
> > > > >
> > > > > JDBC is quite useful in OLAP scenarios, supporting JDBC would
> enable
> > > Flink
> > > > > to be used with existing tools, such as Tableau.
> > > > >
> > > > > Regarding the JDBC interfaces listed in the FLIP, I think they
> looks
> > > good
> > > > > already. Besides `java.sql.Driver`, have you considered also adding
> > > support
> > > > > for `javax.sql.DataSource` interface?
> > > > >
> > > > > Jingsong Li  于2023年3月1日周三 17:53写道：
> > > > >
> > > > > > Thanks Shammon for driving.
> > > > > >
> > > > > > Big +1 for this.
> > > > > >
> > > > > > I heard that many users want to use FlinkGateway + JDBC to do
> some
> > > > > > queries, but at present, only Hive JDBC can be used. It is Hive
> > > > > > dialect by default, and the experience is also different from
> > > > > > FlinkSQL. We need to have our own JDBC.
> > > > > >
> > > > > > I took a look at your `Public Interface` part, only
> > > > > > `FlinkResultSet.getRowKind` is a true new interface, others are
> > just
> > > > > > implementations.
> > > > > >
> > > > > > If the user does not cast into a FlinkResultSet, will there be
> > > serious
> > > > > > consequences here (RowKind is ignored)?
> > > > > >
> > > > > > Best,
> > > > > > Jingsong
> > > > > >
> > > > > > On Wed, Mar 1, 2023 at 4:59 PM Shammon FY 
> > wrote:
> > > > > > >
> > > > > > > Hi devs,
> > > > > > >
> > > > > > > I'd like to start a discussion about FLIP-293: Introduce Flink
> > Jdbc
> > > > > > Driver
> > > > > > > For Sql Gateway[1].
> > > > > > >
> > > > > > > FLIP-275[2] supports remote sql client based on gateway, users
> > can
> > > > > > interact
> > > > > > > with gateway by flink console. However, for users who create
> > > session
> > > > > > > clusters with Flink, they'd like to use Jdbc Driver to interact
> > > with
> > > > > the
> > > > > > > gateway in their applications, such as olap queries..
> > > > > > >
> > > > > > > I have discussed this proposal with @shengkaifang and
> > @jinsonglee.
> > > In
> > > > > > this
> > > > > > > FLIP, we'd like to introduce Jdbc Driver for gateway. Users 

Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

[Jing Ge]

Hi, Shammon,

This FLIP is very interesting. Thanks for driving it! Just out of
curiosity, Could it be used in the Flink JDBC connector[1] after
doing the appropriate implementation there?

Best regards,
Jing

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/jdbc/

On Fri, Mar 3, 2023 at 8:56 AM Shammon FY  wrote:

> Hi,
>
> Thanks jinsong. I think implementing JDBC for batch mode first sounds good.
> This will simplify the implementation and we can also remove the row kind
> first. We can claim this in the FLIP and docs, I will update the FLIP.
>
> Best,
> Shammon
>
>
> On Fri, Mar 3, 2023 at 2:36 PM Jingsong Li  wrote:
>
> > Hi, Shammon,
> >
> > I took a look at JDBC `ResultSet` and `Statement`.  They are
> > complicated and have many interfaces. Some of the interfaces may not
> > be very suitable for streaming.
> >
> > I think maybe we can just implement JDBC for batch/olap only. It is
> > hard to have an integration for JDBC and streaming...
> >
> > Do you need to use JDBC in streaming mode? Or do we just implement
> > JDBC for batch only first?
> >
> > Best,
> > Jingsong
> >
> >
> > On Thu, Mar 2, 2023 at 6:22 PM Shammon FY  wrote:
> > >
> > > Hi
> > >
> > > Thanks for the feedback from Jingsong and Benchao.
> > >
> > > For @Jingsong
> > > > If the user does not cast into a FlinkResultSet, will there be
> > > serious consequences here (RowKind is ignored)?
> > >
> > > I agree with you that it's indeed a big deal if users ignore the row
> kind
> > > when they must know it. One idea that comes to my mind is we can add an
> > > option such as `table.result.changelog-mode`, users can set it through
> > > connection properties or set dynamic parameters. The option value can
> be
> > > `insert-only`, `upset` or `all` and the default value is `insert-only`.
> > >
> > > If the result does not conform to the changelog mode, the jdbc driver
> > > throws an exception. What do you think?
> > >
> > >
> > > For @Benchao
> > > > Besides `java.sql.Driver`, have you considered also adding support
> for
> > > `javax.sql.DataSource` interface?
> > >
> > > I missed the `javax.sql.DataSource` and I have added it to the FLIP,
> > thanks
> > > Benchao
> > >
> > >
> > > Best,
> > > Shammon
> > >
> > > On Wed, Mar 1, 2023 at 7:57 PM Benchao Li 
> wrote:
> > >
> > > > +1 for the FLIP, thanks Shammon for driving this.
> > > >
> > > > JDBC is quite useful in OLAP scenarios, supporting JDBC would enable
> > Flink
> > > > to be used with existing tools, such as Tableau.
> > > >
> > > > Regarding the JDBC interfaces listed in the FLIP, I think they looks
> > good
> > > > already. Besides `java.sql.Driver`, have you considered also adding
> > support
> > > > for `javax.sql.DataSource` interface?
> > > >
> > > > Jingsong Li  于2023年3月1日周三 17:53写道：
> > > >
> > > > > Thanks Shammon for driving.
> > > > >
> > > > > Big +1 for this.
> > > > >
> > > > > I heard that many users want to use FlinkGateway + JDBC to do some
> > > > > queries, but at present, only Hive JDBC can be used. It is Hive
> > > > > dialect by default, and the experience is also different from
> > > > > FlinkSQL. We need to have our own JDBC.
> > > > >
> > > > > I took a look at your `Public Interface` part, only
> > > > > `FlinkResultSet.getRowKind` is a true new interface, others are
> just
> > > > > implementations.
> > > > >
> > > > > If the user does not cast into a FlinkResultSet, will there be
> > serious
> > > > > consequences here (RowKind is ignored)?
> > > > >
> > > > > Best,
> > > > > Jingsong
> > > > >
> > > > > On Wed, Mar 1, 2023 at 4:59 PM Shammon FY 
> wrote:
> > > > > >
> > > > > > Hi devs,
> > > > > >
> > > > > > I'd like to start a discussion about FLIP-293: Introduce Flink
> Jdbc
> > > > > Driver
> > > > > > For Sql Gateway[1].
> > > > > >
> > > > > > FLIP-275[2] supports remote sql client based on gateway, users
> can
> > > > > interact
> > > > > > with gateway by flink console. However, for users who create
> > session
> > > > > > clusters with Flink, they'd like to use Jdbc Driver to interact
> > with
> > > > the
> > > > > > gateway in their applications, such as olap queries..
> > > > > >
> > > > > > I have discussed this proposal with @shengkaifang and
> @jinsonglee.
> > In
> > > > > this
> > > > > > FLIP, we'd like to introduce Jdbc Driver for gateway. Users can
> use
> > > > Jdbc
> > > > > > Driver to submit their queries and get results like a database in
> > their
> > > > > > applications.
> > > > > >
> > > > > > Looking forward to your feedback, thanks.
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway
> > > > > > [2]
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Shammon
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best,
> 

[jira] [Created] (FLINK-31408) Add EXACTLY_ONCE support to upsert-kafka

[Alex Sorokoumov (Jira)]

Alex Sorokoumov created FLINK-31408:
---

 Summary: Add EXACTLY_ONCE support to upsert-kafka
 Key: FLINK-31408
 URL: https://issues.apache.org/jira/browse/FLINK-31408
 Project: Flink
  Issue Type: New Feature
  Components: Connectors / Kafka
Reporter: Alex Sorokoumov


{{upsert-kafka}} connector should support optional {{EXACTLY_ONCE}} delivery 
semantics.

[upsert-kafka 
docs|https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/upsert-kafka/#consistency-guarantees]
 suggest that the connector handles duplicate records from 
{{{}AT_LEAST_ONCE{}}}. However, at least 2 reasons exist to configure the 
connector with {{{}EXACTLY_ONCE{}}}.

First, there might be other non-Flink topic consumers that would rather not 
have duplicated records.

Second, multiple {{upsert-kafka}} producers might cause keys to roll back to 
previous values. Consider a scenario with 2 producing jobs A and B, writing to 
the same topic with {{AT_LEAST_ONCE}} and a consuming job reading from the 
topic. Both producers write unique, monotonically increasing sequences to the 
same key. Job A writes {{x=a1,a2,a3,a4,a5…}} Job B writes 
{{{}x=b1,b2,b3,b4,b5,...{}}}. With this setup, we can have the following 
sequence:
 # Job A produces x=a5.
 # Job B produces x=b5.
 # Job A produces the duplicate write x= 5.

The consuming job would observe {{x}} going to {{{}a5{}}}, then to {{{}b5{}}}, 
then back {{{}a5{}}}. {{EXACTLY_ONCE}} would prevent this behavior.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31407) Upgrade Fabric8 version to 6.5.0

[Gyula Fora (Jira)]

Gyula Fora created FLINK-31407:
--

 Summary: Upgrade Fabric8 version to 6.5.0
 Key: FLINK-31407
 URL: https://issues.apache.org/jira/browse/FLINK-31407
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Gyula Fora


Fabric8 6.5.0 has been released recently with a number of major improvements:
[https://github.com/fabric8io/kubernetes-client/releases/tag/v6.5.0]

This is a very important version for the operator as it also fixes some 
outstanding issues with timing out informers.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31406) Do not delete jobgraph on scale only last-state upgrades

[Gyula Fora (Jira)]

Gyula Fora created FLINK-31406:
--

 Summary: Do not delete jobgraph on scale only last-state upgrades
 Key: FLINK-31406
 URL: https://issues.apache.org/jira/browse/FLINK-31406
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Gyula Fora


Currently the operator always deletes the jobgraph from HA metadata so that 
it's regenerated for last-state upgrades. 

This is unnecessary for scale only operations. Keeping the jobgraph can 
greately speed up startup time for some jobs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

[Yun Tang]

Hi Yuxin,

Thanks for this proposal.

From my understanding, this FLIP looks like a tiered based shuffle extension, 
which seems no need to bind with hybrid shuffle. Normal shuffle which writes 
data to disk could also benefit from this.

Secondly, since this targets to be an extension on hybrid shuffle which 
introduces a lots of changes, will it affect current design of pluggable remote 
shuffle service, such as Apache Celeborn [1].

Thirdly, based on my previous experiences on implementing a tiered based 
state-backend, the condition of min-reserve-space-fraction to kick local data 
to remote storage might not be a good idea in all cases, we still need to 
consider the absolute reserved disk storage. Take a 20GB local data disk as 
example, it might be a bit too late to kick the local data when only 1GB 
(20GB*5%) space left.

Last but not least, will we meet a concurrency problem when different subtasks 
within one process/node start to check the left disk space before deciding to 
write to local or remote?


[1] https://celeborn.apache.org/

Best
Yun Tang


From: Xia Sun 
Sent: Sunday, March 12, 2023 17:16
To: dev@flink.apache.org 
Subject: Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

Hi Yuxin,

Thanks for creating this FLIP!
I'm a flink user, and in our internal scenario we use the colocation
technology to run flink jobs and online service on the same machine
together. We found that flink jobs are occasionally affected by other
non-flink jobs (i.e. if the host disk space is full, that will result in
'No space left on device' error on flink jobs). This flip will really help
us to benefit from hybrid shuffle without being worried about insufficient
disk space problem.

And I also have a few questions.
1. If the same subpartition spans multiple different tiers, how to keep the
order of segments between different storage tiers (if necessary)?
2. In the process of writing to the local disk for a subpartition, what
will happen if the disk space is found to be full? Will it report an error
or automatically transfer to remote storage?
3. For remote storage, I noticed that it uses direct reading, which is
different from the other two, does the switching between different tiers
will bring overhead or waiting? In addition, compared to flink rss, which
optimizes data compression and small file merging to improve throughput and
relieve file system pressure, does the object storage system can meet the
performance requirements and concurrent access challenges of large-scale
batch jobs(parallelism > 1)?

Thanks,
Xia

Zhu Zhu  于2023年3月10日周五 16:44写道：

> Hi Yuxin,
>
> Thanks for creating this FLIP!
> The idea of tiered storage looks good. Instead of choosing one from
> multiple storages, it can help to balance between performance, cost and
> stability. It also has the potential to adaptively select proper tiers
> according to more runtime information, to achieve better performance
> and ease of use.
>
> I have a question about the tier finding of data reading. In the FLIP
> it proposes that the Read Client asks each storage tier whether a
> given segment exists in it, from higher priority tiers to lower priority
> ones. I'm a bit concerned about the cost of it, especially when data
> are written to low priority tiers. Do you have any evaluation of it?
> Is it possible to let the Reader Client know the location of the next
> segment when it has finished reading one segment? Or maybe just let it
> know whether the next segment is located in the same tier, if we can
> have the assumption that tier changing would not be very frequent.
>
> Thanks,
> Zhu
>
> Weihua Hu  于2023年3月10日周五 11:52写道：
> >
> > Thanks Yuxin for your explanation.
> >
> > That sounds reasonable. Looking forward to the new shuffle.
> >
> >
> > Best,
> > Weihua
> >
> >
> > On Fri, Mar 10, 2023 at 11:48 AM Yuxin Tan 
> wrote:
> >
> > > Hi, Weihua,
> > > Thanks for the questions and the ideas.
> > >
> > > > 1. How many performance regressions would there be if we only
> > > used remote storage?
> > >
> > > The new architecture can support to use remote storage only, but this
> > > FLIP target is to improve job stability. And the change in the FLIP has
> > > been significantly complex and the goal of the first version is to
> update
> > > Hybrid Shuffle to the new architecture and support remote storage as
> > > a supplement. The performance of this version is not the first
> priority,
> > > so we haven’t tested the performance of using only remote storage.
> > > If there are indeed regressions, we will keep optimizing the
> performance
> > > of the remote storages and improve it until only remote storage is
> > > available in the production environment.
> > >
> > > > 2. Shall we move the local data to remote storage if the producer is
> > > finished for a long time?
> > >
> > > I agree that it is a good idea, which can release task manager
> resources
> > > more timely. But moving data from TM local disk 

Re: [DISCUSS] FLIP-301: Hybrid Shuffle supports Remote Storage

[Xia Sun]

Hi Yuxin,

Thanks for creating this FLIP!
I'm a flink user, and in our internal scenario we use the colocation
technology to run flink jobs and online service on the same machine
together. We found that flink jobs are occasionally affected by other
non-flink jobs (i.e. if the host disk space is full, that will result in
'No space left on device' error on flink jobs). This flip will really help
us to benefit from hybrid shuffle without being worried about insufficient
disk space problem.

And I also have a few questions.
1. If the same subpartition spans multiple different tiers, how to keep the
order of segments between different storage tiers (if necessary)?
2. In the process of writing to the local disk for a subpartition, what
will happen if the disk space is found to be full? Will it report an error
or automatically transfer to remote storage?
3. For remote storage, I noticed that it uses direct reading, which is
different from the other two, does the switching between different tiers
will bring overhead or waiting? In addition, compared to flink rss, which
optimizes data compression and small file merging to improve throughput and
relieve file system pressure, does the object storage system can meet the
performance requirements and concurrent access challenges of large-scale
batch jobs(parallelism > 1)?

Thanks,
Xia

Zhu Zhu  于2023年3月10日周五 16:44写道：

> Hi Yuxin,
>
> Thanks for creating this FLIP!
> The idea of tiered storage looks good. Instead of choosing one from
> multiple storages, it can help to balance between performance, cost and
> stability. It also has the potential to adaptively select proper tiers
> according to more runtime information, to achieve better performance
> and ease of use.
>
> I have a question about the tier finding of data reading. In the FLIP
> it proposes that the Read Client asks each storage tier whether a
> given segment exists in it, from higher priority tiers to lower priority
> ones. I'm a bit concerned about the cost of it, especially when data
> are written to low priority tiers. Do you have any evaluation of it?
> Is it possible to let the Reader Client know the location of the next
> segment when it has finished reading one segment? Or maybe just let it
> know whether the next segment is located in the same tier, if we can
> have the assumption that tier changing would not be very frequent.
>
> Thanks,
> Zhu
>
> Weihua Hu  于2023年3月10日周五 11:52写道：
> >
> > Thanks Yuxin for your explanation.
> >
> > That sounds reasonable. Looking forward to the new shuffle.
> >
> >
> > Best,
> > Weihua
> >
> >
> > On Fri, Mar 10, 2023 at 11:48 AM Yuxin Tan 
> wrote:
> >
> > > Hi, Weihua,
> > > Thanks for the questions and the ideas.
> > >
> > > > 1. How many performance regressions would there be if we only
> > > used remote storage?
> > >
> > > The new architecture can support to use remote storage only, but this
> > > FLIP target is to improve job stability. And the change in the FLIP has
> > > been significantly complex and the goal of the first version is to
> update
> > > Hybrid Shuffle to the new architecture and support remote storage as
> > > a supplement. The performance of this version is not the first
> priority,
> > > so we haven’t tested the performance of using only remote storage.
> > > If there are indeed regressions, we will keep optimizing the
> performance
> > > of the remote storages and improve it until only remote storage is
> > > available in the production environment.
> > >
> > > > 2. Shall we move the local data to remote storage if the producer is
> > > finished for a long time?
> > >
> > > I agree that it is a good idea, which can release task manager
> resources
> > > more timely. But moving data from TM local disk to remote storage needs
> > > more detailed discussion and design, and it is easier to implement it
> based
> > > on the new architecture. Considering the complexity, the target focus,
> and
> > > the iteration cycle of the FLIP, we decide that the details are not
> > > included
> > > in the first version. We will extend and implement them in the
> subsequent
> > > versions.
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Weihua Hu  于2023年3月9日周四 11:22写道：
> > >
> > > > Hi, Yuxin
> > > >
> > > > Thanks for driving this FLIP.
> > > >
> > > > The remote storage shuffle could improve the stability of Batch jobs.
> > > >
> > > > In our internal scenario, we use a hybrid cluster to run both
> > > > Streaming(high priority)
> > > > and Batch jobs(low priority). When there is not enough
> resources(such as
> > > > cpu usage
> > > > reaches a threshold), the batch containers will be evicted. So this
> will
> > > > cause some re-run
> > > > of batch tasks.
> > > >
> > > > It would be a great help if the remote storage could address this.
> So I
> > > > have a few questions.
> > > >
> > > > 1. How many performance regressions would there be if we only used
> remote
> > > > storage?
> > > >
> > > > 2. In current design, the shuffle data segment will