Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Fabian Paul]

Thanks for the warm welcome, I am looking forward to continuing
working with you all.

Best,
Fabian

[jira] [Created] (FLINK-24918) Support to specify the data dir for state benchmark

[Aitozi (Jira)]

Aitozi created FLINK-24918:
--

 Summary: Support to specify the data dir for state benchmark 
 Key: FLINK-24918
 URL: https://issues.apache.org/jira/browse/FLINK-24918
 Project: Flink
  Issue Type: Improvement
  Components: Benchmarks
Reporter: Aitozi


{{StateBackendBenchmarkUtils}} use null as the parent dir to create temp dir, 
which will finally use the /tmp as the data dir. It has two downsides:

1. the /tmp dir often mount with tmpfs, which may store data in memory. it will 
affect the result of rocksdb benchmark
2. It can not support to use benchmark to measure the performance on a new 
device. 

So I purpose to enhance the state benchmark to support specify the default data 
dir. And avoiding to use the {{/tmp}} as default.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] New Apache Flink Committer - Leonard Xu

[Yun Gao]

Congratulations Leonard!

Best,
Yun


--
From:Terry Wang 
Send Time:2021 Nov. 16 (Tue.) 14:27
To:dev 
Cc:Dawid Wysakowicz 
Subject:Re: [ANNOUNCE] New Apache Flink Committer - Leonard Xu

Congratulations,  Leonard Xu! 
Well deserved!

Best,
Terry Wang



> 2021年11月16日 下午1:59，OpenInx  写道：
> 
> Congrats,  Leonard!
> 
> On Tue, Nov 16, 2021 at 12:10 AM Leonard Xu  wrote:
> 
>> Thank you all, it’s an honor to work in our community with everyone.
>> 
>> I will continue to contribute to the Flink community and Flink
>> ecology(e.g.Flink CDC Connectors).
>> 
>> Best,
>> Leonard
>> 
>> 
>> 
>>> 在 2021年11月15日，22:27，Dawid Wysakowicz  写道：
>>> 
>>> Congrats!
>>> 
>>> On 12/11/2021 05:12, Jark Wu wrote:
 Hi everyone,
 
 On behalf of the PMC, I'm very happy to announce Leonard Xu as a new
>> Flink
 committer.
 
 Leonard has been a very active contributor for more than two year,
>> authored
 150+ PRs and reviewed many PRs which is quite outstanding.
 Leonard mainly works on Flink SQL parts and drives several important
>> FLIPs,
 e.g. FLIP-132 (temporal table join) and FLIP-162 (correct time
>> behaviors).
 He is also the maintainer of flink-cdc-connectors[1] project which
>> helps a
 lot for users building a real-time data warehouse and data lake.
 
 Please join me in congratulating Leonard for becoming a Flink committer!
 
 Cheers,
 Jark Wu
 
 [1]: https://github.com/ververica/flink-cdc-connectors
 
>>> 
>> 
>> 

[jira] [Created] (FLINK-24917) YARNSessionFIFOSecuredITCase hangs on azure

[Yun Gao (Jira)]

Yun Gao created FLINK-24917:
---

 Summary: YARNSessionFIFOSecuredITCase hangs on azure
 Key: FLINK-24917
 URL: https://issues.apache.org/jira/browse/FLINK-24917
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN
Affects Versions: 1.14.0
Reporter: Yun Gao


{code:java}
Nov 15 23:50:37 [INFO] Running 
org.apache.flink.yarn.YARNSessionFIFOSecuredITCase
[Fatal Error] yarn-site.xml:1:1: Premature end of file.
Exception in thread "Thread-208" java.lang.RuntimeException: 
org.xml.sax.SAXParseException; systemId: 
file:/__w/1/s/flink-yarn-tests/target/test-classes/yarn-site.xml; lineNumber: 
1; columnNumber: 1; Premature end of file.
at 
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2338)
at 
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2185)
at 
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2102)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1068)
at 
org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refreshWithLoadedConfiguration(ServiceAuthorizationManager.java:139)
at 
org.apache.hadoop.security.authorize.ServiceAuthorizationManager.refresh(ServiceAuthorizationManager.java:126)
at org.apache.hadoop.ipc.Server.refreshServiceAcl(Server.java:454)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.createServer(ResourceLocalizationService.java:290)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceStart(ResourceLocalizationService.java:261)
at 
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at 
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceStart(ContainerManagerImpl.java:295)
at 
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at 
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at 
org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:197)
at 
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at 
org.apache.hadoop.yarn.server.MiniYARNCluster$NodeManagerWrapper$1.run(MiniYARNCluster.java:561)
Caused by: org.xml.sax.SAXParseException; systemId: 
file:/__w/1/s/flink-yarn-tests/target/test-classes/yarn-site.xml; lineNumber: 
1; columnNumber: 1; Premature end of file.
at 
com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257)
at 
com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2173)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2161)
at 
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2232)
... 16 more
Nov 15, 2021 11:50:58 PM com.google.inject.servlet.GuiceFilter setPipeline
 {code}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26571=logs=f450c1a5-64b1-5955-e215-49cb1ad5ec88=cc452273-9efa-565d-9db8-ef62a38a0c10=29706



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Definition of Done for Apache Flink

[Ingo Bürk]

Hi Joe,

thank you for starting this discussion. Having a common agreement on what
to expect from a PR for it to be merged is very much a worthwhile goal.

I'm slightly worried about the addition to the PR template. We shouldn't
make opening PRs even more difficult (unless it adds sufficient benefit).

There are two main benefits to have from using templates: requiring
information from authors to automate certain feedback, and serving as a
self-control checklist for contributors.

As it stands, a large number of PRs don't fill out the template, and I
haven't yet seen anyone not merge a PR over that, so de-facto we are not
using it for the former.

For the latter purpose of contributors having a checklist for themselves, I
think the current template is too long already and contains the wrong
content. Being short here is key if we want anyone to read it, and
personally I would cut it down significantly to a description and a couple
of checkboxes.

This isn't exactly the scope of your proposal, but personally I wouldn't
like to add even more questions that need to be filled out, especially
since they don't actually need to be filled out. It just creates an
annoying burden for contributors and is ignored by those who might benefit
most from reading it anyway.


Ingo


On Mon, Nov 15, 2021, 22:36 Johannes Moser  wrote:

> Dear Flink Community,
>
> We as the release managers of the 1.15 release are suggesting to introduce
> a “Definition of Done".
>
> Let me elaborate a bit on the reasons:
> * During the release process for 1.14 the stability of master was
> sometimes in a state that made contributing to Apache Flink a bad
> experience.
> * Some of the changes that have been contributed seem to be unusable by
> users because of defects.
> * Documentation is neglected which also leads to users unable to make use
> of changes. One of the reasons is, because documentation is often pushed to
> a later state.
>
> With this definition of done awareness and sensibility for these aspect
> should be increased. Both, for the ones who are committing and for the ones
> that are reviewing.
> We focus on code quality, testing and documentation. A shared
> understanding is created.
>
> The Definition of Done as suggested:
>
> -
> A PR is done and can be merged, when:
>
> 1. It is following the code contribution process
> 2. It is implemented according to the code style and quality guide.
> 3. If it has user facing changes the documentation has been updated
> according to the documentation style guide.
> 4. It is covered by tests.
> 5. All tests passed.
> -
>
> There are two PRs to illustrate the changes.
> https://github.com/apache/flink-web/pull/481 <
> https://github.com/apache/flink-web/pull/481>
> https://github.com/apache/flink/pull/17801 <
> https://github.com/apache/flink/pull/17801>
>
>
> It isn’t the goal to make it harder to get changes into Apache Flink. It
> is rather the opposite of making contributing and using Apache Flink a
> better experience.
> By creating awareness a push towards quality and usability should happen.
>
> I’m happy to hear your feedback.
>
> Best,
> Joe

[jira] [Created] (FLINK-24916) AvroExternalJarProgramITCase JVM crash on Azure

[Yun Gao (Jira)]

Yun Gao created FLINK-24916:
---

 Summary: AvroExternalJarProgramITCase JVM crash on Azure
 Key: FLINK-24916
 URL: https://issues.apache.org/jira/browse/FLINK-24916
 Project: Flink
  Issue Type: Bug
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.12.5
Reporter: Yun Gao


{code:java}
[ERROR] at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
[ERROR] at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
[ERROR] at 
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
[ERROR] at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
[ERROR] at 
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
[ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
[ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
[ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
[ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
[ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
[ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] at java.lang.reflect.Method.invoke(Method.java:498)
[ERROR] at 
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
[ERROR] at 
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
[ERROR] at 
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
[ERROR] at 
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
[ERROR] Caused by: 
org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM 
terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /__w/1/s/flink-formats/flink-avro/target && 
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m 
-Dmvn.forkNumber=1 -XX:+UseG1GC -jar 
/__w/1/s/flink-formats/flink-avro/target/surefire/surefirebooter7435584247231976356.jar
 /__w/1/s/flink-formats/flink-avro/target/surefire 
2021-11-15T21-36-43_499-jvmRun1 surefire7006265724133532959tmp 
surefire_616019020815389588133tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 127
 {code}
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26569=logs=3d12d40f-c62d-5ec4-6acc-0efe94cc3e89=5d6e4255-0ea8-5e2a-f52c-c881b7872361=13929]

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Paul Lam]

Congrats Fabian!

Best,
Paul Lam

> 2021年11月16日 11:26，Zhilong Hong  写道：
> 
> Congratulations, Fabian!
> 
> Best regards,
> Zhilong Hong
> 
> On Tue, Nov 16, 2021 at 10:19 AM Yangze Guo  wrote:
> 
>> Congrats & well deserved!
>> 
>> Best,
>> Yangze Guo
>> 
>> On Tue, Nov 16, 2021 at 10:18 AM Jing Zhang  wrote:
>>> 
>>> Congratulations Fabian!
>>> 
>>> Best,
>>> Jing Zhang
>>> 
>>> Yuepeng Pan  于2021年11月16日周二 上午10:16写道：
>>> 
 Congratulations Fabian!
 
 Best,
 Yuepeng Pan
 
 At 2021-11-15 21:17:13, "Arvid Heise"  wrote:
> Hi everyone,
> 
> On behalf of the PMC, I'm very happy to announce Fabian Paul as a new
 Flink
> committer.
> 
> Fabian Paul has been actively improving the connector ecosystem by
> migrating Kafka and ElasticSearch to the Sink interface and is
>> currently
> driving FLIP-191 [1] to tackle the sink compaction issue. While he is
> active on the project (authored 70 PRs and reviewed 60), it's also
>> worth
> highlighting that he has also been guiding external efforts, such as
>> the
> DeltaLake Flink connector or the Pinot sink in Bahir.
> 
> Please join me in congratulating Fabian for becoming a Flink
>> committer!
> 
> Best,
> 
> Arvid
> 
> [1]
> 
 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
 
>> 

Re: [ANNOUNCE] New Apache Flink Committer - Leonard Xu

[Terry Wang]

Congratulations,  Leonard Xu! 
Well deserved!

Best,
Terry Wang



> 2021年11月16日 下午1:59，OpenInx  写道：
> 
> Congrats,  Leonard!
> 
> On Tue, Nov 16, 2021 at 12:10 AM Leonard Xu  wrote:
> 
>> Thank you all, it’s an honor to work in our community with everyone.
>> 
>> I will continue to contribute to the Flink community and Flink
>> ecology(e.g.Flink CDC Connectors).
>> 
>> Best,
>> Leonard
>> 
>> 
>> 
>>> 在 2021年11月15日，22:27，Dawid Wysakowicz  写道：
>>> 
>>> Congrats!
>>> 
>>> On 12/11/2021 05:12, Jark Wu wrote:
 Hi everyone,
 
 On behalf of the PMC, I'm very happy to announce Leonard Xu as a new
>> Flink
 committer.
 
 Leonard has been a very active contributor for more than two year,
>> authored
 150+ PRs and reviewed many PRs which is quite outstanding.
 Leonard mainly works on Flink SQL parts and drives several important
>> FLIPs,
 e.g. FLIP-132 (temporal table join) and FLIP-162 (correct time
>> behaviors).
 He is also the maintainer of flink-cdc-connectors[1] project which
>> helps a
 lot for users building a real-time data warehouse and data lake.
 
 Please join me in congratulating Leonard for becoming a Flink committer!
 
 Cheers,
 Jark Wu
 
 [1]: https://github.com/ververica/flink-cdc-connectors
 
>>> 
>> 
>> 

Re: [ANNOUNCE] New Apache Flink Committer - Leonard Xu

[OpenInx]

Congrats,  Leonard!

On Tue, Nov 16, 2021 at 12:10 AM Leonard Xu  wrote:

> Thank you all, it’s an honor to work in our community with everyone.
>
> I will continue to contribute to the Flink community and Flink
> ecology(e.g.Flink CDC Connectors).
>
> Best,
> Leonard
>
>
>
> > 在 2021年11月15日，22:27，Dawid Wysakowicz  写道：
> >
> > Congrats!
> >
> > On 12/11/2021 05:12, Jark Wu wrote:
> >> Hi everyone,
> >>
> >> On behalf of the PMC, I'm very happy to announce Leonard Xu as a new
> Flink
> >> committer.
> >>
> >> Leonard has been a very active contributor for more than two year,
> authored
> >> 150+ PRs and reviewed many PRs which is quite outstanding.
> >> Leonard mainly works on Flink SQL parts and drives several important
> FLIPs,
> >> e.g. FLIP-132 (temporal table join) and FLIP-162 (correct time
> behaviors).
> >> He is also the maintainer of flink-cdc-connectors[1] project which
> helps a
> >> lot for users building a real-time data warehouse and data lake.
> >>
> >> Please join me in congratulating Leonard for becoming a Flink committer!
> >>
> >> Cheers,
> >> Jark Wu
> >>
> >> [1]: https://github.com/ververica/flink-cdc-connectors
> >>
> >
>
>

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[OpenInx]

Congrats Jing!

On Tue, Nov 16, 2021 at 11:59 AM Terry Wang  wrote:

> Congratulations,  Jing!
> Well deserved!
>
> Best,
> Terry Wang
>
>
>
> > 2021年11月16日 上午11:27，Zhilong Hong  写道：
> >
> > Congratulations, Jing!
> >
> > Best regards,
> > Zhilong Hong
> >
> > On Mon, Nov 15, 2021 at 9:41 PM Martijn Visser 
> > wrote:
> >
> >> Congratulations Jing!
> >>
> >> On Mon, 15 Nov 2021 at 14:39, Timo Walther  wrote:
> >>
> >>> Hi everyone,
> >>>
> >>> On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
> >>> Flink committer.
> >>>
> >>> Jing has been very active in the Flink community esp. in the Table/SQL
> >>> area for quite some time: 81 PRs [1] in total and is also active on
> >>> answering questions on the user mailing list. She is currently
> >>> contributing a lot around the new windowing table-valued functions [2].
> >>>
> >>> Please join me in congratulating Jing Zhang for becoming a Flink
> >> committer!
> >>>
> >>> Thanks,
> >>> Timo
> >>>
> >>> [1] https://github.com/apache/flink/pulls/beyond1920
> >>> [2] https://issues.apache.org/jira/browse/FLINK-23997
> >>>
> >>
>
>

[jira] [Created] (FLINK-24915) StreamElementSerializer#deserialize(StreamElement reuse, DataInputView source) forgets to handle tag == TAG_STREAM_STATUS

[bx123 (Jira)]

bx123 created FLINK-24915:
-

 Summary: StreamElementSerializer#deserialize(StreamElement reuse, 
DataInputView source) forgets to handle tag == TAG_STREAM_STATUS
 Key: FLINK-24915
 URL: https://issues.apache.org/jira/browse/FLINK-24915
 Project: Flink
  Issue Type: Bug
Reporter: bx123


when StreamElement is reused, we also have to handle tag == TAG_WATERMARK as 
object reuse disabled do. 

See also Flink-5017.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Terry Wang]

Congratulations,  Jing! 
Well deserved!

Best,
Terry Wang



> 2021年11月16日 上午11:27，Zhilong Hong  写道：
> 
> Congratulations, Jing!
> 
> Best regards,
> Zhilong Hong
> 
> On Mon, Nov 15, 2021 at 9:41 PM Martijn Visser 
> wrote:
> 
>> Congratulations Jing!
>> 
>> On Mon, 15 Nov 2021 at 14:39, Timo Walther  wrote:
>> 
>>> Hi everyone,
>>> 
>>> On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
>>> Flink committer.
>>> 
>>> Jing has been very active in the Flink community esp. in the Table/SQL
>>> area for quite some time: 81 PRs [1] in total and is also active on
>>> answering questions on the user mailing list. She is currently
>>> contributing a lot around the new windowing table-valued functions [2].
>>> 
>>> Please join me in congratulating Jing Zhang for becoming a Flink
>> committer!
>>> 
>>> Thanks,
>>> Timo
>>> 
>>> [1] https://github.com/apache/flink/pulls/beyond1920
>>> [2] https://issues.apache.org/jira/browse/FLINK-23997
>>> 
>> 

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Zhilong Hong]

Congratulations, Jing!

Best regards,
Zhilong Hong

On Mon, Nov 15, 2021 at 9:41 PM Martijn Visser 
wrote:

> Congratulations Jing!
>
> On Mon, 15 Nov 2021 at 14:39, Timo Walther  wrote:
>
> > Hi everyone,
> >
> > On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
> > Flink committer.
> >
> > Jing has been very active in the Flink community esp. in the Table/SQL
> > area for quite some time: 81 PRs [1] in total and is also active on
> > answering questions on the user mailing list. She is currently
> > contributing a lot around the new windowing table-valued functions [2].
> >
> > Please join me in congratulating Jing Zhang for becoming a Flink
> committer!
> >
> > Thanks,
> > Timo
> >
> > [1] https://github.com/apache/flink/pulls/beyond1920
> > [2] https://issues.apache.org/jira/browse/FLINK-23997
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Zhilong Hong]

Congratulations, Fabian!

Best regards,
Zhilong Hong

On Tue, Nov 16, 2021 at 10:19 AM Yangze Guo  wrote:

> Congrats & well deserved!
>
> Best,
> Yangze Guo
>
> On Tue, Nov 16, 2021 at 10:18 AM Jing Zhang  wrote:
> >
> > Congratulations Fabian!
> >
> > Best,
> > Jing Zhang
> >
> > Yuepeng Pan  于2021年11月16日周二 上午10:16写道：
> >
> > > Congratulations Fabian!
> > >
> > > Best,
> > > Yuepeng Pan
> > >
> > > At 2021-11-15 21:17:13, "Arvid Heise"  wrote:
> > > >Hi everyone,
> > > >
> > > >On behalf of the PMC, I'm very happy to announce Fabian Paul as a new
> > > Flink
> > > >committer.
> > > >
> > > >Fabian Paul has been actively improving the connector ecosystem by
> > > >migrating Kafka and ElasticSearch to the Sink interface and is
> currently
> > > >driving FLIP-191 [1] to tackle the sink compaction issue. While he is
> > > >active on the project (authored 70 PRs and reviewed 60), it's also
> worth
> > > >highlighting that he has also been guiding external efforts, such as
> the
> > > >DeltaLake Flink connector or the Pinot sink in Bahir.
> > > >
> > > >Please join me in congratulating Fabian for becoming a Flink
> committer!
> > > >
> > > >Best,
> > > >
> > > >Arvid
> > > >
> > > >[1]
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
> > >
>

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Paul Lam]

Congrats Jing!

Best,
Paul Lam

> 2021年11月16日 10:48，Yun Tang  写道：
> 
> Congratulations, Jing!
> 
> Best
> Yun Tang
> 
> From: Yuepeng Pan 
> Sent: Tuesday, November 16, 2021 10:12
> To: dev@flink.apache.org 
> Cc: beyond1...@gmail.com 
> Subject: Re:[ANNOUNCE] New Apache Flink Committer - Jing Zhang
> 
> Congrats.
> 
> 
> Best,
> Yuepeng Pan
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> At 2021-11-15 21:39:47, "Timo Walther"  wrote:
>> Hi everyone,
>> 
>> On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
>> Flink committer.
>> 
>> Jing has been very active in the Flink community esp. in the Table/SQL
>> area for quite some time: 81 PRs [1] in total and is also active on
>> answering questions on the user mailing list. She is currently
>> contributing a lot around the new windowing table-valued functions [2].
>> 
>> Please join me in congratulating Jing Zhang for becoming a Flink committer!
>> 
>> Thanks,
>> Timo
>> 
>> [1] https://github.com/apache/flink/pulls/beyond1920
>> [2] https://issues.apache.org/jira/browse/FLINK-23997

Re: Re:[ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Yun Tang]

Congratulations, Jing!

Best
Yun Tang

From: Yuepeng Pan 
Sent: Tuesday, November 16, 2021 10:12
To: dev@flink.apache.org 
Cc: beyond1...@gmail.com 
Subject: Re:[ANNOUNCE] New Apache Flink Committer - Jing Zhang

Congrats.


Best,
Yuepeng Pan











At 2021-11-15 21:39:47, "Timo Walther"  wrote:
>Hi everyone,
>
>On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
>Flink committer.
>
>Jing has been very active in the Flink community esp. in the Table/SQL
>area for quite some time: 81 PRs [1] in total and is also active on
>answering questions on the user mailing list. She is currently
>contributing a lot around the new windowing table-valued functions [2].
>
>Please join me in congratulating Jing Zhang for becoming a Flink committer!
>
>Thanks,
>Timo
>
>[1] https://github.com/apache/flink/pulls/beyond1920
>[2] https://issues.apache.org/jira/browse/FLINK-23997

[jira] [Created] (FLINK-24914) Support implicit type conversion in SQL like "select int_field = varchar_field"

[xuyang (Jira)]

xuyang created FLINK-24914:
--

 Summary: Support implicit type conversion in SQL like "select 
int_field = varchar_field"
 Key: FLINK-24914
 URL: https://issues.apache.org/jira/browse/FLINK-24914
 Project: Flink
  Issue Type: Improvement
Reporter: xuyang


Before, "select int_field = varchar_field" will be converted to 
"BinaryStringData.equals (int xxx)", so we always get FALSE.

Currently, we temporarily forbid "=" and "<>" between numeric and (var)char 
fields in SQL API, just like what have been done in Table API.

In future, we can support the implicit type conversion in Flink SQL and let 
this kind of SQL have the correct result.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Yangze Guo]

Congrats & well deserved!

Best,
Yangze Guo

On Tue, Nov 16, 2021 at 10:18 AM Jing Zhang  wrote:
>
> Congratulations Fabian!
>
> Best,
> Jing Zhang
>
> Yuepeng Pan  于2021年11月16日周二 上午10:16写道：
>
> > Congratulations Fabian!
> >
> > Best,
> > Yuepeng Pan
> >
> > At 2021-11-15 21:17:13, "Arvid Heise"  wrote:
> > >Hi everyone,
> > >
> > >On behalf of the PMC, I'm very happy to announce Fabian Paul as a new
> > Flink
> > >committer.
> > >
> > >Fabian Paul has been actively improving the connector ecosystem by
> > >migrating Kafka and ElasticSearch to the Sink interface and is currently
> > >driving FLIP-191 [1] to tackle the sink compaction issue. While he is
> > >active on the project (authored 70 PRs and reviewed 60), it's also worth
> > >highlighting that he has also been guiding external efforts, such as the
> > >DeltaLake Flink connector or the Pinot sink in Bahir.
> > >
> > >Please join me in congratulating Fabian for becoming a Flink committer!
> > >
> > >Best,
> > >
> > >Arvid
> > >
> > >[1]
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
> >

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Yangze Guo]

Congrats & well deserved!

Best,
Yangze Guo

On Tue, Nov 16, 2021 at 10:14 AM Yuepeng Pan  wrote:
>
> Congrats.
>
>
> Best,
> Yuepeng Pan
>
>
>
>
>
>
>
>
>
>
>
> At 2021-11-15 21:39:47, "Timo Walther"  wrote:
> >Hi everyone,
> >
> >On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
> >Flink committer.
> >
> >Jing has been very active in the Flink community esp. in the Table/SQL
> >area for quite some time: 81 PRs [1] in total and is also active on
> >answering questions on the user mailing list. She is currently
> >contributing a lot around the new windowing table-valued functions [2].
> >
> >Please join me in congratulating Jing Zhang for becoming a Flink committer!
> >
> >Thanks,
> >Timo
> >
> >[1] https://github.com/apache/flink/pulls/beyond1920
> >[2] https://issues.apache.org/jira/browse/FLINK-23997

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Jing Zhang]

Congratulations Fabian!

Best,
Jing Zhang

Yuepeng Pan  于2021年11月16日周二 上午10:16写道：

> Congratulations Fabian!
>
> Best,
> Yuepeng Pan
>
> At 2021-11-15 21:17:13, "Arvid Heise"  wrote:
> >Hi everyone,
> >
> >On behalf of the PMC, I'm very happy to announce Fabian Paul as a new
> Flink
> >committer.
> >
> >Fabian Paul has been actively improving the connector ecosystem by
> >migrating Kafka and ElasticSearch to the Sink interface and is currently
> >driving FLIP-191 [1] to tackle the sink compaction issue. While he is
> >active on the project (authored 70 PRs and reviewed 60), it's also worth
> >highlighting that he has also been guiding external efforts, such as the
> >DeltaLake Flink connector or the Pinot sink in Bahir.
> >
> >Please join me in congratulating Fabian for becoming a Flink committer!
> >
> >Best,
> >
> >Arvid
> >
> >[1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
>

Re:[ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Yuepeng Pan]

Congratulations Fabian!

Best,
Yuepeng Pan

At 2021-11-15 21:17:13, "Arvid Heise"  wrote:
>Hi everyone,
>
>On behalf of the PMC, I'm very happy to announce Fabian Paul as a new Flink
>committer.
>
>Fabian Paul has been actively improving the connector ecosystem by
>migrating Kafka and ElasticSearch to the Sink interface and is currently
>driving FLIP-191 [1] to tackle the sink compaction issue. While he is
>active on the project (authored 70 PRs and reviewed 60), it's also worth
>highlighting that he has also been guiding external efforts, such as the
>DeltaLake Flink connector or the Pinot sink in Bahir.
>
>Please join me in congratulating Fabian for becoming a Flink committer!
>
>Best,
>
>Arvid
>
>[1]
>https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction

Re:[ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Yuepeng Pan]

Congrats.


Best,
Yuepeng Pan











At 2021-11-15 21:39:47, "Timo Walther"  wrote:
>Hi everyone,
>
>On behalf of the PMC, I'm very happy to announce Jing Zhang as a new 
>Flink committer.
>
>Jing has been very active in the Flink community esp. in the Table/SQL 
>area for quite some time: 81 PRs [1] in total and is also active on 
>answering questions on the user mailing list. She is currently 
>contributing a lot around the new windowing table-valued functions [2].
>
>Please join me in congratulating Jing Zhang for becoming a Flink committer!
>
>Thanks,
>Timo
>
>[1] https://github.com/apache/flink/pulls/beyond1920
>[2] https://issues.apache.org/jira/browse/FLINK-23997

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Shengkai Fang]

Congratulations Jing!

Best,
Shengkai

wenlong.lwl  于2021年11月16日周二 上午10:02写道：

> Congratulations Jing！
>
> Best，
> Wenlong
>
> On Tue, 16 Nov 2021 at 09:54, Kurt Young  wrote:
>
> > Congrats Jing!
> >
> > Best,
> > Kurt
> >
> > On Tue, Nov 16, 2021 at 9:52 AM Xintong Song 
> > wrote:
> >
> > > Congratulations~!
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Tue, Nov 16, 2021 at 9:50 AM Dian Fu  wrote:
> > >
> > > > Congratulations!
> > > >
> > > > Regards,
> > > > Dian
> > > >
> > > > On Tue, Nov 16, 2021 at 9:48 AM godfrey he 
> > wrote:
> > > >
> > > > > Congrats & well deserved, Jing!
> > > > >
> > > > > Best,
> > > > > Godfrey
> > > > >
> > > > > Leonard Xu  于2021年11月16日周二 上午12:11写道：
> > > > > >
> > > > > > Congratulations Jing! Well Deserved.
> > > > > >
> > > > > >
> > > > > > > 在 2021年11月15日，22:30，Dawid Wysakowicz 
> > 写道：
> > > > > > >
> > > > > > > Congrats!
> > > > > > >
> > > > > > > On 15/11/2021 14:39, Timo Walther wrote:
> > > > > > >> Hi everyone,
> > > > > > >>
> > > > > > >> On behalf of the PMC, I'm very happy to announce Jing Zhang
> as a
> > > new
> > > > > > >> Flink committer.
> > > > > > >>
> > > > > > >> Jing has been very active in the Flink community esp. in the
> > > > Table/SQL
> > > > > > >> area for quite some time: 81 PRs [1] in total and is also
> active
> > > on
> > > > > > >> answering questions on the user mailing list. She is currently
> > > > > > >> contributing a lot around the new windowing table-valued
> > functions
> > > > > [2].
> > > > > > >>
> > > > > > >> Please join me in congratulating Jing Zhang for becoming a
> Flink
> > > > > > >> committer!
> > > > > > >>
> > > > > > >> Thanks,
> > > > > > >> Timo
> > > > > > >>
> > > > > > >> [1] https://github.com/apache/flink/pulls/beyond1920
> > > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-23997
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[wenlong.lwl]

Congratulations Jing！

Best，
Wenlong

On Tue, 16 Nov 2021 at 09:54, Kurt Young  wrote:

> Congrats Jing!
>
> Best,
> Kurt
>
> On Tue, Nov 16, 2021 at 9:52 AM Xintong Song 
> wrote:
>
> > Congratulations~!
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Tue, Nov 16, 2021 at 9:50 AM Dian Fu  wrote:
> >
> > > Congratulations!
> > >
> > > Regards,
> > > Dian
> > >
> > > On Tue, Nov 16, 2021 at 9:48 AM godfrey he 
> wrote:
> > >
> > > > Congrats & well deserved, Jing!
> > > >
> > > > Best,
> > > > Godfrey
> > > >
> > > > Leonard Xu  于2021年11月16日周二 上午12:11写道：
> > > > >
> > > > > Congratulations Jing! Well Deserved.
> > > > >
> > > > >
> > > > > > 在 2021年11月15日，22:30，Dawid Wysakowicz 
> 写道：
> > > > > >
> > > > > > Congrats!
> > > > > >
> > > > > > On 15/11/2021 14:39, Timo Walther wrote:
> > > > > >> Hi everyone,
> > > > > >>
> > > > > >> On behalf of the PMC, I'm very happy to announce Jing Zhang as a
> > new
> > > > > >> Flink committer.
> > > > > >>
> > > > > >> Jing has been very active in the Flink community esp. in the
> > > Table/SQL
> > > > > >> area for quite some time: 81 PRs [1] in total and is also active
> > on
> > > > > >> answering questions on the user mailing list. She is currently
> > > > > >> contributing a lot around the new windowing table-valued
> functions
> > > > [2].
> > > > > >>
> > > > > >> Please join me in congratulating Jing Zhang for becoming a Flink
> > > > > >> committer!
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Timo
> > > > > >>
> > > > > >> [1] https://github.com/apache/flink/pulls/beyond1920
> > > > > >> [2] https://issues.apache.org/jira/browse/FLINK-23997
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[godfrey he]

Congratulations Fabian!

Best,
Godfrey

Xintong Song  于2021年11月16日周二 上午9:53写道：
>
> Congratulations~!
>
> Thank you~
>
> Xintong Song
>
>
>
> On Tue, Nov 16, 2021 at 9:47 AM Qingsheng Ren  wrote:
>
> > Congratulations Fabian!
> >
> > --
> > Best Regards,
> >
> > Qingsheng Ren
> > Email: renqs...@gmail.com
> > On Nov 15, 2021, 9:17 PM +0800, Arvid Heise , wrote:
> > > Hi everyone,
> > >
> > > On behalf of the PMC, I'm very happy to announce Fabian Paul as a new
> > Flink
> > > committer.
> > >
> > > Fabian Paul has been actively improving the connector ecosystem by
> > > migrating Kafka and ElasticSearch to the Sink interface and is currently
> > > driving FLIP-191 [1] to tackle the sink compaction issue. While he is
> > > active on the project (authored 70 PRs and reviewed 60), it's also worth
> > > highlighting that he has also been guiding external efforts, such as the
> > > DeltaLake Flink connector or the Pinot sink in Bahir.
> > >
> > > Please join me in congratulating Fabian for becoming a Flink committer!
> > >
> > > Best,
> > >
> > > Arvid
> > >
> > > [1]
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
> >

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Kurt Young]

Congrats Jing!

Best,
Kurt

On Tue, Nov 16, 2021 at 9:52 AM Xintong Song  wrote:

> Congratulations~!
>
> Thank you~
>
> Xintong Song
>
>
>
> On Tue, Nov 16, 2021 at 9:50 AM Dian Fu  wrote:
>
> > Congratulations!
> >
> > Regards,
> > Dian
> >
> > On Tue, Nov 16, 2021 at 9:48 AM godfrey he  wrote:
> >
> > > Congrats & well deserved, Jing!
> > >
> > > Best,
> > > Godfrey
> > >
> > > Leonard Xu  于2021年11月16日周二 上午12:11写道：
> > > >
> > > > Congratulations Jing! Well Deserved.
> > > >
> > > >
> > > > > 在 2021年11月15日，22:30，Dawid Wysakowicz  写道：
> > > > >
> > > > > Congrats!
> > > > >
> > > > > On 15/11/2021 14:39, Timo Walther wrote:
> > > > >> Hi everyone,
> > > > >>
> > > > >> On behalf of the PMC, I'm very happy to announce Jing Zhang as a
> new
> > > > >> Flink committer.
> > > > >>
> > > > >> Jing has been very active in the Flink community esp. in the
> > Table/SQL
> > > > >> area for quite some time: 81 PRs [1] in total and is also active
> on
> > > > >> answering questions on the user mailing list. She is currently
> > > > >> contributing a lot around the new windowing table-valued functions
> > > [2].
> > > > >>
> > > > >> Please join me in congratulating Jing Zhang for becoming a Flink
> > > > >> committer!
> > > > >>
> > > > >> Thanks,
> > > > >> Timo
> > > > >>
> > > > >> [1] https://github.com/apache/flink/pulls/beyond1920
> > > > >> [2] https://issues.apache.org/jira/browse/FLINK-23997
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Xintong Song]

Congratulations~!

Thank you~

Xintong Song



On Tue, Nov 16, 2021 at 9:47 AM Qingsheng Ren  wrote:

> Congratulations Fabian!
>
> --
> Best Regards,
>
> Qingsheng Ren
> Email: renqs...@gmail.com
> On Nov 15, 2021, 9:17 PM +0800, Arvid Heise , wrote:
> > Hi everyone,
> >
> > On behalf of the PMC, I'm very happy to announce Fabian Paul as a new
> Flink
> > committer.
> >
> > Fabian Paul has been actively improving the connector ecosystem by
> > migrating Kafka and ElasticSearch to the Sink interface and is currently
> > driving FLIP-191 [1] to tackle the sink compaction issue. While he is
> > active on the project (authored 70 PRs and reviewed 60), it's also worth
> > highlighting that he has also been guiding external efforts, such as the
> > DeltaLake Flink connector or the Pinot sink in Bahir.
> >
> > Please join me in congratulating Fabian for becoming a Flink committer!
> >
> > Best,
> >
> > Arvid
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
>

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Xintong Song]

Congratulations~!

Thank you~

Xintong Song



On Tue, Nov 16, 2021 at 9:50 AM Dian Fu  wrote:

> Congratulations!
>
> Regards,
> Dian
>
> On Tue, Nov 16, 2021 at 9:48 AM godfrey he  wrote:
>
> > Congrats & well deserved, Jing!
> >
> > Best,
> > Godfrey
> >
> > Leonard Xu  于2021年11月16日周二 上午12:11写道：
> > >
> > > Congratulations Jing! Well Deserved.
> > >
> > >
> > > > 在 2021年11月15日，22:30，Dawid Wysakowicz  写道：
> > > >
> > > > Congrats!
> > > >
> > > > On 15/11/2021 14:39, Timo Walther wrote:
> > > >> Hi everyone,
> > > >>
> > > >> On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
> > > >> Flink committer.
> > > >>
> > > >> Jing has been very active in the Flink community esp. in the
> Table/SQL
> > > >> area for quite some time: 81 PRs [1] in total and is also active on
> > > >> answering questions on the user mailing list. She is currently
> > > >> contributing a lot around the new windowing table-valued functions
> > [2].
> > > >>
> > > >> Please join me in congratulating Jing Zhang for becoming a Flink
> > > >> committer!
> > > >>
> > > >> Thanks,
> > > >> Timo
> > > >>
> > > >> [1] https://github.com/apache/flink/pulls/beyond1920
> > > >> [2] https://issues.apache.org/jira/browse/FLINK-23997
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Dian Fu]

Congratulations!

Regards,
Dian

On Tue, Nov 16, 2021 at 9:47 AM Qingsheng Ren  wrote:

> Congratulations Fabian!
>
> --
> Best Regards,
>
> Qingsheng Ren
> Email: renqs...@gmail.com
> On Nov 15, 2021, 9:17 PM +0800, Arvid Heise , wrote:
> > Hi everyone,
> >
> > On behalf of the PMC, I'm very happy to announce Fabian Paul as a new
> Flink
> > committer.
> >
> > Fabian Paul has been actively improving the connector ecosystem by
> > migrating Kafka and ElasticSearch to the Sink interface and is currently
> > driving FLIP-191 [1] to tackle the sink compaction issue. While he is
> > active on the project (authored 70 PRs and reviewed 60), it's also worth
> > highlighting that he has also been guiding external efforts, such as the
> > DeltaLake Flink connector or the Pinot sink in Bahir.
> >
> > Please join me in congratulating Fabian for becoming a Flink committer!
> >
> > Best,
> >
> > Arvid
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
>

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Dian Fu]

Congratulations!

Regards,
Dian

On Tue, Nov 16, 2021 at 9:48 AM godfrey he  wrote:

> Congrats & well deserved, Jing!
>
> Best,
> Godfrey
>
> Leonard Xu  于2021年11月16日周二 上午12:11写道：
> >
> > Congratulations Jing! Well Deserved.
> >
> >
> > > 在 2021年11月15日，22:30，Dawid Wysakowicz  写道：
> > >
> > > Congrats!
> > >
> > > On 15/11/2021 14:39, Timo Walther wrote:
> > >> Hi everyone,
> > >>
> > >> On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
> > >> Flink committer.
> > >>
> > >> Jing has been very active in the Flink community esp. in the Table/SQL
> > >> area for quite some time: 81 PRs [1] in total and is also active on
> > >> answering questions on the user mailing list. She is currently
> > >> contributing a lot around the new windowing table-valued functions
> [2].
> > >>
> > >> Please join me in congratulating Jing Zhang for becoming a Flink
> > >> committer!
> > >>
> > >> Thanks,
> > >> Timo
> > >>
> > >> [1] https://github.com/apache/flink/pulls/beyond1920
> > >> [2] https://issues.apache.org/jira/browse/FLINK-23997
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[godfrey he]

Congrats & well deserved, Jing!

Best,
Godfrey

Leonard Xu  于2021年11月16日周二 上午12:11写道：
>
> Congratulations Jing! Well Deserved.
>
>
> > 在 2021年11月15日，22:30，Dawid Wysakowicz  写道：
> >
> > Congrats!
> >
> > On 15/11/2021 14:39, Timo Walther wrote:
> >> Hi everyone,
> >>
> >> On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
> >> Flink committer.
> >>
> >> Jing has been very active in the Flink community esp. in the Table/SQL
> >> area for quite some time: 81 PRs [1] in total and is also active on
> >> answering questions on the user mailing list. She is currently
> >> contributing a lot around the new windowing table-valued functions [2].
> >>
> >> Please join me in congratulating Jing Zhang for becoming a Flink
> >> committer!
> >>
> >> Thanks,
> >> Timo
> >>
> >> [1] https://github.com/apache/flink/pulls/beyond1920
> >> [2] https://issues.apache.org/jira/browse/FLINK-23997
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Qingsheng Ren]

Congratulations Fabian!

--
Best Regards,

Qingsheng Ren
Email: renqs...@gmail.com
On Nov 15, 2021, 9:17 PM +0800, Arvid Heise , wrote:
> Hi everyone,
>
> On behalf of the PMC, I'm very happy to announce Fabian Paul as a new Flink
> committer.
>
> Fabian Paul has been actively improving the connector ecosystem by
> migrating Kafka and ElasticSearch to the Sink interface and is currently
> driving FLIP-191 [1] to tackle the sink compaction issue. While he is
> active on the project (authored 70 PRs and reviewed 60), it's also worth
> highlighting that he has also been guiding external efforts, such as the
> DeltaLake Flink connector or the Pinot sink in Bahir.
>
> Please join me in congratulating Fabian for becoming a Flink committer!
>
> Best,
>
> Arvid
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction

Re: [DISCUSS] Definition of Done for Apache Flink

[tison]

Hi Joe,

Thanks for starting this thread! It seems the proposal emphasizes
documentation and test coverage, which are normal practices in
releasing a large software.

It looks all good but the point is that how you'd like to apply it?

We have a checklist in the PR template and @flinkbot will toast a
checklist which, unfortunately, is almost ignored in most cases.
A process may work or help but all in all, it depends on committers
who moderate and merge the pull request.

The Rust community defines a Final Comment Period (FCP)[1]
for attention on documentation and robust behaviors. You may think
of it with our FLIP process.

For tests and every PR documents, updating the template and bot toast,
and letting the consensus becomes one in every committer is more important
and easy to exercise. A new process cannot ensure a PR is well tested
logically, it creates formalism.

Best,
tison.

[1]
https://rustc-dev-guide.rust-lang.org/getting-started.html#breaking-changes


Johannes Moser  于2021年11月16日周二 上午5:36写道：

> Dear Flink Community,
>
> We as the release managers of the 1.15 release are suggesting to introduce
> a “Definition of Done".
>
> Let me elaborate a bit on the reasons:
> * During the release process for 1.14 the stability of master was
> sometimes in a state that made contributing to Apache Flink a bad
> experience.
> * Some of the changes that have been contributed seem to be unusable by
> users because of defects.
> * Documentation is neglected which also leads to users unable to make use
> of changes. One of the reasons is, because documentation is often pushed to
> a later state.
>
> With this definition of done awareness and sensibility for these aspect
> should be increased. Both, for the ones who are committing and for the ones
> that are reviewing.
> We focus on code quality, testing and documentation. A shared
> understanding is created.
>
> The Definition of Done as suggested:
>
> -
> A PR is done and can be merged, when:
>
> 1. It is following the code contribution process
> 2. It is implemented according to the code style and quality guide.
> 3. If it has user facing changes the documentation has been updated
> according to the documentation style guide.
> 4. It is covered by tests.
> 5. All tests passed.
> -
>
> There are two PRs to illustrate the changes.
> https://github.com/apache/flink-web/pull/481 <
> https://github.com/apache/flink-web/pull/481>
> https://github.com/apache/flink/pull/17801 <
> https://github.com/apache/flink/pull/17801>
>
>
> It isn’t the goal to make it harder to get changes into Apache Flink. It
> is rather the opposite of making contributing and using Apache Flink a
> better experience.
> By creating awareness a push towards quality and usability should happen.
>
> I’m happy to hear your feedback.
>
> Best,
> Joe

[jira] [Created] (FLINK-24913) Upgrade KPL version in flink-connector-kinesis to support ARM

[Melody (Jira)]

Melody created FLINK-24913:
--

 Summary: Upgrade KPL version in flink-connector-kinesis to support 
ARM
 Key: FLINK-24913
 URL: https://issues.apache.org/jira/browse/FLINK-24913
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kinesis
Affects Versions: 1.14.0
 Environment: Java 11 + EMR + Flink + Kinesis.
The use case is using EMR Flink Application with c6g.4xlarge(Graviton ARM64) 
based instances to connect to Kinesis Data streams.
Reporter: Melody


KPL supports Graviton (ARM64) based EC2 instance since the [0.14.4 
release|[https://github.com/awslabs/amazon-kinesis-producer/releases/tag/v0.14.4|https://github.com/awslabs/amazon-kinesis-producer/releases/tag/v0.14.4).]][.|https://github.com/awslabs/amazon-kinesis-producer/releases/tag/v0.14.4).]
 However, the latest flink-connector-kinesis in Flink doesn't support 0.14.4 or 
higher version yet. Please upgrade the KPL to support ARM.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24912) Migrate state processor API to DataStream API

[Seth Wiesman (Jira)]

Seth Wiesman created FLINK-24912:


 Summary: Migrate state processor API to DataStream API
 Key: FLINK-24912
 URL: https://issues.apache.org/jira/browse/FLINK-24912
 Project: Flink
  Issue Type: Improvement
  Components: API / State Processor
Affects Versions: 1.15.0
Reporter: Seth Wiesman
Assignee: Jun Qin


Now that DataStream supports bounded execution, we should investigate migrating 
the State Processor API off DataSet. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24911) Enable line numbers in SQL Client

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-24911:
---

 Summary: Enable line numbers in SQL Client
 Key: FLINK-24911
 URL: https://issues.apache.org/jira/browse/FLINK-24911
 Project: Flink
  Issue Type: Sub-task
Reporter: Sergey Nuyanzin


Should be enabled/disabled via property {{sql-client.prompt.show-line-numbers}}
Also  add widget to make it possible to toggle with a key-stroke



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24910) Propagate the Calcite parser config to SQL Client

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-24910:
---

 Summary: Propagate the Calcite parser config to SQL Client
 Key: FLINK-24910
 URL: https://issues.apache.org/jira/browse/FLINK-24910
 Project: Flink
  Issue Type: Sub-task
Reporter: Sergey Nuyanzin


It's required to get Dialect specific info like keywords, sql quote identifier



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24909) SQL syntax highlighting in SQL Client

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-24909:
---

 Summary: SQL syntax highlighting in SQL Client
 Key: FLINK-24909
 URL: https://issues.apache.org/jira/browse/FLINK-24909
 Project: Flink
  Issue Type: Sub-task
Reporter: Sergey Nuyanzin


What to hightlight: keywords, quoted strings, sql identifier quoted string, 
line comments, block comments, hints.
Property {{sql-client.color-schema}} to set current highlighting schema



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24908) Improve SQL Error description for SQL Client

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-24908:
---

 Summary: Improve SQL Error description for SQL Client
 Key: FLINK-24908
 URL: https://issues.apache.org/jira/browse/FLINK-24908
 Project: Flink
  Issue Type: Sub-task
Reporter: Sergey Nuyanzin


For instance for misprint like 
{code:sql}
ELECT 1;
{code}
The error looks like 
{noformat}
ELECT 1;
[ERROR] Could not execute SQL statement. Reason:
org.apache.calcite.runtime.CalciteException: Non-query expression encountered 
in illegal context
{noformat}
At the same time ParseException wrapped by CalciteException contains position 
of a problem word



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24907) Enable Side Output for late events on IntervalJoins

[Andrew Straussman (Jira)]

Andrew Straussman created FLINK-24907:
-

 Summary: Enable Side Output for late events on IntervalJoins
 Key: FLINK-24907
 URL: https://issues.apache.org/jira/browse/FLINK-24907
 Project: Flink
  Issue Type: Improvement
Reporter: Andrew Straussman


Currently it appears that interval joins silently drop late events: 
[https://github.com/apache/flink/blob/83a2541475228a4ff9e9a9def4049fb742353549/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/co/IntervalJoinOperator.java#L231]

It would be great if those events would instead get sent to a side output, like 
you can enable for other windowed join operations.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Johannes Moser]

Congratulations Fabian.

> On 15.11.2021, at 18:58, David Morávek  wrote:
> 
> Congratulations Fabian!!
> 
> Best,
> D.
> 
> On Mon, Nov 15, 2021 at 5:10 PM Leonard Xu  wrote:
> 
>> Congratulations Fabian!
>> 
>>> 在 2021年11月15日，22:50，Roman Khachatryan  写道：
>>> 
>>> Congratulations Fabian!
>>> 
>>> Regards,
>>> Roman
>>> 
>>> On Mon, Nov 15, 2021 at 3:26 PM Dawid Wysakowicz 
>> wrote:
 
 Congratulations Fabian!
 
 On 15/11/2021 15:22, Marios Trivyzas wrote:
> Congrats Fabian!
> 
> On Mon, Nov 15, 2021 at 3:02 PM Francesco Guardiani <
>> france...@ververica.com>
> wrote:
> 
>> Congratulations Fabian!
>> 
>> On Mon, Nov 15, 2021 at 2:29 PM Yun Gao >> 
>> wrote:
>> 
>>> Congratulations Fabian!
>>> 
>>> Best,
>>> Yun
>>> --Original Mail --
>>> Sender:Ingo Bürk 
>>> Send Date:Mon Nov 15 21:27:36 2021
>>> Recipients:dev 
>>> CC: , 
>>> Subject:Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul
>>> Congratulations, Fabian!
>>> 
>>> On Mon, Nov 15, 2021 at 2:17 PM Arvid Heise 
>> wrote:
>>> 
 Hi everyone,
 
 On behalf of the PMC, I'm very happy to announce Fabian Paul as a
>> new
>>> Flink
 committer.
 
 Fabian Paul has been actively improving the connector ecosystem by
 migrating Kafka and ElasticSearch to the Sink interface and is
>> currently
 driving FLIP-191 [1] to tackle the sink compaction issue. While he
>> is
 active on the project (authored 70 PRs and reviewed 60), it's also
>> worth
 highlighting that he has also been guiding external efforts, such as
>> the
 DeltaLake Flink connector or the Pinot sink in Bahir.
 
 Please join me in congratulating Fabian for becoming a Flink
>> committer!
 
 Best,
 
 Arvid
 
 [1]
 
 
>> 
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
> 
 
>> 
>> 

1.15 release cycle feature overview

[Johannes Moser]

Dear Flink Community.

In case you want to contribute something to the 1.15 Flink release I’d kindly 
asking you to add your features/changes to the 1.15 release overview page [1].
I, as one of the release managers, will keep track of the state of the changes.
In case you have questions or need support feel free to reach out to me.

Tomorrow, on the 16th of November, we will start with the 1.15. release 
bi-weeklies at 9am CET/4pm CST. See the sync meeting link on the overview page 
[1].
Everyone, especially those who are contributing, are welcome to join.

Best,
Joe

[1] https://cwiki.apache.org/confluence/display/FLINK/1.15+Release

[DISCUSS] Definition of Done for Apache Flink

[Johannes Moser]

Dear Flink Community,

We as the release managers of the 1.15 release are suggesting to introduce a 
“Definition of Done".

Let me elaborate a bit on the reasons:
* During the release process for 1.14 the stability of master was sometimes in 
a state that made contributing to Apache Flink a bad experience.
* Some of the changes that have been contributed seem to be unusable by users 
because of defects.
* Documentation is neglected which also leads to users unable to make use of 
changes. One of the reasons is, because documentation is often pushed to a 
later state.

With this definition of done awareness and sensibility for these aspect should 
be increased. Both, for the ones who are committing and for the ones that are 
reviewing.
We focus on code quality, testing and documentation. A shared understanding is 
created.

The Definition of Done as suggested:

-
A PR is done and can be merged, when:

1. It is following the code contribution process
2. It is implemented according to the code style and quality guide.
3. If it has user facing changes the documentation has been updated according 
to the documentation style guide.
4. It is covered by tests.
5. All tests passed.
-

There are two PRs to illustrate the changes.
https://github.com/apache/flink-web/pull/481 

https://github.com/apache/flink/pull/17801 
 


It isn’t the goal to make it harder to get changes into Apache Flink. It is 
rather the opposite of making contributing and using Apache Flink a better 
experience.
By creating awareness a push towards quality and usability should happen.

I’m happy to hear your feedback.

Best,
Joe

Re: [DISCUSS] Update Policy for old releases

[Thomas Weise]

Nice to see this discussion!

If we make Flink upgrades easy, why would users not want to upgrade? I
think most hesitation today stems from the fact that X.Y.0 releases
tend to break downstream in one way or the other due to unexpected
upstream changes. If so, it would be nice if we could address that
problem. For sure, if there was a choice between expending finite
resources on supporting more releases vs. reducing compatibility
issues in the next release, I would vote for the latter.

There is another reason why some users are stuck on old releases that
cannot be solved by the community: Heavily customized forks of Flink
are hard to maintain and naturally defer upgrade as much as possible.
I believe the solution to that is in user land (and not a community
burden): contribute changes back to Flink, participate in the
community to ensure robust extension hooks are provided etc.

Cheers,
Thomas

On Mon, Nov 15, 2021 at 3:03 AM Till Rohrmann  wrote:
>
> Hi everyone,
>
> I want to second Stephan's points. I think that supporting LTS versions
> rather fights symptoms than addressing the underlying problem that are
> incompatible/behaviour changing changes between versions. I believe if we
> can address this problem, then our Flink users will probably upgrade more
> frequently.
>
> Making upgrades as smooth as possible won't probably help all our users but
> we as a community also have to think about the costs of maintaining more
> Flink versions/supporting certain versions for a longer period of time. One
> thing is that we would need more complex processes that ensure that older
> Flink versions/LTS receives all the required bug fixes. I think this is
> already hard enough when only supporting the last two releases. Moreover,
> according to my experience, backporting fixes is usually not a problem if
> there are no merge conflicts. However, the older the Flink code is, the
> more likely it is that there are merge conflicts. In the worst case, issues
> need to be fixed in a completely different way than they are solved in
> master.
>
> Additionally, we have to consider that longer support periods will mean
> that we need to maintain our CI infrastructure and tooling the same period
> as well. To give you an example, we probably would still have to operate
> Travis if we had a LTS version. Given our existing problems with CI this
> would add a lot more problems to this heap. Moreover, supporting more
> versions would add more load to our CI infrastructure.
>
> All in all, I would expect that longer support periods would slow us down
> with other things. One of these things could be to make upgrades as smooth
> as possible so that more Flink users upgrade more frequently.
>
> @Tison: Concerning the problem that an older version contains a bug fix
> that is not contained yet in the latest release of a newer version, I think
> this is a valid problem. Right now, our users will have to check via JIRA
> whether their problems are solved in the latest version of the new release
> (e.g. when upgrading from 1.13.y to 1.14.x). Faster and more frequent
> releases could mitigate the problem. Faster and lower overhead releases
> could also allow us to release all versions in sync.
>
> Cheers,
> Till
>
> On Sat, Nov 13, 2021 at 1:40 AM tison  wrote:
>
> > Hi Stephan,
> >
> > Thanks for your email! I agree your points about compatibility and the
> > upgrade experience should be smooth.
> >
> > The problem raised by Pitor is that, even if we officially support two
> > latest versions, many users stay in an early version end-of-support.
> > So the downside "no one ends up using the other versions" doesn't
> > stand. I see that a number of companies like to test the latest version
> > but not apply on prod if it's not robust - 1.9 & 1.10 is less used.
> >
> > If a non-LTS version resolves users' (urgent) issues and the release itself
> > is robust, they will upgrade to that version. We have tried to support
> > Java 9 and so do other projects.
> >
> > I'm a fan to keep two latest supported versions and am willing to work on
> > improving compatibility to help users migrate. But if I make a choice
> > between
> > 4 supported versions and the LTS option, I like the LTS option.
> >
> > Here are several issues I met when trying to support a series of versions:
> >
> > 1. cheery-pick overhead grows, obviously.
> > 2. cheery-pick owner is unclear. I think committers play the role of
> > "reveal
> > all the details", and it makes (1) worse.
> > 3. version policy is harder. Generally a user wants to upgrade from
> > 1.14.3 (latest 1.14.x) to 1.15.2 (latest 1.15.x) without regression.
> > Given 1 & 2 the cherry-pick won't always get merged in time and
> > it's possible that when 1.14.3 released, 1.15.2 is preparing. We have
> > to do some simultaneous releases among 4 versions. And because there
> > are 4 versions, such simultaneous releases will be more required.
> >
> > If a company wants to maintain an early version, it's welcome 

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[David Morávek]

Congratulations Fabian!!

Best,
D.

On Mon, Nov 15, 2021 at 5:10 PM Leonard Xu  wrote:

> Congratulations Fabian!
>
> > 在 2021年11月15日，22:50，Roman Khachatryan  写道：
> >
> > Congratulations Fabian!
> >
> > Regards,
> > Roman
> >
> > On Mon, Nov 15, 2021 at 3:26 PM Dawid Wysakowicz 
> wrote:
> >>
> >> Congratulations Fabian!
> >>
> >> On 15/11/2021 15:22, Marios Trivyzas wrote:
> >>> Congrats Fabian!
> >>>
> >>> On Mon, Nov 15, 2021 at 3:02 PM Francesco Guardiani <
> france...@ververica.com>
> >>> wrote:
> >>>
>  Congratulations Fabian!
> 
>  On Mon, Nov 15, 2021 at 2:29 PM Yun Gao  >
>  wrote:
> 
> > Congratulations Fabian!
> >
> > Best,
> > Yun
> > --Original Mail --
> > Sender:Ingo Bürk 
> > Send Date:Mon Nov 15 21:27:36 2021
> > Recipients:dev 
> > CC: , 
> > Subject:Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul
> > Congratulations, Fabian!
> >
> > On Mon, Nov 15, 2021 at 2:17 PM Arvid Heise 
> wrote:
> >
> >> Hi everyone,
> >>
> >> On behalf of the PMC, I'm very happy to announce Fabian Paul as a
> new
> > Flink
> >> committer.
> >>
> >> Fabian Paul has been actively improving the connector ecosystem by
> >> migrating Kafka and ElasticSearch to the Sink interface and is
>  currently
> >> driving FLIP-191 [1] to tackle the sink compaction issue. While he
> is
> >> active on the project (authored 70 PRs and reviewed 60), it's also
>  worth
> >> highlighting that he has also been guiding external efforts, such as
>  the
> >> DeltaLake Flink connector or the Pinot sink in Bahir.
> >>
> >> Please join me in congratulating Fabian for becoming a Flink
> committer!
> >>
> >> Best,
> >>
> >> Arvid
> >>
> >> [1]
> >>
> >>
> 
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
> >>>
> >>
>
>

Re: [DISCUSS] Conventions on assertions to use in tests

[Francesco Guardiani]

> It is a bit unfortunate that we have tests that follow different patterns.
This, however, is mainly due to organic growth. I think the community
started with Junit4, then we chose to use Hamcrest because of its better
expressiveness.

That is fine, I'm sorry if my mail felt like a rant :)

> Personally, I don't have a strong preference for which testing tools to
use. The important bit is that we agree as a community, then document the
choice and finally stick to it. So before starting to use assertj, we
should probably align with the folks working on the Junit5 effort first.

As Arvid pointed out, using assertj might help the people working on the
junit5 effort as well, since assertj works seamlessly with junit4, junit5
and even other java testing frameworks.

> But I'm not sure if it's wise to change everything at once also
from the perspective of less active contributors. We may alleviate that
pain by providing good guides though. So maybe, we should also include a
temporal dimension into the discussion.

This is why I'm proposing a convention and not a rewrite of all the tests
at once, that's unfeasible. As you sad, we can provide guides, like in our
contribution guides, explaining our assertion convention, that is use
assertj or whatever other library we want to use and how. So then we can
ask contributors to use such assertion convention when they PR new tests or
when they modify existing ones. Something like that:
https://github.com/apache/flink-web/commit/87c572ccd4e0ae48eeff3eb15ad9847d302e659d

On Fri, Nov 12, 2021 at 5:07 PM Arvid Heise  wrote:

> JUnit5 migration is currently mostly prepared. The rules are being migrated
> [1] and Hang and Qingsheng have migrated most tests in their branch afaik
> (Kudos to them!).
>
> Using assertj would make migration easier as it's independent of the JUnit
> version. But the same can be said about hamcrest, albeit less expressive.
>
> I'm personally in favor of assertj (disclaimer I contributed to the project
> a bit). But I'm not sure if it's wise to change everything at once also
> from the perspective of less active contributors. We may alleviate that
> pain by providing good guides though. So maybe, we should also include a
> temporal dimension into the discussion.
>
> [1] https://github.com/apache/flink/pull/17556
>
> On Fri, Nov 12, 2021 at 3:58 PM Till Rohrmann 
> wrote:
>
> > Thanks for starting this discussion Francesco. I think there is a lot of
> > value in consistency because it makes it a lot easier to navigate and
> > contribute to the code base. The testing tools are definitely one
> important
> > aspect of consistency.
> >
> > It is a bit unfortunate that we have tests that follow different
> patterns.
> > This, however, is mainly due to organic growth. I think the community
> > started with Junit4, then we chose to use Hamcrest because of its better
> > expressiveness. Most recently, there was an effort started that aimed at
> > switching over to Junit5 [1, 2]. @Arvid Heise  knows
> > more about the current status.
> >
> > Personally, I don't have a strong preference for which testing tools to
> > use. The important bit is that we agree as a community, then document the
> > choice and finally stick to it. So before starting to use assertj, we
> > should probably align with the folks working on the Junit5 effort first.
> >
> > [1] https://lists.apache.org/thread/jsjvc2cqb91pyh47d4p6olk3c1vxqm3w
> > [2] https://lists.apache.org/thread/d9y5tzcl8wpk6ozmf8575qfzww450jpk
> >
> > Cheers,
> > Till
> >
> > On Fri, Nov 12, 2021 at 3:41 PM David Anderson 
> > wrote:
> >
> >> For what it's worth, I recently rewrote all of the tests in
> flink-training
> >> to use assertj, removing a mixture of junit4 assertions and hamcrest in
> >> the
> >> process. I chose assertj because I found it to be more expressive and
> made
> >> the tests more readable.
> >>
> >> +1 from me
> >>
> >> David
> >>
> >> On Fri, Nov 12, 2021 at 10:03 AM Francesco Guardiani <
> >> france...@ververica.com> wrote:
> >>
> >> > Hi all,
> >> >
> >> > I wonder If we have a convention of the testing tools (in particular
> >> > assertions) to use in our tests. If not, are modules free to decide
> on a
> >> > convention on their own?
> >> >
> >> > In case of table, we have a mixed bag of different assertions of all
> >> kinds,
> >> > sometimes mixed even in the same test:
> >> >
> >> >- Assertions from junit 4
> >> >- Assertions from junit 5
> >> >- Hamcrest
> >> >- Some custom assertions classes (e.g. RowDataHarnessAssertor)
> >> >- assert instructions
> >> >
> >> > The result is that most tests are very complicated to read and
> >> understand,
> >> > and we have a lot of copy pasted "assertion methods" all around our
> >> > codebase.
> >> >
> >> > For table in particular, I propose to introduce assertj [1] and
> develop
> >> a
> >> > couple of custom assertions [2] for the types we use most in our
> tests,
> >> > e.g. Row, RowData, DataType, LogicalType, etc... For 

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Leonard Xu]

Congratulations Jing! Well Deserved.


> 在 2021年11月15日，22:30，Dawid Wysakowicz  写道：
> 
> Congrats!
> 
> On 15/11/2021 14:39, Timo Walther wrote:
>> Hi everyone,
>> 
>> On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
>> Flink committer.
>> 
>> Jing has been very active in the Flink community esp. in the Table/SQL
>> area for quite some time: 81 PRs [1] in total and is also active on
>> answering questions on the user mailing list. She is currently
>> contributing a lot around the new windowing table-valued functions [2].
>> 
>> Please join me in congratulating Jing Zhang for becoming a Flink
>> committer!
>> 
>> Thanks,
>> Timo
>> 
>> [1] https://github.com/apache/flink/pulls/beyond1920
>> [2] https://issues.apache.org/jira/browse/FLINK-23997
> 

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Leonard Xu]

Congratulations Fabian!

> 在 2021年11月15日，22:50，Roman Khachatryan  写道：
> 
> Congratulations Fabian!
> 
> Regards,
> Roman
> 
> On Mon, Nov 15, 2021 at 3:26 PM Dawid Wysakowicz  
> wrote:
>> 
>> Congratulations Fabian!
>> 
>> On 15/11/2021 15:22, Marios Trivyzas wrote:
>>> Congrats Fabian!
>>> 
>>> On Mon, Nov 15, 2021 at 3:02 PM Francesco Guardiani 
>>> 
>>> wrote:
>>> 
 Congratulations Fabian!
 
 On Mon, Nov 15, 2021 at 2:29 PM Yun Gao 
 wrote:
 
> Congratulations Fabian!
> 
> Best,
> Yun
> --Original Mail --
> Sender:Ingo Bürk 
> Send Date:Mon Nov 15 21:27:36 2021
> Recipients:dev 
> CC: , 
> Subject:Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul
> Congratulations, Fabian!
> 
> On Mon, Nov 15, 2021 at 2:17 PM Arvid Heise  wrote:
> 
>> Hi everyone,
>> 
>> On behalf of the PMC, I'm very happy to announce Fabian Paul as a new
> Flink
>> committer.
>> 
>> Fabian Paul has been actively improving the connector ecosystem by
>> migrating Kafka and ElasticSearch to the Sink interface and is
 currently
>> driving FLIP-191 [1] to tackle the sink compaction issue. While he is
>> active on the project (authored 70 PRs and reviewed 60), it's also
 worth
>> highlighting that he has also been guiding external efforts, such as
 the
>> DeltaLake Flink connector or the Pinot sink in Bahir.
>> 
>> Please join me in congratulating Fabian for becoming a Flink committer!
>> 
>> Best,
>> 
>> Arvid
>> 
>> [1]
>> 
>> 
 https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
>>> 
>> 

Re: [ANNOUNCE] New Apache Flink Committer - Leonard Xu

[Leonard Xu]

Thank you all, it’s an honor to work in our community with everyone.

I will continue to contribute to the Flink community and Flink 
ecology(e.g.Flink CDC Connectors).

Best,
Leonard



> 在 2021年11月15日，22:27，Dawid Wysakowicz  写道：
> 
> Congrats!
> 
> On 12/11/2021 05:12, Jark Wu wrote:
>> Hi everyone,
>> 
>> On behalf of the PMC, I'm very happy to announce Leonard Xu as a new Flink
>> committer.
>> 
>> Leonard has been a very active contributor for more than two year, authored
>> 150+ PRs and reviewed many PRs which is quite outstanding.
>> Leonard mainly works on Flink SQL parts and drives several important FLIPs,
>> e.g. FLIP-132 (temporal table join) and FLIP-162 (correct time behaviors).
>> He is also the maintainer of flink-cdc-connectors[1] project which helps a
>> lot for users building a real-time data warehouse and data lake.
>> 
>> Please join me in congratulating Leonard for becoming a Flink committer!
>> 
>> Cheers,
>> Jark Wu
>> 
>> [1]: https://github.com/ververica/flink-cdc-connectors
>> 
> 

[jira] [Created] (FLINK-24906) Improve CSV format handling and support

[Martijn Visser (Jira)]

Martijn Visser created FLINK-24906:
--

 Summary: Improve CSV format handling and support
 Key: FLINK-24906
 URL: https://issues.apache.org/jira/browse/FLINK-24906
 Project: Flink
  Issue Type: Improvement
Reporter: Martijn Visser


This is an umbrella issue to track improvements how Flink supports and handles 
the CSV format



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24905) KDS implementation of Async Sink Table API

[Zichen Liu (Jira)]

Zichen Liu created FLINK-24905:
--

 Summary: KDS implementation of Async Sink Table API
 Key: FLINK-24905
 URL: https://issues.apache.org/jira/browse/FLINK-24905
 Project: Flink
  Issue Type: New Feature
  Components: Connectors / Common
Reporter: Zichen Liu
Assignee: Zichen Liu
 Fix For: 1.15.0


h2. Motivation

*User stories:*
 As a Flink user, I’d like to use Kinesis Data Streams as sink for my data 
pipeline.

*Scope:*
 * Implement an asynchronous sink for Kinesis Data Streams (KDS) by inheriting 
the AsyncSinkBase class. The implementation can for now reside in its own 
module in flink-connectors. The module and package name can be anything 
reasonable e.g. {{flink-connector-aws-kinesis}} for the module name and 
{{org.apache.flink.connector.aws.kinesis}} for the package name.
 * The implementation must use [the Kinesis Java 
Client|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/AmazonKinesisClient.html].
 * The implementation must allow users to configure the Kinesis Client, with 
reasonable default settings.
 * Implement an asynchornous sink writer for KDS by extending the 
AsyncSinkWriter. The implementation must deal with failed requests and retry 
them using the {{requeueFailedRequestEntry}} method. If possible, the 
implementation should batch multiple requests (PutRecordsRequestEntry objects) 
to KDS for increased throughput. The implemented Sink Writer will be used by 
the Sink class that will be created as part of this story.
 * Unit/Integration testing. Use Kinesalite (in-memory Kinesis simulation). We 
already use this in {{KinesisTableApiITCase}}.
 * Java / code-level docs.
 * End to end testing: add tests that hits a real AWS instance. (How to best 
donate resources to the Flink project to allow this to happen?)

h2. References

More details to be found 
[https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Roman Khachatryan]

Congratulations Fabian!

Regards,
Roman

On Mon, Nov 15, 2021 at 3:26 PM Dawid Wysakowicz  wrote:
>
> Congratulations Fabian!
>
> On 15/11/2021 15:22, Marios Trivyzas wrote:
> > Congrats Fabian!
> >
> > On Mon, Nov 15, 2021 at 3:02 PM Francesco Guardiani 
> > 
> > wrote:
> >
> >> Congratulations Fabian!
> >>
> >> On Mon, Nov 15, 2021 at 2:29 PM Yun Gao 
> >> wrote:
> >>
> >>> Congratulations Fabian!
> >>>
> >>> Best,
> >>> Yun
> >>>  --Original Mail --
> >>> Sender:Ingo Bürk 
> >>> Send Date:Mon Nov 15 21:27:36 2021
> >>> Recipients:dev 
> >>> CC: , 
> >>> Subject:Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul
> >>> Congratulations, Fabian!
> >>>
> >>> On Mon, Nov 15, 2021 at 2:17 PM Arvid Heise  wrote:
> >>>
>  Hi everyone,
> 
>  On behalf of the PMC, I'm very happy to announce Fabian Paul as a new
> >>> Flink
>  committer.
> 
>  Fabian Paul has been actively improving the connector ecosystem by
>  migrating Kafka and ElasticSearch to the Sink interface and is
> >> currently
>  driving FLIP-191 [1] to tackle the sink compaction issue. While he is
>  active on the project (authored 70 PRs and reviewed 60), it's also
> >> worth
>  highlighting that he has also been guiding external efforts, such as
> >> the
>  DeltaLake Flink connector or the Pinot sink in Bahir.
> 
>  Please join me in congratulating Fabian for becoming a Flink committer!
> 
>  Best,
> 
>  Arvid
> 
>  [1]
> 
> 
> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Dawid Wysakowicz]

Congrats!

On 15/11/2021 14:39, Timo Walther wrote:
> Hi everyone,
>
> On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
> Flink committer.
>
> Jing has been very active in the Flink community esp. in the Table/SQL
> area for quite some time: 81 PRs [1] in total and is also active on
> answering questions on the user mailing list. She is currently
> contributing a lot around the new windowing table-valued functions [2].
>
> Please join me in congratulating Jing Zhang for becoming a Flink
> committer!
>
> Thanks,
> Timo
>
> [1] https://github.com/apache/flink/pulls/beyond1920
> [2] https://issues.apache.org/jira/browse/FLINK-23997



OpenPGP_signature
Description: OpenPGP digital signature

Re: [ANNOUNCE] New Apache Flink Committer - Yangze Guo

[Dawid Wysakowicz]

Congrats!

On 15/11/2021 15:22, Marios Trivyzas wrote:
> Congrats Yangze!
>
> On Mon, Nov 15, 2021 at 1:34 PM Martijn Visser 
> wrote:
>
>> Congrats Yangze!
>>
>> On Mon, 15 Nov 2021 at 09:13, Yangze Guo  wrote:
>>
>>> Thank you all very much!
>>> It's my honor to work with you in such a great community.
>>>
>>> Best,
>>> Yangze Guo
>>>
>>> On Mon, Nov 15, 2021 at 2:08 PM Yang Wang  wrote:
 Congratulations Yangze!

 Best,
 Yang

 Yu Li  于2021年11月15日周一 下午2:01写道：
> Congratulations, Yangze!
>
> Best Regards,
> Yu
>
>
> On Mon, 15 Nov 2021 at 12:30, Rui Li  wrote:
>
>> Congratulations Yangze!
>>
>> On Mon, Nov 15, 2021 at 11:44 AM wenlong.lwl <
>> wenlong88@gmail.com
>> wrote:
>>
>>> Congratulations, Yangze!
>>>
>>> On Mon, 15 Nov 2021 at 10:37, Zhilong Hong 
>>> wrote:
 Congratulations, Yangze!

 On Mon, Nov 15, 2021 at 10:13 AM Qingsheng Ren <
>>> renqs...@gmail.com>
>>> wrote:
> Congratulations Yangze!
>
> --
> Best Regards,
>
> Qingsheng Ren
> Email: renqs...@gmail.com
> On Nov 12, 2021, 10:11 AM +0800, Xintong Song <
>>> tonysong...@gmail.com
>>> ,
> wrote:
>> Hi everyone,
>>
>> On behalf of the PMC, I'm very happy to announce Yangze Guo
>>> as a
>> new
> Flink
>> committer.
>>
>> Yangze has been consistently contributing to this project
>> for
>> almost
>>> 3
>> years. His contributions are mainly in the resource
>>> management and
>> deployment areas, represented by the fine-grained resource
>> management
 and
>> external resource framework. In addition to feature works,
>>> he's
>> also
> active
>> in miscellaneous contributions, including PR reviews,
>> document
> enhancement,
>> mailing list services and meetup/FF talks.
>>
>> Please join me in congratulating Yangze Guo for becoming a
>>> Flink
> committer!
>> Thank you~
>>
>> Xintong Song
>>
>> --
>> Best regards!
>> Rui Li
>>
>



OpenPGP_signature
Description: OpenPGP digital signature

Re: [ANNOUNCE] New Apache Flink Committer - Leonard Xu

[Dawid Wysakowicz]

Congrats!

On 12/11/2021 05:12, Jark Wu wrote:
> Hi everyone,
>
> On behalf of the PMC, I'm very happy to announce Leonard Xu as a new Flink
> committer.
>
> Leonard has been a very active contributor for more than two year, authored
> 150+ PRs and reviewed many PRs which is quite outstanding.
> Leonard mainly works on Flink SQL parts and drives several important FLIPs,
> e.g. FLIP-132 (temporal table join) and FLIP-162 (correct time behaviors).
> He is also the maintainer of flink-cdc-connectors[1] project which helps a
> lot for users building a real-time data warehouse and data lake.
>
> Please join me in congratulating Leonard for becoming a Flink committer!
>
> Cheers,
> Jark Wu
>
> [1]: https://github.com/ververica/flink-cdc-connectors
>



OpenPGP_signature
Description: OpenPGP digital signature

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Dawid Wysakowicz]

Congratulations Fabian!

On 15/11/2021 15:22, Marios Trivyzas wrote:
> Congrats Fabian!
>
> On Mon, Nov 15, 2021 at 3:02 PM Francesco Guardiani 
> wrote:
>
>> Congratulations Fabian!
>>
>> On Mon, Nov 15, 2021 at 2:29 PM Yun Gao 
>> wrote:
>>
>>> Congratulations Fabian!
>>>
>>> Best,
>>> Yun
>>>  --Original Mail --
>>> Sender:Ingo Bürk 
>>> Send Date:Mon Nov 15 21:27:36 2021
>>> Recipients:dev 
>>> CC: , 
>>> Subject:Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul
>>> Congratulations, Fabian!
>>>
>>> On Mon, Nov 15, 2021 at 2:17 PM Arvid Heise  wrote:
>>>
 Hi everyone,

 On behalf of the PMC, I'm very happy to announce Fabian Paul as a new
>>> Flink
 committer.

 Fabian Paul has been actively improving the connector ecosystem by
 migrating Kafka and ElasticSearch to the Sink interface and is
>> currently
 driving FLIP-191 [1] to tackle the sink compaction issue. While he is
 active on the project (authored 70 PRs and reviewed 60), it's also
>> worth
 highlighting that he has also been guiding external efforts, such as
>> the
 DeltaLake Flink connector or the Pinot sink in Bahir.

 Please join me in congratulating Fabian for becoming a Flink committer!

 Best,

 Arvid

 [1]


>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
>



OpenPGP_signature
Description: OpenPGP digital signature

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Shuo Cheng]

Congratulations Jing!

On Mon, Nov 15, 2021 at 10:23 PM Marios Trivyzas  wrote:

> Congrats Jing!
>
> On Mon, Nov 15, 2021 at 3:20 PM Till Rohrmann 
> wrote:
>
> > Congratulations Jing :-)
> >
> > Cheers,
> > Till
> >
> > On Mon, Nov 15, 2021 at 2:48 PM Martijn Visser 
> > wrote:
> >
> > > Congratulations Jing!
> > >
> > > On Mon, 15 Nov 2021 at 14:39, Timo Walther  wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
> > > > Flink committer.
> > > >
> > > > Jing has been very active in the Flink community esp. in the
> Table/SQL
> > > > area for quite some time: 81 PRs [1] in total and is also active on
> > > > answering questions on the user mailing list. She is currently
> > > > contributing a lot around the new windowing table-valued functions
> [2].
> > > >
> > > > Please join me in congratulating Jing Zhang for becoming a Flink
> > > committer!
> > > >
> > > > Thanks,
> > > > Timo
> > > >
> > > > [1] https://github.com/apache/flink/pulls/beyond1920
> > > > [2] https://issues.apache.org/jira/browse/FLINK-23997
> > > >
> > >
> >
>
>
> --
> Marios
>

Re: Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Yun Gao]

Congratulations Jing!

Best,
Yun



 --Original Mail --
Sender:Till Rohrmann 
Send Date:Mon Nov 15 22:20:52 2021
Recipients:dev 
CC:JING ZHANG 
Subject:Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang
Congratulations Jing :-)

Cheers,
Till

On Mon, Nov 15, 2021 at 2:48 PM Martijn Visser 
wrote:

> Congratulations Jing!
>
> On Mon, 15 Nov 2021 at 14:39, Timo Walther  wrote:
>
> > Hi everyone,
> >
> > On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
> > Flink committer.
> >
> > Jing has been very active in the Flink community esp. in the Table/SQL
> > area for quite some time: 81 PRs [1] in total and is also active on
> > answering questions on the user mailing list. She is currently
> > contributing a lot around the new windowing table-valued functions [2].
> >
> > Please join me in congratulating Jing Zhang for becoming a Flink
> committer!
> >
> > Thanks,
> > Timo
> >
> > [1] https://github.com/apache/flink/pulls/beyond1920
> > [2] https://issues.apache.org/jira/browse/FLINK-23997
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Yangze Guo

[Marios Trivyzas]

Congrats Yangze!

On Mon, Nov 15, 2021 at 1:34 PM Martijn Visser 
wrote:

> Congrats Yangze!
>
> On Mon, 15 Nov 2021 at 09:13, Yangze Guo  wrote:
>
> > Thank you all very much!
> > It's my honor to work with you in such a great community.
> >
> > Best,
> > Yangze Guo
> >
> > On Mon, Nov 15, 2021 at 2:08 PM Yang Wang  wrote:
> > >
> > > Congratulations Yangze!
> > >
> > > Best,
> > > Yang
> > >
> > > Yu Li  于2021年11月15日周一 下午2:01写道：
> > >>
> > >> Congratulations, Yangze!
> > >>
> > >> Best Regards,
> > >> Yu
> > >>
> > >>
> > >> On Mon, 15 Nov 2021 at 12:30, Rui Li  wrote:
> > >>
> > >> > Congratulations Yangze!
> > >> >
> > >> > On Mon, Nov 15, 2021 at 11:44 AM wenlong.lwl <
> wenlong88@gmail.com
> > >
> > >> > wrote:
> > >> >
> > >> > > Congratulations, Yangze!
> > >> > >
> > >> > > On Mon, 15 Nov 2021 at 10:37, Zhilong Hong 
> > wrote:
> > >> > >
> > >> > > > Congratulations, Yangze!
> > >> > > >
> > >> > > > On Mon, Nov 15, 2021 at 10:13 AM Qingsheng Ren <
> > renqs...@gmail.com>
> > >> > > wrote:
> > >> > > >
> > >> > > > > Congratulations Yangze!
> > >> > > > >
> > >> > > > > --
> > >> > > > > Best Regards,
> > >> > > > >
> > >> > > > > Qingsheng Ren
> > >> > > > > Email: renqs...@gmail.com
> > >> > > > > On Nov 12, 2021, 10:11 AM +0800, Xintong Song <
> > tonysong...@gmail.com
> > >> > >,
> > >> > > > > wrote:
> > >> > > > > > Hi everyone,
> > >> > > > > >
> > >> > > > > > On behalf of the PMC, I'm very happy to announce Yangze Guo
> > as a
> > >> > new
> > >> > > > > Flink
> > >> > > > > > committer.
> > >> > > > > >
> > >> > > > > > Yangze has been consistently contributing to this project
> for
> > >> > almost
> > >> > > 3
> > >> > > > > > years. His contributions are mainly in the resource
> > management and
> > >> > > > > > deployment areas, represented by the fine-grained resource
> > >> > management
> > >> > > > and
> > >> > > > > > external resource framework. In addition to feature works,
> > he's
> > >> > also
> > >> > > > > active
> > >> > > > > > in miscellaneous contributions, including PR reviews,
> document
> > >> > > > > enhancement,
> > >> > > > > > mailing list services and meetup/FF talks.
> > >> > > > > >
> > >> > > > > > Please join me in congratulating Yangze Guo for becoming a
> > Flink
> > >> > > > > committer!
> > >> > > > > >
> > >> > > > > > Thank you~
> > >> > > > > >
> > >> > > > > > Xintong Song
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >> >
> > >> > --
> > >> > Best regards!
> > >> > Rui Li
> > >> >
> >
>


-- 
Marios

Re: Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Marios Trivyzas]

Congrats Fabian!

On Mon, Nov 15, 2021 at 3:02 PM Francesco Guardiani 
wrote:

> Congratulations Fabian!
>
> On Mon, Nov 15, 2021 at 2:29 PM Yun Gao 
> wrote:
>
> > Congratulations Fabian!
> >
> > Best,
> > Yun
> >  --Original Mail --
> > Sender:Ingo Bürk 
> > Send Date:Mon Nov 15 21:27:36 2021
> > Recipients:dev 
> > CC: , 
> > Subject:Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul
> > Congratulations, Fabian!
> >
> > On Mon, Nov 15, 2021 at 2:17 PM Arvid Heise  wrote:
> >
> > > Hi everyone,
> > >
> > > On behalf of the PMC, I'm very happy to announce Fabian Paul as a new
> > Flink
> > > committer.
> > >
> > > Fabian Paul has been actively improving the connector ecosystem by
> > > migrating Kafka and ElasticSearch to the Sink interface and is
> currently
> > > driving FLIP-191 [1] to tackle the sink compaction issue. While he is
> > > active on the project (authored 70 PRs and reviewed 60), it's also
> worth
> > > highlighting that he has also been guiding external efforts, such as
> the
> > > DeltaLake Flink connector or the Pinot sink in Bahir.
> > >
> > > Please join me in congratulating Fabian for becoming a Flink committer!
> > >
> > > Best,
> > >
> > > Arvid
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
> > >
> >
>


-- 
Marios

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Marios Trivyzas]

Congrats Jing!

On Mon, Nov 15, 2021 at 3:20 PM Till Rohrmann  wrote:

> Congratulations Jing :-)
>
> Cheers,
> Till
>
> On Mon, Nov 15, 2021 at 2:48 PM Martijn Visser 
> wrote:
>
> > Congratulations Jing!
> >
> > On Mon, 15 Nov 2021 at 14:39, Timo Walther  wrote:
> >
> > > Hi everyone,
> > >
> > > On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
> > > Flink committer.
> > >
> > > Jing has been very active in the Flink community esp. in the Table/SQL
> > > area for quite some time: 81 PRs [1] in total and is also active on
> > > answering questions on the user mailing list. She is currently
> > > contributing a lot around the new windowing table-valued functions [2].
> > >
> > > Please join me in congratulating Jing Zhang for becoming a Flink
> > committer!
> > >
> > > Thanks,
> > > Timo
> > >
> > > [1] https://github.com/apache/flink/pulls/beyond1920
> > > [2] https://issues.apache.org/jira/browse/FLINK-23997
> > >
> >
>


-- 
Marios

Re: Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Till Rohrmann]

Congratulations Fabian :-)

Cheers,
Till

On Mon, Nov 15, 2021 at 3:03 PM Francesco Guardiani 
wrote:

> Congratulations Fabian!
>
> On Mon, Nov 15, 2021 at 2:29 PM Yun Gao 
> wrote:
>
> > Congratulations Fabian!
> >
> > Best,
> > Yun
> >  --Original Mail --
> > Sender:Ingo Bürk 
> > Send Date:Mon Nov 15 21:27:36 2021
> > Recipients:dev 
> > CC: , 
> > Subject:Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul
> > Congratulations, Fabian!
> >
> > On Mon, Nov 15, 2021 at 2:17 PM Arvid Heise  wrote:
> >
> > > Hi everyone,
> > >
> > > On behalf of the PMC, I'm very happy to announce Fabian Paul as a new
> > Flink
> > > committer.
> > >
> > > Fabian Paul has been actively improving the connector ecosystem by
> > > migrating Kafka and ElasticSearch to the Sink interface and is
> currently
> > > driving FLIP-191 [1] to tackle the sink compaction issue. While he is
> > > active on the project (authored 70 PRs and reviewed 60), it's also
> worth
> > > highlighting that he has also been guiding external efforts, such as
> the
> > > DeltaLake Flink connector or the Pinot sink in Bahir.
> > >
> > > Please join me in congratulating Fabian for becoming a Flink committer!
> > >
> > > Best,
> > >
> > > Arvid
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Till Rohrmann]

Congratulations Jing :-)

Cheers,
Till

On Mon, Nov 15, 2021 at 2:48 PM Martijn Visser 
wrote:

> Congratulations Jing!
>
> On Mon, 15 Nov 2021 at 14:39, Timo Walther  wrote:
>
> > Hi everyone,
> >
> > On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
> > Flink committer.
> >
> > Jing has been very active in the Flink community esp. in the Table/SQL
> > area for quite some time: 81 PRs [1] in total and is also active on
> > answering questions on the user mailing list. She is currently
> > contributing a lot around the new windowing table-valued functions [2].
> >
> > Please join me in congratulating Jing Zhang for becoming a Flink
> committer!
> >
> > Thanks,
> > Timo
> >
> > [1] https://github.com/apache/flink/pulls/beyond1920
> > [2] https://issues.apache.org/jira/browse/FLINK-23997
> >
>

Re: Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Francesco Guardiani]

Congratulations Fabian!

On Mon, Nov 15, 2021 at 2:29 PM Yun Gao 
wrote:

> Congratulations Fabian!
>
> Best,
> Yun
>  --Original Mail --
> Sender:Ingo Bürk 
> Send Date:Mon Nov 15 21:27:36 2021
> Recipients:dev 
> CC: , 
> Subject:Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul
> Congratulations, Fabian!
>
> On Mon, Nov 15, 2021 at 2:17 PM Arvid Heise  wrote:
>
> > Hi everyone,
> >
> > On behalf of the PMC, I'm very happy to announce Fabian Paul as a new
> Flink
> > committer.
> >
> > Fabian Paul has been actively improving the connector ecosystem by
> > migrating Kafka and ElasticSearch to the Sink interface and is currently
> > driving FLIP-191 [1] to tackle the sink compaction issue. While he is
> > active on the project (authored 70 PRs and reviewed 60), it's also worth
> > highlighting that he has also been guiding external efforts, such as the
> > DeltaLake Flink connector or the Pinot sink in Bahir.
> >
> > Please join me in congratulating Fabian for becoming a Flink committer!
> >
> > Best,
> >
> > Arvid
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Martijn Visser]

Congratulations Jing!

On Mon, 15 Nov 2021 at 14:39, Timo Walther  wrote:

> Hi everyone,
>
> On behalf of the PMC, I'm very happy to announce Jing Zhang as a new
> Flink committer.
>
> Jing has been very active in the Flink community esp. in the Table/SQL
> area for quite some time: 81 PRs [1] in total and is also active on
> answering questions on the user mailing list. She is currently
> contributing a lot around the new windowing table-valued functions [2].
>
> Please join me in congratulating Jing Zhang for becoming a Flink committer!
>
> Thanks,
> Timo
>
> [1] https://github.com/apache/flink/pulls/beyond1920
> [2] https://issues.apache.org/jira/browse/FLINK-23997
>

[ANNOUNCE] New Apache Flink Committer - Jing Zhang

[Timo Walther]

Hi everyone,

On behalf of the PMC, I'm very happy to announce Jing Zhang as a new 
Flink committer.


Jing has been very active in the Flink community esp. in the Table/SQL 
area for quite some time: 81 PRs [1] in total and is also active on 
answering questions on the user mailing list. She is currently 
contributing a lot around the new windowing table-valued functions [2].


Please join me in congratulating Jing Zhang for becoming a Flink committer!

Thanks,
Timo

[1] https://github.com/apache/flink/pulls/beyond1920
[2] https://issues.apache.org/jira/browse/FLINK-23997

[jira] [Created] (FLINK-24904) Add documentation for KDS Async Sink

[Zichen Liu (Jira)]

Zichen Liu created FLINK-24904:
--

 Summary: Add documentation for KDS Async Sink
 Key: FLINK-24904
 URL: https://issues.apache.org/jira/browse/FLINK-24904
 Project: Flink
  Issue Type: New Feature
  Components: Connectors / Common
Reporter: Zichen Liu
Assignee: Zichen Liu
 Fix For: 1.15.0


h2. Motivation

*User stories:*
 As a Flink user, I’d like to use Kinesis Data Streams as sink for my data 
pipeline.

*Scope:*
 * Implement an asynchronous sink for Kinesis Data Streams (KDS) by inheriting 
the AsyncSinkBase class. The implementation can for now reside in its own 
module in flink-connectors. The module and package name can be anything 
reasonable e.g. {{flink-connector-aws-kinesis}} for the module name and 
{{org.apache.flink.connector.aws.kinesis}} for the package name.
 * The implementation must use [the Kinesis Java 
Client|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/AmazonKinesisClient.html].
 * The implementation must allow users to configure the Kinesis Client, with 
reasonable default settings.
 * Implement an asynchornous sink writer for KDS by extending the 
AsyncSinkWriter. The implementation must deal with failed requests and retry 
them using the {{requeueFailedRequestEntry}} method. If possible, the 
implementation should batch multiple requests (PutRecordsRequestEntry objects) 
to KDS for increased throughput. The implemented Sink Writer will be used by 
the Sink class that will be created as part of this story.
 * Unit/Integration testing. Use Kinesalite (in-memory Kinesis simulation). We 
already use this in {{KinesisTableApiITCase}}.
 * Java / code-level docs.
 * End to end testing: add tests that hits a real AWS instance. (How to best 
donate resources to the Flink project to allow this to happen?)

h2. References

More details to be found 
[https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Yun Gao]

Congratulations Fabian!

Best,
Yun
 --Original Mail --
Sender:Ingo Bürk 
Send Date:Mon Nov 15 21:27:36 2021
Recipients:dev 
CC: , 
Subject:Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul
Congratulations, Fabian!

On Mon, Nov 15, 2021 at 2:17 PM Arvid Heise  wrote:

> Hi everyone,
>
> On behalf of the PMC, I'm very happy to announce Fabian Paul as a new Flink
> committer.
>
> Fabian Paul has been actively improving the connector ecosystem by
> migrating Kafka and ElasticSearch to the Sink interface and is currently
> driving FLIP-191 [1] to tackle the sink compaction issue. While he is
> active on the project (authored 70 PRs and reviewed 60), it's also worth
> highlighting that he has also been guiding external efforts, such as the
> DeltaLake Flink connector or the Pinot sink in Bahir.
>
> Please join me in congratulating Fabian for becoming a Flink committer!
>
> Best,
>
> Arvid
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
>

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Martijn Visser]

Congratulations Fabian!

On Mon, 15 Nov 2021 at 14:27, Ingo Bürk  wrote:

> Congratulations, Fabian!
>
> On Mon, Nov 15, 2021 at 2:17 PM Arvid Heise  wrote:
>
> > Hi everyone,
> >
> > On behalf of the PMC, I'm very happy to announce Fabian Paul as a new
> Flink
> > committer.
> >
> > Fabian Paul has been actively improving the connector ecosystem by
> > migrating Kafka and ElasticSearch to the Sink interface and is currently
> > driving FLIP-191 [1] to tackle the sink compaction issue. While he is
> > active on the project (authored 70 PRs and reviewed 60), it's also worth
> > highlighting that he has also been guiding external efforts, such as the
> > DeltaLake Flink connector or the Pinot sink in Bahir.
> >
> > Please join me in congratulating Fabian for becoming a Flink committer!
> >
> > Best,
> >
> > Arvid
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Ingo Bürk]

Congratulations, Fabian!

On Mon, Nov 15, 2021 at 2:17 PM Arvid Heise  wrote:

> Hi everyone,
>
> On behalf of the PMC, I'm very happy to announce Fabian Paul as a new Flink
> committer.
>
> Fabian Paul has been actively improving the connector ecosystem by
> migrating Kafka and ElasticSearch to the Sink interface and is currently
> driving FLIP-191 [1] to tackle the sink compaction issue. While he is
> active on the project (authored 70 PRs and reviewed 60), it's also worth
> highlighting that he has also been guiding external efforts, such as the
> DeltaLake Flink connector or the Pinot sink in Bahir.
>
> Please join me in congratulating Fabian for becoming a Flink committer!
>
> Best,
>
> Arvid
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
>

[ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Arvid Heise]

Hi everyone,

On behalf of the PMC, I'm very happy to announce Fabian Paul as a new Flink
committer.

Fabian Paul has been actively improving the connector ecosystem by
migrating Kafka and ElasticSearch to the Sink interface and is currently
driving FLIP-191 [1] to tackle the sink compaction issue. While he is
active on the project (authored 70 PRs and reviewed 60), it's also worth
highlighting that he has also been guiding external efforts, such as the
DeltaLake Flink connector or the Pinot sink in Bahir.

Please join me in congratulating Fabian for becoming a Flink committer!

Best,

Arvid

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction

[jira] [Created] (FLINK-24903) AdaptiveSchedulerTest.testJobStatusListenerNotifiedOfJobStatusChanges unstable

[Roman Khachatryan (Jira)]

Roman Khachatryan created FLINK-24903:
-

 Summary: 
AdaptiveSchedulerTest.testJobStatusListenerNotifiedOfJobStatusChanges unstable
 Key: FLINK-24903
 URL: https://issues.apache.org/jira/browse/FLINK-24903
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.15.0
Reporter: Roman Khachatryan
 Fix For: 1.15.0


[https://dev.azure.com/khachatryanroman/flink/_build/results?buildId=1225=logs=9dc1b5dc-bcfa-5f83-eaa7-0cb181ddc267=511d2595-ec54-5ab7-86ce-92f328796f20=7753]

 

Locally, it fails ~14 runs out of 100 (when running only 
testJobStatusListenerNotifiedOfJobStatusChanges in a loop).

 

It looks like job termination future is always completed before the 
jobStatusChangeListener is notified (AdaptiveScheduler.transitionToState, 
targetState.getState() completes the future).

Sleeping for 1ms before checking the assertion prevents the failure.

 

cc: [~trohrmann] 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] New Apache Flink Committer - Yangze Guo

[Martijn Visser]

Congrats Yangze!

On Mon, 15 Nov 2021 at 09:13, Yangze Guo  wrote:

> Thank you all very much!
> It's my honor to work with you in such a great community.
>
> Best,
> Yangze Guo
>
> On Mon, Nov 15, 2021 at 2:08 PM Yang Wang  wrote:
> >
> > Congratulations Yangze!
> >
> > Best,
> > Yang
> >
> > Yu Li  于2021年11月15日周一 下午2:01写道：
> >>
> >> Congratulations, Yangze!
> >>
> >> Best Regards,
> >> Yu
> >>
> >>
> >> On Mon, 15 Nov 2021 at 12:30, Rui Li  wrote:
> >>
> >> > Congratulations Yangze!
> >> >
> >> > On Mon, Nov 15, 2021 at 11:44 AM wenlong.lwl  >
> >> > wrote:
> >> >
> >> > > Congratulations, Yangze!
> >> > >
> >> > > On Mon, 15 Nov 2021 at 10:37, Zhilong Hong 
> wrote:
> >> > >
> >> > > > Congratulations, Yangze!
> >> > > >
> >> > > > On Mon, Nov 15, 2021 at 10:13 AM Qingsheng Ren <
> renqs...@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > > Congratulations Yangze!
> >> > > > >
> >> > > > > --
> >> > > > > Best Regards,
> >> > > > >
> >> > > > > Qingsheng Ren
> >> > > > > Email: renqs...@gmail.com
> >> > > > > On Nov 12, 2021, 10:11 AM +0800, Xintong Song <
> tonysong...@gmail.com
> >> > >,
> >> > > > > wrote:
> >> > > > > > Hi everyone,
> >> > > > > >
> >> > > > > > On behalf of the PMC, I'm very happy to announce Yangze Guo
> as a
> >> > new
> >> > > > > Flink
> >> > > > > > committer.
> >> > > > > >
> >> > > > > > Yangze has been consistently contributing to this project for
> >> > almost
> >> > > 3
> >> > > > > > years. His contributions are mainly in the resource
> management and
> >> > > > > > deployment areas, represented by the fine-grained resource
> >> > management
> >> > > > and
> >> > > > > > external resource framework. In addition to feature works,
> he's
> >> > also
> >> > > > > active
> >> > > > > > in miscellaneous contributions, including PR reviews, document
> >> > > > > enhancement,
> >> > > > > > mailing list services and meetup/FF talks.
> >> > > > > >
> >> > > > > > Please join me in congratulating Yangze Guo for becoming a
> Flink
> >> > > > > committer!
> >> > > > > >
> >> > > > > > Thank you~
> >> > > > > >
> >> > > > > > Xintong Song
> >> > > > >
> >> > > >
> >> > >
> >> >
> >> >
> >> > --
> >> > Best regards!
> >> > Rui Li
> >> >
>

[jira] [Created] (FLINK-24902) Port boolean <-> numeric casting logic to CastRule

[Francesco Guardiani (Jira)]

Francesco Guardiani created FLINK-24902:
---

 Summary: Port boolean <-> numeric casting logic to CastRule
 Key: FLINK-24902
 URL: https://issues.apache.org/jira/browse/FLINK-24902
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: Francesco Guardiani


More details on the parent task



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24901) Some further improvements of the pluggable shuffle framework

[Yingjie Cao (Jira)]

Yingjie Cao created FLINK-24901:
---

 Summary: Some further improvements of the pluggable shuffle 
framework
 Key: FLINK-24901
 URL: https://issues.apache.org/jira/browse/FLINK-24901
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Network
Reporter: Yingjie Cao
 Fix For: 1.15.0


This is an umbrella issue including several further improvements of the 
pluggable shuffle framework.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24900) Support to run multiple shuffle plugins in one session cluster

[Yingjie Cao (Jira)]

Yingjie Cao created FLINK-24900:
---

 Summary: Support to run multiple shuffle plugins in one session 
cluster
 Key: FLINK-24900
 URL: https://issues.apache.org/jira/browse/FLINK-24900
 Project: Flink
  Issue Type: Improvement
Reporter: Yingjie Cao


Currently, one Flink cluster can only use one shuffle plugin. However, there 
are cases where different jobs may need different shuffle implementations. By 
loading shuffle plugin with the plugin manager and letting jobs select their 
shuffle service freely, Flink can support to run multiple shuffle plugins in 
one session cluster.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Update Policy for old releases

[Till Rohrmann]

Hi everyone,

I want to second Stephan's points. I think that supporting LTS versions
rather fights symptoms than addressing the underlying problem that are
incompatible/behaviour changing changes between versions. I believe if we
can address this problem, then our Flink users will probably upgrade more
frequently.

Making upgrades as smooth as possible won't probably help all our users but
we as a community also have to think about the costs of maintaining more
Flink versions/supporting certain versions for a longer period of time. One
thing is that we would need more complex processes that ensure that older
Flink versions/LTS receives all the required bug fixes. I think this is
already hard enough when only supporting the last two releases. Moreover,
according to my experience, backporting fixes is usually not a problem if
there are no merge conflicts. However, the older the Flink code is, the
more likely it is that there are merge conflicts. In the worst case, issues
need to be fixed in a completely different way than they are solved in
master.

Additionally, we have to consider that longer support periods will mean
that we need to maintain our CI infrastructure and tooling the same period
as well. To give you an example, we probably would still have to operate
Travis if we had a LTS version. Given our existing problems with CI this
would add a lot more problems to this heap. Moreover, supporting more
versions would add more load to our CI infrastructure.

All in all, I would expect that longer support periods would slow us down
with other things. One of these things could be to make upgrades as smooth
as possible so that more Flink users upgrade more frequently.

@Tison: Concerning the problem that an older version contains a bug fix
that is not contained yet in the latest release of a newer version, I think
this is a valid problem. Right now, our users will have to check via JIRA
whether their problems are solved in the latest version of the new release
(e.g. when upgrading from 1.13.y to 1.14.x). Faster and more frequent
releases could mitigate the problem. Faster and lower overhead releases
could also allow us to release all versions in sync.

Cheers,
Till

On Sat, Nov 13, 2021 at 1:40 AM tison  wrote:

> Hi Stephan,
>
> Thanks for your email! I agree your points about compatibility and the
> upgrade experience should be smooth.
>
> The problem raised by Pitor is that, even if we officially support two
> latest versions, many users stay in an early version end-of-support.
> So the downside "no one ends up using the other versions" doesn't
> stand. I see that a number of companies like to test the latest version
> but not apply on prod if it's not robust - 1.9 & 1.10 is less used.
>
> If a non-LTS version resolves users' (urgent) issues and the release itself
> is robust, they will upgrade to that version. We have tried to support
> Java 9 and so do other projects.
>
> I'm a fan to keep two latest supported versions and am willing to work on
> improving compatibility to help users migrate. But if I make a choice
> between
> 4 supported versions and the LTS option, I like the LTS option.
>
> Here are several issues I met when trying to support a series of versions:
>
> 1. cheery-pick overhead grows, obviously.
> 2. cheery-pick owner is unclear. I think committers play the role of
> "reveal
> all the details", and it makes (1) worse.
> 3. version policy is harder. Generally a user wants to upgrade from
> 1.14.3 (latest 1.14.x) to 1.15.2 (latest 1.15.x) without regression.
> Given 1 & 2 the cherry-pick won't always get merged in time and
> it's possible that when 1.14.3 released, 1.15.2 is preparing. We have
> to do some simultaneous releases among 4 versions. And because there
> are 4 versions, such simultaneous releases will be more required.
>
> If a company wants to maintain an early version, it's welcome and can
> release
> all in its own - just like those who maintains Java 1.6. It's fine but not
> under
> committers/PMC maintenance.
>
> Additional input, 1.5, 1.7. 1.8, 1.11, 1.13 are loved.
>
> Best,
> tison.
>
>
> Stephan Ewen  于2021年11月13日周六 上午2:35写道：
>
> > I am of a bit different opinion here, I don't think LTS is the best way
> to
> > go.
> >
> > To my understanding, the big issue faced by users is that an upgrade is
> > simply too hard. Too many things change in subtle ways, you cannot just
> > take the previous application and configuration and expect it to run the
> > same after the upgrade. If that was much easier, users would be able to
> > upgrade more easily and more frequently (maybe not every month, but every
> > six months or so).
> >
> > In contrast, LTS is more about how long one provides patches and
> releases.
> > The upgrade problem is the same, between LTS versions, upgrades should
> > still be smooth. To make LTS to LTS smooth, we need to solve the same
> issue
> > as making it smooth from individual version to individual version. Unless
> > we expect non-linear upgrade 

Re: Re: [DISCUSS] Improve the name and structure of job vertex and operator name for job

[wenlong.lwl]

Hi, all, FYI the FLIP doc has been created ：
https://cwiki.apache.org/confluence/display/FLINK/FLIP-195%3A+Improve+the+name+and+structure+of+vertex+and+operator+name+for+sql+job

Best,
Wenlong

On Mon, 15 Nov 2021 at 11:41, wenlong.lwl  wrote:

> Hi all,
> Thanks for the feedback, It seems that the proposal is accepted by all of
> you guys. I will prepare a formal FLIP document and then go ahead to the
> vote stage.
> If any one has any other comments or suggestions, please let me know,
> thanks.
>
> Best,
> Wenlong
>
> On Fri, 12 Nov 2021 at 05:54, Neng Lu  wrote:
>
>> +1 (non-binding)
>> This change will really help to ease developer life.
>>
>> On Thu, Nov 11, 2021 at 6:33 AM Guowei Ma  wrote:
>>
>>> +1
>>> This would be very helpful for our debugging online job.
>>>
>>> Best,
>>> Guowei
>>>
>>>
>>> On Thu, Nov 11, 2021 at 8:03 PM Yuepeng Pan  wrote:
>>>
>>> > +1. It's useful to understand the job topology.
>>> > Looking forward to this feature.
>>> > Best,
>>> > Yuepeng Pan.
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > At 2021-11-11 19:44:44, "Yangze Guo"  wrote:
>>> > >+1. That's gonna help a lot for debugging.
>>> > >
>>> > >Best,
>>> > >Yangze Guo
>>> > >
>>> > >On Thu, Nov 11, 2021 at 7:37 PM Till Rohrmann 
>>> > wrote:
>>> > >>
>>> > >> This improvement looks like it makes the life of our users a lot
>>> easier
>>> > >> when it comes to understanding logs and reading the UI. Hence +1.
>>> > >>
>>> > >> Cheers,
>>> > >> Till
>>> > >>
>>> > >> On Thu, Nov 11, 2021 at 11:59 AM JING ZHANG 
>>> > wrote:
>>> > >>
>>> > >> > Big +1.
>>> > >> >
>>> > >> > This is a problem frequently encountered in our production
>>> platform,
>>> > look
>>> > >> > forward to this improvement.
>>> > >> >
>>> > >> > Best,
>>> > >> > Jing Zhang
>>> > >> >
>>> > >> > Martijn Visser  于2021年11月11日周四 下午6:26写道：
>>> > >> >
>>> > >> > > +1. Looks much better now
>>> > >> > >
>>> > >> > > On Thu, 11 Nov 2021 at 11:07, godfrey he 
>>> > wrote:
>>> > >> > >
>>> > >> > > > Thanks for driving this, this improvement solves a
>>> long-complained
>>> > >> > > > problem, +1
>>> > >> > > >
>>> > >> > > > Best,
>>> > >> > > > Godfrey
>>> > >> > > >
>>> > >> > > > Jark Wu  于2021年11月11日周四 下午5:40写道：
>>> > >> > > > >
>>> > >> > > > > +1 for this. It looks much more clear and structured.
>>> > >> > > > >
>>> > >> > > > > Best,
>>> > >> > > > > Jark
>>> > >> > > > >
>>> > >> > > > > On Thu, 11 Nov 2021 at 17:23, Chesnay Schepler <
>>> > ches...@apache.org>
>>> > >> > > > wrote:
>>> > >> > > > >
>>> > >> > > > > > I'm generally in favor of it, and there are already
>>> tickets
>>> > that
>>> > >> > > > > > proposed a dedicated operator/vertex description:
>>> > >> > > > > >
>>> > >> > > > > > https://issues.apache.org/jira/browse/FLINK-20388
>>> > >> > > > > > https://issues.apache.org/jira/browse/FLINK-21858
>>> > >> > > > > >
>>> > >> > > > > > On 11/11/2021 10:02, wenlong.lwl wrote:
>>> > >> > > > > > > Hi, all, I would like to start a discussion about an
>>> > improvement
>>> > >> > on
>>> > >> > > > name
>>> > >> > > > > > > and structure of job vertex name, mainly to improve
>>> > experience of
>>> > >> > > > > > debugging
>>> > >> > > > > > > and analyzing sql job at runtime.
>>> > >> > > > > > >
>>> > >> > > > > > > the main proposed changes including:
>>> > >> > > > > > > 1. separate description and name for operator, so that
>>> we
>>> > can
>>> > >> > have
>>> > >> > > > > > detailed
>>> > >> > > > > > > info at description and shorter name, which could be
>>> more
>>> > >> > friendly
>>> > >> > > > for
>>> > >> > > > > > > external systems like logging/metrics without losing
>>> useful
>>> > >> > > > information.
>>> > >> > > > > > > 2. introduce a tree-mode vertex description which can
>>> make
>>> > the
>>> > >> > > > > > description
>>> > >> > > > > > > more readable and easier to understand
>>> > >> > > > > > > 3. clean up and improve description for sql operator
>>> > >> > > > > > >
>>> > >> > > > > > > here is an example with the changes for a sql job:
>>> > >> > > > > > >
>>> > >> > > > > > > vertex name:
>>> > >> > > > > > > GlobalGroupAggregate[52] -> (Calc[53] ->
>>> > NotNullEnforcer[54] ->
>>> > >> > > Sink:
>>> > >> > > > > > > tb_ads_dwi_pub_hbd_spm_dtr_002_003[54], Calc[55] ->
>>> > >> > > > NotNullEnforcer[56]
>>> > >> > > > > > ->
>>> > >> > > > > > > Sink: tb_ads_dwi_pub_hbd_spm_dtr_002_004[56])
>>> > >> > > > > > > vertex description:
>>> > >> > > > > > > [52]:GlobalGroupAggregate(groupBy=[stat_date,
>>> spm_url_ab,
>>> > >> > client],
>>> > >> > > > > > > select=[stat_date, spm_url_ab, client, COUNT(count1$0)
>>> AS
>>> > >> > > > > > > clk_cnt_app_mtr_001, COUNT(distinct$0 count$1) AS
>>> > >> > > clk_uv_app_mtr_001,
>>> > >> > > > > > > COUNT(count1$2) AS clk_cnt_app_mtr_002, COUNT(distinct$0
>>> > count$3)
>>> > >> > > AS
>>> > >> > > > > > > clk_uv_app_mtr_002, COUNT(count1$4) AS
>>> clk_cnt_app_mtr_003,
>>> > >> > > > > > > COUNT(distinct$0 count$5) AS clk_uv_app_mtr_003]) :-
>>> > >> > > > > > > 

[jira] [Created] (FLINK-24899) Enable data compression for blocking shuffle by default

[Yingjie Cao (Jira)]

Yingjie Cao created FLINK-24899:
---

 Summary: Enable data compression for blocking shuffle by default
 Key: FLINK-24899
 URL: https://issues.apache.org/jira/browse/FLINK-24899
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Reporter: Yingjie Cao
 Fix For: 1.15.0


Currently, shuffle data compression is not enabled by default. Shuffle data 
compression is important for blocking data shuffle and enabling shuffle data 
compression by default can improve the usability.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24898) Some further improvements of sort-shuffle

[Yingjie Cao (Jira)]

Yingjie Cao created FLINK-24898:
---

 Summary: Some further improvements of sort-shuffle
 Key: FLINK-24898
 URL: https://issues.apache.org/jira/browse/FLINK-24898
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Reporter: Yingjie Cao
 Fix For: 1.15.0


This is an umbrella issue including several further improvements of 
sort-shuffle.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Flink support for OrderedListState

[Jan Lukavský]

Hi Reuven,

cc dev@flink.apache.org.

 Jan

On 11/12/21 04:35, Reuven Lax wrote:
OrderedListState 
 was 
added to Beam over a year ago. To date it is only supported by the 
Dataflow runner and the DirectRunner. I want to see if it's 
possible to support this well on the Flink runner (and eventually on 
other runners as well).


This is a state type that is indexed by an int64 sorting key (the Beam 
API exposes this as a 64-bit timestamp, as that's the most-common use 
case, but fundamentally it's just an integer). Users can 
insert element, fetch ranges of elements, and delete ranges of elements.


Is there any way to implement this on Flink? I could of course add a 
naive implementation - store everything in a ListState and have the 
Flink runner sort the list every time it is fetched. This seems quite 
inefficient, and the range deletes will be even less efficient. 
https://issues.apache.org/jira/browse/FLINK-6219 seems to imply that 
Flink has considered sorted state primitives, but I don't see any 
activity on this issue.


Is there any way to do this, or should I just add the naive 
implementation?


Reuven

Re: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage

[Jingsong Li]

Hi Timo,

> It would be great if we can add not only `Receive any type of changelog` but 
> also `Receive any type of datatype`.

Nice, I think we can.

> Please clarify whether the compact DDL is a synchronous or asynchrounous 
> operation in the API? So far all DDL was synchrounous. And only DML 
> asynchrounous.

It should be a synchronous operation.

> I find this 'change-tracking' = 'false' a bit confusing. Even in batch 
> scenarios we have a changelog, only with insert-only changes. Can you 
> elaborate? Wouldn't 'exclude-from-log-store' or 'exclude-log-store' or 
> 'log.disabled' be more accurate?

Change tracking is from Oracle and snowflake [1][2][3]. It matches the
"emit-changes" syntax. It means that after closing, the downstream
consumption cannot obtain the corresponding changes.

> DESCRIBE DETAIL TABLE

+1 to `DESCRIBE TABLE EXTENDED`.

> Set checkpoint interval to 1 min if checkpoint is not enabled
when the planner detects a sink to built-in dynamic table.
This sounds like too much magic to me.

You are right. And one minute may not be enough for all situations. +1
to throw detailed exception to alert user.

> GenericCatalog to `Catalog#supportesTableStorage`

I originally thought about completely distinguishing it from external
catalog, but it is also possible to add a new method.

> CatalogBaseTable.TableKind

Yes, we can create a new TableKind for this table. Catalog can easily
distinguish them.

> enrichOptions(Context context)
Why is this method returning a Map? Shouldn't the caller
assume that all options enriched via `CatalogTable.copy` should have
been applied by `enrichOptions`?

Yes, the planner adds some options to the table before creating it.

>  Partitioning and Event-time:
Have you considered to support semantics similar to
`sink.partition-commit.trigger` based on `partition-time`. It could
beneficial to have the partitions committed by watermarks as well. My
biggest concern is how we can enable watermarking end-to-end using a
file store (I think for log store this should not be a problem?).

Yes, we can also write watermark to storage, which has many advantages:
- Users can see the progress of a partition.
- Partition commit can have some optimization for this partition, like compact.
- Lookup join a table, we can use the watermark to align.   Many users
complain that the lookup join cannot find data when the lookup table
is not ready, it is difficult to align the main stream and the lookup
table.

But this is not the current blocker. We can improve this in future.

[1] https://docs.snowflake.com/en/user-guide/streams.html
[2] 
https://docs.oracle.com/database/121/ADMQS/GUID-3BAA0D48-CA35-4CD7-810E-50C703DC6FEB.htm
[3] 
https://docs.oracle.com/database/121/DWHSG/advmv.htm#DWHSG-GUID-F7394DFE-7CF6-401C-A312-C36603BEB01B

Best,
Jingsong

[jira] [Created] (FLINK-24897) Enable application mode on YARN to use usrlib

[Biao Geng (Jira)]

Biao Geng created FLINK-24897:
-

 Summary: Enable application mode on YARN to use usrlib
 Key: FLINK-24897
 URL: https://issues.apache.org/jira/browse/FLINK-24897
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / YARN
Reporter: Biao Geng


Hi there, 
I am working to utilize application mode to submit flink jobs to YARN cluster 
but I find that currently there is no easy way to ship my user-defined 
jars(e.g. some custom connectors or udf jars that would be shared by all jobs) 
and ask the FlinkUserCodeClassLoader to load classes in these jars. 

I checked some relevant jiras, like  [#FLINK-21289]. In k8s mode, there is a 
solution that users can use `usrlib` directory to store there user-defined jars 
and these jars would be loaded by FlinkUserCodeClassLoader when the job is 
executed on JM/TM.

But on YARN mode, `usrlib` does not work as that:

In this method(org.apache.flink.yarn.YarnClusterDescriptor#addShipFiles), if I 
want to use `yarn.ship-files` to ship `usrlib` from my flink client(in my local 
machine) to remote cluster, I must not set  UserJarInclusion to DISABLED due to 
the checkArgument(). However, if I do not set that option to DISABLED, the user 
jars to be shipped will be added into systemClassPaths. As a result, classes in 
those user jars will be loaded by AppClassLoader. 

But if I do not ship these jars, there is no convenient way to utilize these 
jars in my flink run command. Currently, all I can do is to use `-C` option, 
which means I have to upload my jars to some shared store first and then use 
these remote paths. It is not so perfect as we have already make it possible to 
ship jars or files directly and we also introduce `usrlib` in application mode 
on YARN. It would be more user-friendly, if we can allow shipping `usrlib` from 
local to remote cluster while using FlinkUserCodeClassLoader to load classes in 
the jars in `usrlib`.

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[DISCUSS] Allow user to unsubscribe by sending "Unsubscribe" directly to user mailing list

[Caizhi Weng]

Hi devs!

I've seen unsubscribe emails coming into the mailing lists now and then
these days (both user and user-zh mailing lists).

Instead of telling the user to send emails to some specific email addresses
again and again, can we allow user to unsubscribe by sending
"Unsubscribe" directly to user mailing list? By doing so we can also filter
out these emails to keep a cleaner topic lists in the mailing lists.

Re: [ANNOUNCE] Documentation now available at nightlies.apache.org

[Chesnay Schepler]

flink.apache.org/docs/1.14 never existed though and would also result in 
a 404.


On 15/11/2021 09:21, Yu Li wrote:

Thank you all for the efforts! However, it seems to me that
https://flink.apache.org/docs/1.14/ will be redirected to
https://nightlies.apache.org/flink/flink-docs-master/1.14/ and get a 404
error. The correct link to redirect to should be
https://nightlies.apache.org/flink/flink-docs-release-1.14/. Would be great
if any of us could further correct this (I guess some regex correction
could do the trick), thanks!

Best Regards,
Yu


On Thu, 11 Nov 2021 at 15:26, David Morávek  wrote:


Also big thanks to Gavin for setting the redirects up! ;)

Best,
D.

On Thu, Nov 11, 2021 at 2:34 AM Leonard Xu  wrote:


Nice !
Thanks chesnay for the continuous effort.


在 2021年11月11日，06:58，Chesnay Schepler  写道：

A redirect from ci.apache.org to nightlies.apache.org has been set up.

On 05/11/2021 04:44, Yun Tang wrote:

Hi Chesnay,

It seems that the redirection has not been completed. As old

documentation hasn't updated for days, shall we add a waring on top of
documentation, like current warning of "This documentation is for an
out-of-date version of Apache Flink." to tell people to go to new

location?

Best
Yun Tang

From: Chesnay Schepler 
Sent: Friday, September 10, 2021 13:32
To: dev@flink.apache.org ; Leonard Xu <

xbjt...@gmail.com>

Subject: Re: [ANNOUNCE] Documentation now available at

nightlies.apache.org

A redirection will be setup by infra at some point.

On 10/09/2021 05:23, Leonard Xu wrote:

Thanks Chesnay for the migration work.

Should we add a redirection for the old documentation site:

https://ci.apache.org/flink/flink-docs-master/  to make

it redirect to the new one:

https://nightlies.apache.org/flink/flink-docs-master/ ?

The bookmark in users’ browser should still be the old one, I googled

"flink documents" which also returned the old one.

And the old one won’t be updated and would be outdated soon.

Best,
Leonard


在 2021年9月6日，17:11，Chesnay Schepler  写道：

Website has been updated to point to nightlies.apache.org as well.

On 03/09/2021 08:03, Chesnay Schepler wrote:

The migration is pretty much complete and the documentation is now

available at nightlies.apache.org .

Please click around a bit and check if anything is broken.

If no issues are reported by the end of today I will update the

links on the website.

On 01/09/2021 10:11, Chesnay Schepler wrote:

We are in the final steps of migrating the documentation to the

new

buildbot setup.

Because of that the documentation currently available at

ci.apache.org will NOT be updated until further notice because the old
builders have been deactivated while we iron out kinks in the new ones.

I will keep you updated on the progress.

[jira] [Created] (FLINK-24896) Failed to execute goal com.github.siom79.japicmp:japicmp-maven-plugin:0.11.0:cmp (default) on project flink-metrics-core

[Yun Gao (Jira)]

Yun Gao created FLINK-24896:
---

 Summary: Failed to execute goal 
com.github.siom79.japicmp:japicmp-maven-plugin:0.11.0:cmp (default) on project 
flink-metrics-core
 Key: FLINK-24896
 URL: https://issues.apache.org/jira/browse/FLINK-24896
 Project: Flink
  Issue Type: Bug
  Components: Build System / Azure Pipelines
Affects Versions: 1.15.0
Reporter: Yun Gao


{code:java}
[ERROR] Failed to execute goal 
com.github.siom79.japicmp:japicmp-maven-plugin:0.11.0:cmp (default) on project 
flink-metrics-core: Execution default of goal 
com.github.siom79.japicmp:japicmp-maven-plugin:0.11.0:cmp failed: Marshalling 
of XML document failed: Implementation of JAXB-API has not been found on module 
path or classpath. com.sun.xml.internal.bind.v2.ContextFactory -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn  -rf :flink-metrics-core
 {code}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24895) SqlTimestampSerializer#deserialize forgets to fully read from source when the record is null.

[bx123 (Jira)]

bx123 created FLINK-24895:
-

 Summary: SqlTimestampSerializer#deserialize forgets to fully read 
from source when the record is null.
 Key: FLINK-24895
 URL: https://issues.apache.org/jira/browse/FLINK-24895
 Project: Flink
  Issue Type: Bug
  Components: API / Type Serialization System
Reporter: bx123


As we support null Timestamp in this Serializer. In the 2 deserialize() 
methods, when we read the first long and get Long.MIN_VALUE, we directly return 
null and forget to read the next int which responds to nanos.

If there is really a null Timestamp that has been serialized before, we may get 
wrong value when we deserialize the Object after this timestamp. So before 
return null,  source.skipBytesToRead(4) is needed to make the source in the 
right position.

By the way, I find the copy(source, target) is implemented by deserialize the 
intermediate value, while its more efficient to just copy 12 bytes from source 
to target like target.write(source, 12).

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-24894) Flink on k8s, open the HA mode based on KubernetesHaServicesFactory ，When I deleted the job, the config map created under the HA mechanism was not deleted.

[john (Jira)]

john created FLINK-24894:


 Summary: Flink on k8s, open the HA mode based on 
KubernetesHaServicesFactory ，When I deleted the job, the config map created 
under the HA mechanism was not deleted.
 Key: FLINK-24894
 URL: https://issues.apache.org/jira/browse/FLINK-24894
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Kubernetes
 Environment: 1.13.2 
Reporter: john


Flink on k8s, open the HA mode based on KubernetesHaServicesFactory. When I 
deleted the job, the config map created under the HA mechanism was not deleted. 
This leads to a problem: if my last concurrency was 100, changing to 40 this 
time will not take effect. This can be understood because jobgraph recovered 
from high-availability.storageDir and ignored the client's.
My question is: When deleting a job, the config map created under the HA 
mechanism is not deleted. Is this the default mechanism of HA, or is it a bug?



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Re: [VOTE] FLIP-174: Improve the WithParam interface

[Dong Lin]

Hi everyone,

We would like to update WithParams::set(...) to throw an
IllegalArgumentException if the given Param instance is not defined on the
WithParams instance.

The reason is that there is no use-case where users will want to set a
parameter that is not defined on the WithParams instance. And by making
this limitation, the save/load implementation could be simpler as it does
not need to store the entire Param definition (including validator) which
can not be expressed as a json string.

Please let us know if you have any suggestions on the proposed changes.

Thanks!
Dong



On Sat, Nov 6, 2021 at 3:35 PM Dong Lin  wrote:

> Thanks everyone for your votes!
>
> The proposal has passed with the following votes:
>
> +1 (Binding): 3 (Jiangjie, Yun, Dian)
> +1 (Non-binding): 0
> -1: 0
>
> Cheers,
> Dong
>
>
> On Sat, Nov 6, 2021 at 9:41 AM Dian Fu  wrote:
>
>> +1
>>
>> On Fri, Nov 5, 2021 at 10:09 PM Yun Gao 
>> wrote:
>>
>> > +1.
>> >
>> > Very thanks for drafting the FLIP!
>> >
>> >
>> > Best,
>> > Yun
>> >
>> >
>> > --
>> > Sender:Becket Qin
>> > Date:2021/11/05 18:15:11
>> > Recipient:dev
>> > Theme:Re: [VOTE] FLIP-174: Improve the WithParam interface
>> >
>> > +1. Thanks for the writeup, Dong and Zhipeng.
>> >
>> > Cheers,
>> >
>> > Jiangjie (Becket) Qin
>> >
>> >
>> >
>> > On Fri, Oct 15, 2021 at 2:11 PM Dong Lin  wrote:
>> >
>> > > Hi all,
>> > >
>> > > We would like to start the vote for FLIP-174: Improve the WithParam
>> > > interface [1]. This FLIP was discussed in this thread [2].
>> > >
>> > > The goal of this FLIP is to improve the experience of defining, saving
>> > and
>> > > loading parameters of stages (e.g. Transformer and Estimator) in the
>> > Flink
>> > > ML repo。
>> > >
>> > > The vote will be open for at least 72 hours, following the consensus
>> > voting
>> > > process.
>> > >
>> > > Thanks!
>> > > Dong Lin and Zhipeng Zhang
>> > >
>> > > [1]
>> > >
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=181311361
>> > > [2]
>> > >
>> > >
>> >
>> https://lists.apache.org/thread.html/re56b61ec0a4a20dadea8f4fe757b8baf94f2eb42b1044e9dd003bf23%40%3Cdev.flink.apache.org%3E
>> > >
>> >
>> >
>>
>

Re: [DISCUSS] FLIP-188: Introduce Built-in Dynamic Table Storage

[Jingsong Li]

Hi Stephan,

###

about PK:

> (6) Different consistency guarantees with PK and without PK
> (7) Eventual Consistency Mode vs. Faster Checkpoints

Let's clarify one thing first:
For sink with PK, Flink SQL cannot generate complete changes (without
UPDATE_BEFORE).
- For databases like mysql, MySQL will generate the UPDATE_BEFORE,
that is, MySQL will query the old value corresponding to each data PK
before generating binlog. Mysql upsert sink does not rely on
checkpoint, during failing, Mysql will read binlog and recover the
database, then get eventual consistency.
- For Upsert-Kakfa, regardless of the consistency guarantee, the
resulting log may not contain a complete/correct UPDATE_BEFORE, so
there needs to be a normalized node downstream to remove the
duplication.

So for us, either option 1 or option 2, we don't have the ability to
query per PK at the moment, so we choose 2, which needs downstream de
duplicated records.
If we enable log transactions for cases with PK, the only advantage is
" transactional consistency vs eventual consistency". I don't know if
the price is reasonable. We are moving towards the fast checkpoint,
which is correct, but it will not arrive so soon. For storage, it also
requires a lot of work to adapt to this mode (This is not very easy,
and efficiency is often considered). This means that the current
default mode is "slow". We need to explain to users under what
scenario they can add option to speed up the pipeline.

> (5) Different formats for cases with PK and without PK

Jing Zhang also mentioned that we may need to expose the log format.
So I am +1 to unify log format with PK and without PK.
We can introduce an option to let users choose debezium-json or
debezium-avro-confluent in future. They are all public formats.

###

about Concurrent Write:

The story is that the streaming pipeline is running all the time, and
it is difficult to control and modify it.
However, in many cases, we need to recalculate the historical data in
the data warehouse. For example, there are some data problems in the
partition 10 days ago. I need to recalculate the whole pipeline. In
this case, we need a batch job pipeline instead of correcting the root
table of the streaming pipeline. Because the streaming job can not
recalculate the data 10 days ago.
- The streaming job state may have expired.
- Computing is not as efficient as batch jobs.

Conceptually, all changes will generate logs, but in this case, we
need to turn off log generation, so this is "Change Tracking".
So yes. it just skip writing to the change log.

Meanwhile, the streaming job is still running, so two jobs are
committing snapshots at the same time.

The lock is just for multiple snapshot commits. For example, the
insert statements and compact will produce new snapshots.

> What exactly is the optimistic locking behavior

Optimism is reflected in the deletion of files. If the file to be
deleted is found missing when committing, it will fail instead of
locking at the beginning.

> what is the lock scoped to? a single query (via jobID)?

Lock scope is database and table name.

> what happens if a system that holds (like an entry in the Hive Metastore 
> Database) a lock crashes and is manually restarted as a new job (from the 
> latest checkpoint)? does that need manual admin intervention to release the 
> lock (manually update the HMS Database)?

If the client crashes, the lock it holds will also expire with the
heartbeat timeout.

Best,
Jingsong

Re: [ANNOUNCE] Documentation now available at nightlies.apache.org

[Yu Li]

Thank you all for the efforts! However, it seems to me that
https://flink.apache.org/docs/1.14/ will be redirected to
https://nightlies.apache.org/flink/flink-docs-master/1.14/ and get a 404
error. The correct link to redirect to should be
https://nightlies.apache.org/flink/flink-docs-release-1.14/. Would be great
if any of us could further correct this (I guess some regex correction
could do the trick), thanks!

Best Regards,
Yu


On Thu, 11 Nov 2021 at 15:26, David Morávek  wrote:

> Also big thanks to Gavin for setting the redirects up! ;)
>
> Best,
> D.
>
> On Thu, Nov 11, 2021 at 2:34 AM Leonard Xu  wrote:
>
> > Nice !
> > Thanks chesnay for the continuous effort.
> >
> > > 在 2021年11月11日，06:58，Chesnay Schepler  写道：
> > >
> > > A redirect from ci.apache.org to nightlies.apache.org has been set up.
> > >
> > > On 05/11/2021 04:44, Yun Tang wrote:
> > >> Hi Chesnay,
> > >>
> > >> It seems that the redirection has not been completed. As old
> > documentation hasn't updated for days, shall we add a waring on top of
> > documentation, like current warning of "This documentation is for an
> > out-of-date version of Apache Flink." to tell people to go to new
> location?
> > >>
> > >> Best
> > >> Yun Tang
> > >> 
> > >> From: Chesnay Schepler 
> > >> Sent: Friday, September 10, 2021 13:32
> > >> To: dev@flink.apache.org ; Leonard Xu <
> > xbjt...@gmail.com>
> > >> Subject: Re: [ANNOUNCE] Documentation now available at
> > nightlies.apache.org
> > >>
> > >> A redirection will be setup by infra at some point.
> > >>
> > >> On 10/09/2021 05:23, Leonard Xu wrote:
> > >>> Thanks Chesnay for the migration work.
> > >>>
> > >>> Should we add a redirection for the old documentation site:
> > https://ci.apache.org/flink/flink-docs-master/  to make
> > >>> it redirect to the new one:
> > https://nightlies.apache.org/flink/flink-docs-master/ ?
> > >>>
> > >>> The bookmark in users’ browser should still be the old one, I googled
> > "flink documents" which also returned the old one.
> > >>> And the old one won’t be updated and would be outdated soon.
> > >>>
> > >>> Best,
> > >>> Leonard
> > >>>
> >  在 2021年9月6日，17:11，Chesnay Schepler  写道：
> > 
> >  Website has been updated to point to nightlies.apache.org as well.
> > 
> >  On 03/09/2021 08:03, Chesnay Schepler wrote:
> > > The migration is pretty much complete and the documentation is now
> > available at nightlies.apache.org .
> > >
> > > Please click around a bit and check if anything is broken.
> > >
> > > If no issues are reported by the end of today I will update the
> > links on the website.
> > >
> > > On 01/09/2021 10:11, Chesnay Schepler wrote:
> > >> We are in the final steps of migrating the documentation to the
> new
> > buildbot setup.
> > >>
> > >> Because of that the documentation currently available at
> > ci.apache.org will NOT be updated until further notice because the old
> > builders have been deactivated while we iron out kinks in the new ones.
> > >>
> > >> I will keep you updated on the progress.
> > >>
> > >>
> > >
> >
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Yangze Guo

[Yangze Guo]

Thank you all very much!
It's my honor to work with you in such a great community.

Best,
Yangze Guo

On Mon, Nov 15, 2021 at 2:08 PM Yang Wang  wrote:
>
> Congratulations Yangze!
>
> Best,
> Yang
>
> Yu Li  于2021年11月15日周一 下午2:01写道：
>>
>> Congratulations, Yangze!
>>
>> Best Regards,
>> Yu
>>
>>
>> On Mon, 15 Nov 2021 at 12:30, Rui Li  wrote:
>>
>> > Congratulations Yangze!
>> >
>> > On Mon, Nov 15, 2021 at 11:44 AM wenlong.lwl 
>> > wrote:
>> >
>> > > Congratulations, Yangze!
>> > >
>> > > On Mon, 15 Nov 2021 at 10:37, Zhilong Hong  wrote:
>> > >
>> > > > Congratulations, Yangze!
>> > > >
>> > > > On Mon, Nov 15, 2021 at 10:13 AM Qingsheng Ren 
>> > > wrote:
>> > > >
>> > > > > Congratulations Yangze!
>> > > > >
>> > > > > --
>> > > > > Best Regards,
>> > > > >
>> > > > > Qingsheng Ren
>> > > > > Email: renqs...@gmail.com
>> > > > > On Nov 12, 2021, 10:11 AM +0800, Xintong Song > > >,
>> > > > > wrote:
>> > > > > > Hi everyone,
>> > > > > >
>> > > > > > On behalf of the PMC, I'm very happy to announce Yangze Guo as a
>> > new
>> > > > > Flink
>> > > > > > committer.
>> > > > > >
>> > > > > > Yangze has been consistently contributing to this project for
>> > almost
>> > > 3
>> > > > > > years. His contributions are mainly in the resource management and
>> > > > > > deployment areas, represented by the fine-grained resource
>> > management
>> > > > and
>> > > > > > external resource framework. In addition to feature works, he's
>> > also
>> > > > > active
>> > > > > > in miscellaneous contributions, including PR reviews, document
>> > > > > enhancement,
>> > > > > > mailing list services and meetup/FF talks.
>> > > > > >
>> > > > > > Please join me in congratulating Yangze Guo for becoming a Flink
>> > > > > committer!
>> > > > > >
>> > > > > > Thank you~
>> > > > > >
>> > > > > > Xintong Song
>> > > > >
>> > > >
>> > >
>> >
>> >
>> > --
>> > Best regards!
>> > Rui Li
>> >

Re: Re: [VOTE] FLIP-173: Support DAG of algorithms

[Dong Lin]

Hi everyone,

We would like to make the following API changes to this FLIP:

1) Updates Stage's Java doc to say the following
Every Stage subclass should implement a static method with signature
"static T load(StreamExecutionEnvironment env, String path)", where T
refers to the concrete subclass. This static method should instantiate a
new stage instance based on the data read from the given path.

2) Updates Model::setModelData(...) to return the Model instance itself.

The reason for the first change is that the static load(...) method will
need to create Table instances to read model data. And it needs to use the
same StreamExecutionEnvironment to create those Table instances as the
StreamExecutionEnvironment used in the rest of the program. The reason for
the second change is to improve usability. This changes allows the users to
do something like "return new KMeansModel.setModelData(...)"

Please let us know if you have any suggestions on the proposed changes.

Thanks!
Dong


On Mon, Sep 20, 2021 at 10:19 PM Dong Lin  wrote:

> Thanks everyone for your votes!
>
> The proposal has passed with the following votes:
>
> +1 (Binding): 3 (Stephan, Becket, Yun)
> +1 (Non-binding): 0
> -1: 0
>
> Thanks,
> Dong
>
> On Sun, Sep 19, 2021 at 8:21 PM Yun Gao 
> wrote:
>
>> Very thanks Dong and Zhipeng for proposing the FLIP
>> on Flink ML!
>>
>> The FLIP looks good to me and +1 from my side,
>>
>> Best,
>> Yun
>>
>>
>>
>>  --Original Mail --
>> Sender:Becket Qin 
>> Send Date:Sun Sep 19 12:35:39 2021
>> Recipients:+dev 
>> Subject:Re: [VOTE] FLIP-173: Support DAG of algorithms
>> +1 on the FLIP, really excited to see the relaunch of the Flink ML
>> development!
>>
>> Thanks Dong and Zhipeng for driving this.
>>
>> Cheers,
>>
>> Jiangjie (Becket) Qin
>>
>> On Sat, Sep 18, 2021 at 12:17 AM Stephan Ewen  wrote:
>>
>> > I think this will be a useful addition.
>> >
>> > Regarding the API and specific design decisions: I think this looks ok.
>> > I didn't dig very deep and would be fine to just go with the author's
>> > proposal. The main motivation for having a separate flink-ml repository
>> was
>> > to develop more easily, make changes and iterate faster without having
>> to
>> > weight every design as carefully as we need to do it in core Flink.
>> >
>> > So +1 from my side
>> >
>> > On Fri, Sep 10, 2021 at 4:33 AM Dong Lin  wrote:
>> >
>> > > Hi all,
>> > >
>> > > We would like to start the vote for FLIP-173: Support DAG of
>> > > algorithms [1]. This FLIP was discussed in this thread [2].
>> > >
>> > > The proposal extended the Flink ML API to support DAG of algorithms
>> where
>> > > each algorithm could have multiple inputs and multiple outputs. It
>> also
>> > > extended Flink ML API to support online learning scenarios where a
>> > > long-running Model instance needs to be continuously updated by the
>> > latest
>> > > model data generated by another long-running Estimator instance.
>> > >
>> > > The vote will be open for at least 72 hours, following the consensus
>> > voting
>> > > process.
>> > >
>> > > Thanks!
>> > > Dong Lin and Zhipeng Zhang
>> > >
>> > > [1]
>> > >
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=184615783
>> > > [2]
>> > >
>> > >
>> >
>> https://lists.apache.org/thread.html/r6729f351fb1bc13a93754c199d5fee1051cc8146e22374737c578779%40%3Cdev.flink.apache.org%3E
>> > >
>> >
>>
>